Thanks Lorenzo!
Now I understand how to make a 3d FFT.
It is important to use
void TFFTRealComplex::SetPoint(const Int_t *ipoint, ...)
(not the method with 1-dimensional int ipoint) to set function values (ipoint is a multidimensional vector of bins).
Another example of a convolution (in 1d) is in RooFFTConvPdf::fillCacheSlice.
However, I found that I don’t know how to update RooFit variables, so I decided not to use a 3-dimensional convolution, but use 3 one-dimensional convolutions (probably that is fine from the statistical point of view). The distribution of a projection is again a convolution, because each projection is a sum of projections (x = x_gs + x_exp).
Memory is probably still another concern for 3d convolutions: @Wouter_Verkerke wrote in 2009, that “10000 x 100 x 100 bins = 100 million bins” and takes about 7.8 Gb of memory.
Yaroslav