Well, a “brutal fix” would be to redefine your function:
if (x >= y) return f(x, y); else return 0;
and then integrate it from 0 to 1 in both dimensions (i.e. in a 2D rectangular region).
Some warnings: Creating TF1 from TGraph - #10 by Pepe_Le_Pew and Convolution of two TF1 - #8 by Pepe_Le_Pew
BTW. Note that there’s a special “Stat and Math Tool Support” forum here.