I don’t know if there is a dedicated function, but what you can do is:
1 - define the closed polygon as a TCutg (https://root.cern/doc/master/classTCutG.html)
2 - loop over the histogram bins, get the bin centres and use IsInside() to check whether the point is inside the area; if it is, get its contents and add them.
for (Int_t ix=1; ix<Nbx; ++ix) {
for (Int_t iy=1; iy<Nby; ++iy) {
x = h->GetXaxis()->GetBinCenter(ix);
y = h->GetYaxis()->GetBinCenter(iy);
if (cutg->IsInside(x,y)) {
// add the bin contents
}
}
}