In a search, we have very low backgound yield (0.68) and 0.28 signals. We use FrequentistCalculator to get the p-value scanning curves. For each point we use about 10000 toys.
The issue is the +2sigma band seems never converge, no matter how we change the PoI scanning ranges. The other curves (expected, ±1sigma, -2sigma) look quite OK.
please see the attached 4 plots, they are with scanning range 0-20 and 0-4000 respectively. Please kindly help.
The problem could be caused by some failing fits. It is difficult to see from the attached test statistic plots, but the fact that the 2-sigma band does not converge means that you have a long tail of the test statistic distribution. Is this correct ? Could you make a plot for the test statistic distributions (in log-scale) to see this better ?
If this is correct you should investigate better (using for example the detailed output information) what causes this large tail in the S+B test statistic distribution.
Thanks a lot for the suggestions! We checked the fittings carefully, wouldn’t see any failed/strange things.
Would you look the attached new test-statistic distribution plots again, we wouldn’t see there are large tails in the S+B test-statistics … maybe I mis-understand what you mean ?
I also copy the workspace to be here:
/afs/cern.ch/work/x/xulou/public/test.root