I am using Root/Roostat to calculate a CLs, using the class HybridCalculatorOriginal.
However I noticed I obtain different results between two version of Root/Roostat.
I attached the (short?) code I’m using and the different files in input.
I’m using it as an executable, compiled with gcc/g++.
With ROOT version 5.27, I obtain the following results :
In those last result, I obtained a CLs value which is higher than 1.
Given the definition of CLs (A.L.Read, Modified frequentist analysis of search results (The CLs method)), this is unexpected. I can’t see how it could be possible to have a CLs > 1.
Is there an explanation for the difference of results between the 2 versions ?
Was there any change in the definition of the CLs between the 2 versions ?
Hi,
There have been a lots of changes between 5.27 and 5.32. The definition of CLs in the class HypoTestResult terms of Null and Alternate p values has been changed between the versions and this could cause a problem, if the HybridCalculatorOriginal class is not updated accordingly.
You should use the HybridCalculator class, which should provide correct results. If you have problem using this class please let us know,
Best Regards
Lorenzo
P.S. I have moved the post to the “Stat and Math Tool Support Board”
I tried to look at the class you mentioned, HybridCalculator. I took a look at the example provided here : root.cern.ch/viewvc/trunk/tutori … uctional.C
I couldnt get the original file to compile properly, so I changed the statistic test using the RatioOfProfiledLikelihoodsTestStat class. I attached the source code I am using along with the different outputs I get. I am compiling the cpp file with g++ and the appropriate root/roofit flags.
Again, I observed different results between Root v5.27 and 5.32. Results from 5.27 sounds ok, but with 5.32 I obtain a CLs higher than 1. To make sure that my previous environment setup isn’t in cause, I also compiled a clean version of Root v5.34 this time. With it, I obtained results similar to 5.32 with CLs higher than 1.
I’m still not sure if this is a bug or a feature. According to [1], one is supposed to calculate the CL_b and CL_s+b using the following formula :
CL_b = P_b(Q < Q_obs)
CL_s+b = P_s+b(Q < Q_obs)
However in my results, everything looks like CL_b is taken equal to P_b(Q > Q_obs) in root version >= 5.32 ?
Am I completely missing something :s ?
Cheers,
Alex
[1] A.L.Read, Modified frequentist analysis of search results (The CLs method) myHybridCalculatorTest.zip (9.49 KB)
After some more investigation, I finally understood that my issue must be linked to the fPValueIsRightTail boolean of HypoTestResult. The default is to consider the integral on the right tail to compute the CLb and CLsb.
However, according to the plot generated by the tutorial (HybridInstructional), the quantity computed for ProfileLikelihoodRatio and SimpleLikelihoodRatio is ln(L_1 / L_0), not -2ln(L_1 / L_0) as I expected. Therefore, the p-value should correspond to the left tail to be consistent with A.L.Read CL’s definition.
I solved my problem using
r1->SetPValueIsRightTail(false);
… thought I still don’t understand why the default is “true” ?
[EDIT : Also I couldn’t find in the code why 5.27 gives me different results from 5.34. Maybe I would with more time.]
The way CLs is defined in the HypoTestResult class changed between 5.28 and 5.32 so if you are using with the same code is normal you get different result.
In 5.32 is now more consistent, but you need to use the right null and alternate hypothesis.
If you want to computes the p0, discovery significance, i.e. the p-value for the background hypothesis you need background model. You need also to set in the HypoTestResult class the flag
HypoTestResult::SetBackgroundAsAlt(false);
If you want to compute the p-values for a limit (e.g. CLs) the Null model is now the signal + background model and the alt the backgorund model. You need to set in this case the flag HypoTestResult::SetBackgroundAsAlt(true);
I don’t think you need to set the flag SetPValueIsRightTail if you are using the standard test statistics