TProfile::Chi2Test is scale invariant

ROOT
1

This is a follow-up to Chi2Test using TProfile , which ended up with [hist] Enable chi2 tests for TProfile by hageboeck · Pull Request #19930 · root-project/root · GitHub enabling computation of chi2 for TProfile.

I think there is another issue. The computation uses the TH1 implementation with WW option. This means that the statistic is exactly invariant under rescaling one of the two profiles.

For two profiles, I believe this is not what users expect.

Example with ROOT 6.40.02:

import ROOT

p1 = ROOT.TProfile("p1", "p1", 3, 0, 3)
p2 = ROOT.TProfile("p2", "p2", 3, 0, 3)

for i, mean in enumerate([10.0, 20.0, 30.0]):
    for _ in range(50):
        p1.Fill(i + 0.5, mean - 1.0)        # bin i of p1: mean, spread 1
        p1.Fill(i + 0.5, mean + 1.0)
        p2.Fill(i + 0.5, 2 * (mean - 1.0))  # bin i of p2: exactly twice, spread 2
        p2.Fill(i + 0.5, 2 * (mean + 1.0))

print("bin        p1                 p2")
for i in range(1, 4):
    print("  %d   %6.2f +- %.4f    %6.2f +- %.4f"
          % (i, p1.GetBinContent(i), p1.GetBinError(i),
             p2.GetBinContent(i), p2.GetBinError(i)))

print()
print("Chi2Test(p2, 'CHI2') = %.6g" % p1.Chi2Test(p2, "CHI2"))
print("Chi2Test(p2)         = %.6g" % p1.Chi2Test(p2))

chi2 = sum((p1.GetBinContent(i) - p2.GetBinContent(i)) ** 2
           / (p1.GetBinError(i) ** 2 + p2.GetBinError(i) ** 2)
           for i in range(1, 4))
print()
print("bin by bin, sum (y1-y2)^2/(e1^2+e2^2) = %.6g for 3 d.o.f. -> p = %.3g"
      % (chi2, ROOT.TMath.Prob(chi2, 3)))

Output

bin        p1                 p2
  1    10.00 +- 0.1000     20.00 +- 0.2000
  2    20.00 +- 0.1000     40.00 +- 0.2000
  3    30.00 +- 0.1000     60.00 +- 0.2000

Chi2Test(p2, 'CHI2') = 0
Chi2Test(p2)         = 1

bin by bin, sum (y1-y2)^2/(e1^2+e2^2) = 28000 for 3 d.o.f. -> p = 0
2

@StephanH maybe again for you?

4

Hello @wiso,

sorry for picking this up only now. I went back to the papers describing WW Chi^2 tests, and indeed, the test statistic is defined as
image

Where W is the sum of weights of the full histogram and w are the weights per bin. If you set histo_2 = 2*histo_1, the numerator becomes:
W_1*2*w_1i - 2*W_1 * w_1i which is always zero.

How we choose or treat the errors doesn’t even matter. This results from the fact that the paper arXiv:physics/0605123 (2006) defines compatibility as having the same relative probabilities in each bin, independent of an overall scale factor.

Long story short, we can use the test to check if the (normalised) distributions are compatible, but it doesn’t test for scale differences. Now the question is if that’s acceptable?
We could extend the documentation and maybe even add a warning if scale differences are detected.

Update: But the more I read the docs, the clearer it gets that scale invariance is a desired property of the test. We should probably write this down explicitly, but then we still have to decide what to do about the case of profile-profile comparison.

6

Hello @StephanH , thank you for your reply. I am not sure to understand what you mean:

we can use the test to check if the (normalised) distributions are compatible

we are not comparing distributions (histograms), we are comparing TProfiles. Have a look at my example in my original post: two completely different TProfiles give a fully compatible p-value.

For profile-profile I would either compute the comparison of the means — χ² = Σ (y₁−y₂)²/(e₁²+e₂²), ndf = bins with a positive error on both sides