Overtraining test obtained from TMVA

[JungChang]

Dear Sir,

I have a question about the KS test for overtraining check.
From this topic, it seems the KS test number should be close to 0.5 to avoid overtraining.

About the BDT control plot and the decision boundary

Dear Kim, Oh I just saw them in the user guide and the TMVA GUI, then tried to interpret them to help us understand what’s happening in the training. But it seems not that easy… Thanks for the explanation. And BTW I have another question on the Kolmogorov-Smirnov test(to estimate the overfitting) and hope you can help me with the print out: Testing efficiency compared to training efficiency (overtraining check) — Perform Kolmogorov-Smirnov tests — Goodness of signal (background) consisten…

In my case the KS test for signal and background is always smaller than 0.2. See attached picture.
I trained with BDT. May I ask how can I avoid the overtraining? Through the understanding of KS test? Thanks!

Best,
Jung

[kialbert]

Hi Jung!

Indeed the KS-test should return 0.5 for identically distributed data. To my understanding however the KS-test is rather weak in that it requires a large sample size to be effective.

In my quick test of 2 slightly separated normal distributions for signal and background using a shallow BDT with 800 trees the KS-test started being informative around a sample size of 100000.

To reduce overtraining, use either more data or regularise your BDT with e.g. simpler trees (depth=1 or 2), large MinNodeSize, and large nCuts. (As I assume you understand already looking at your table.)

You can also look into bagged boosting (UseBaggedBoost and BaggedSampleFraction) and feature subsampling with UseRandomisedTrees and UseNvars.

Cheers,
Kim

[JungChang]

Dear Kim,

Thanks a lot for the reply! I got it.

Best,
Jung