Who to believe, ROC-integ or ROC curve?

TMVA
1

Hi all,
sometimes when I train several methods the ROC-integ presented at the end of the training clearly do not respect the ROC curves accessed through TMVAGui. See for example:

 Name:         Method:          ROC-integ
 dataset       BDT_Tracker2   : 0.566
 dataset       BDT_Tracker    : 0.855

Selection_138
Selection_138.png1203×500 47.3 KB

Selection_139
Selection_139.png1035×764 31.9 KB

Selection_140
Selection_140.png1204×500 50 KB

The ROC Curve of BDT_Tracker2 clearly show a better classificator but the ROC-integ displayed report a measly 0.566.

My question is: what is more accurate and less prone to be an error? My bet is on the curve but I want your opinion as TMVA expert.

Thanks for your time,

Alberto

2

HI,

Which version of ROOT are you using ? I think there was a bug with the ROC integral calculation that should have been fixed in the current master version.
If you can check with the latest version would be great, otherwise, you can send me the output file and I can double-check the result.
But it is clear that the result of the curve in this case seems to be correct

Best Regards

Lorenzo

3

Hi,
I’m using ROOT 6.10/08.
The bug is only in the integral calculation or even in the roc curve?

Regards,
Alberto

4

Hi,

I think the bug might be in both. Could you try to use 6.11.02 or wait for the new 6.12 release, that will be done later this week ?
Otherwise if you post your macro and data I can try it

Best

Lorenzo

5

I’ll wait for 6.12, thanks.

Alberto