ROC AUC discrepancy

TMVA
1

Hi forum!

I have a dilemma, so any help would be appreciated! When I train the DNN with TMVA (PyKeras method), I see the ROC-integ value on the terminal once the training ends. And recently, I also learned how to extract the ROC-Curve histogram from the root-file which TMVA creates after training. And for comparison, I take the integral from this ROC-curve histogram, and the results are not the same with the one which I see on the terminal.

Which result should I take as a truth? Any suggestions?

Cheers,

2

I think @moneta can help

3

Hi,

How are you getting the ROC integral afterwards ?
What is printed after training when using Factory::EvaluateAllMethod is obtained from the full output data results. You can verify it using the function TMVA::Factory::GetRocIntegral or to see the curve calling GetROCCurve.

When you extract from the ROOT file, you should be careful to what you use, training or testing data and if you compute from an histogram there could be some interpolation effects that can cause some differences.
If you have very big differences on the same data, then something else might be wrong, in that case please let us know and provide something to reproduce it

Cheers

Lorenzo

4

Hi,

Can you please provide a snapshot of the example how to use these functions?! I tried to invoke them, but have not succeeded. To be specific, I invoked GetRocCurve() function before EvaluateAllMethods() method, and GetRocIntegral after.

Regarding the question how I obtain the ROC AUC from the histogram TMVA saves, please find my code in the attachment.

Cheers,
TMVAplot_RocCurve.C (1.6 KB)

5

Hi,

See at the end of some TMVA tutorials, such as this one, where the ROC curve are displayed:
https://root.cern.ch/doc/master/TMVA__Higgs__Classification_8C.html

I see in your code that you compute from some the histograms and this can cause some differences compare to the other case where there are computed using the full data points.

Cheers

Lorenzo

6

Hi,

Thank you for clairification! I can get now the ROC-Curve plot, however, the method GetRocIntegral does not still work. The error message is as follows:

AttributeError: ‘Factory’ object has no attribute ‘GetRocIntegral’

I’m running my project on lxplus and I use certain shell script to set the environmental variables. My guess is maybe I should use some other shell to set the latest version of TMVA ???

I just would like to confirm the true value of ROC AUC.

Cheers,

7

Hi,
This is strange, which TMVA version are you using ?
Normally you would need to add a line like this (this works for the TMVA_Higgs_Classification.C tutorial):

double auc = factory.GetROCIntegral(loader, "BDT");

Lorenzo

1 Like
8

Hi,

Thank you very much!
I’ve found very trivial mistake in my code! It worked!

Cheers,