I have been using TMVA for evaluating different MVA techniques on a dataset, and when I plotted the ROC curve for Cuts (found by Simulated Annealing), I found something that I thought was rather odd.

Instead of a monotonically decreasing curve like the other MVA techniques, I found that there were random spikes in the curve, which I could not explain. Does anyone have an explanation of why this is.

Here is the plot of various MVA techniques which demonstrates the oddities of the cuts ROC curve:

roc-curves.pdf (16.1 KB)

Hi, sorry for the late answer. Did you find out the answer to your question?

How did you generate the ROC curve?

Can you also describe a bit more how the optimal cuts (simulated annealing) works; How is the underlying classifier generated?

These kind of fluctuations can occur when you have a low number of signal events and draw the ROC using cubic interpolation or similar, but it seems that isn’t applicable here. The only other thing I can think of without further details is if the simulated annealing regenerates the underlying classifier for each cut.

Hi,

No, I never found the answer to my question. It is a combination of the ROC curves that are generated by TMVA when using the TMVA::Factory class.

I think that the simulated annealing would regenerate the classifier for each cut within TMVA, but I’m not sure; that would explain it somewhat though!

Would it be possible for you to share a minimally working example for the simulated annealing case? Or at least a schematic description of it. I can then have a look when I have some free time.