I have been using TMVA for evaluating different MVA techniques on a dataset, and when I plotted the ROC curve for Cuts (found by Simulated Annealing), I found something that I thought was rather odd.
Instead of a monotonically decreasing curve like the other MVA techniques, I found that there were random spikes in the curve, which I could not explain. Does anyone have an explanation of why this is.
Here is the plot of various MVA techniques which demonstrates the oddities of the cuts ROC curve:
roc-curves.pdf (16.1 KB)
Hi, sorry for the late answer. Did you find out the answer to your question?
How did you generate the ROC curve?
Can you also describe a bit more how the optimal cuts (simulated annealing) works; How is the underlying classifier generated?
These kind of fluctuations can occur when you have a low number of signal events and draw the ROC using cubic interpolation or similar, but it seems that isn’t applicable here. The only other thing I can think of without further details is if the simulated annealing regenerates the underlying classifier for each cut.
Hi,
No, I never found the answer to my question. It is a combination of the ROC curves that are generated by TMVA when using the TMVA::Factory class.
I think that the simulated annealing would regenerate the classifier for each cut within TMVA, but I’m not sure; that would explain it somewhat though!
Would it be possible for you to share a minimally working example for the simulated annealing case? Or at least a schematic description of it. I can then have a look when I have some free time.