Using a larger sample size, TMVA appears to crash at TMVA:: Factory::TestAllMethods()

Hello,

The following macro
trainKLCoeffsWithImpulsivityAndLinPolFracResponsesSemiweighted.C (17.6 KB)
ran to completion when I ran it, but when I scaled it up to a data set that was 10x larger using
trainKLCoeffsFullResponsesSemiweighted.C (12.2 KB)
it appeared to crash at the TMVA:: Factory ::TestAllMethods() step, after the first training. The output I got was
trainKLCoeffsFullResponsesSemiweighted0.txt (31.8 KB)
My first suspicion is that this could be related to the TMVA::Factory objects being dynamically allocated as opposed to statically, so I have modified the above macro in the following way
trainKLCoeffsFullResponsesSemiweightedStatic.C (12.4 KB)

I am waiting to see if this resolves the issue. I will report back whether or not this resolves it, but any feedback before or after the macro ends would be appreciated.

So it crashed at the same area as before.

I am now looking into statically allocating DataLoader objects, using DataLoaderCopy() instead of MakeCopy(). Below is the updated version.

trainKLCoeffsFullResponsesSemiweightedStatic.C (12.2 KB)

I will update as to whether or not this resolves the issue.

Unfortunately, it did not.
trainKLCoeffsFullResponsesSemiweightedStatic.txt (31.8 KB)

Hi,

Can you share also your ROOT files so one can run the macro and reproduce the problem.
Probably you can try run it with the minimal amount of data needed

Lorenzo

Hi Lorenzo,

Thank you for the reply. The files used are all rather large, ranging on the order of tens of MB to tens of GB. I am able to run the macro with files corresponding to decimated samples of these larger files. This is something I meant to suggest through this post’s title.

As not everything is being used in these files, it makes me think that a possible workaround could be to create files which copy over only the variables being used?

What I will try and do here is copy over and upload one of the smaller decimated files, on the off chance that the problem could be due to file structure.

KLCoeffsOriginal-decimated-HiCal2A-part_all_1.root (369.2 KB)

Unfortunately, the other type of file from which variables are pulled all appear to be too large to upload here, the smallest of these decimated files appears to be 4.2 MB.

If structure isn’t the problem, might I ask for a recommendation on how I can share the files in question? Files containing one set of variables total 34 G, while files containing the other set of variables total 642 G. The corresponding decimated versions of these file total 7.8 G and 119 G, respectively.

Hi,
I will check the structure of the file. Thank you for uploading this smaller one.

For sharing the large file, if you have a CERN account you could use cernbox, there a user should have a space of around 1TB.

I see also the crash is happening in the function TTreeFormula::LoadCurrentDim() that is called by
TTreeFormulaManager::GetNdata(bool) .
Maybe @pcanal is having an idea what could be the problem

Lorenzo

1 Like

Hi Lorenzo,

Thank you for your reply. I don’t have a CERN account, but perhaps I could get one? How would I go about doing that?

I don’t know why I didn’t think of this before, but if file structure is the culprit, I can just attach here two even smaller sample files here.

KLCoeffsOriginal-decimated-HiCal2A-part_88.root (25.3 KB)
summary-decimated-HiCal2A-part_88-sinsub_10_3_ad_2.root (373.6 KB)

The first of the files contains the first few entries of the file I already attached earlier. The other, larger, file contains other variables corresponding to these entries. Opening this other file, ROOT may complain about custom classes, but the file should be readable. If not, it should become readable after git cloneing the repository https://github.com/anitaNeutrino/anitaBuildTool, and then running ./buildAnita.sh from within the anitaBuildTool directory.

Best,
John

Thank you John, I will try with those file.
For the CERN account, if you don’t have an association or collaboration with CERN, then it is unfortunatly not possible to get it. And I guess sharing tools like Google drive or dropbox don’t allow free sharing of such large files.
I hope we can solve the problem looking at the small files.
Best,

Lorenzo

HI,
Can you also post a version of the TMVA macro working with the two small sample file ?
Thank you
Lorenzo