Memory exhausted: reading GetEntries

ROOT
1

Hello all,

I have a root file of ~6GB in castor and I am trying my code on it.
For many events, it runs fine. I am also printing the value returned by
GetEntries() (“nb = …”)
For the first few events, it prints fine. But then it starts printing -1
and ultimately my program crashes at “this line in the code”:

for(int jentry=0; jentry<nentries; jentry++) { nb = fChain->GetEntries(jentry) /////// "this line in the code" ....... }

printouts:

====NEW EVENT===== entered into the loop nb = 13118 event no. = 31 ====NEW EVENT===== entered into the loop nb = 15105 event no. = 32 ====NEW EVENT===== entered into the loop nb = 22041 event no. = 33 ====NEW EVENT===== entered into the loop nb = 17495 event no. = 34 ====NEW EVENT===== entered into the loop nb = 15700 event no. = 35 ====NEW EVENT===== entered into the loop nb = 13613 event no. = 36 ====NEW EVENT===== entered into the loop nb = 15839 event no. = 37 ====NEW EVENT===== ....................... entered into the loop nb = -1 event no. = 395198 ====NEW EVENT===== entered into the loop nb = -1 event no. = 395199 ====NEW EVENT===== entered into the loop nb = -1 event no. = 395200 ====NEW EVENT===== entered into the loop Fatal in <operator new>: storage exhausted (failed to allocate 1186257581 bytes) aborting Attaching to program: /proc/22153/exe, process 22153 [Thread debugging using libthread_db enabled] 0xffffe410 in __kernel_vsyscall () #1 0x00549713 in __waitpid_nocancel () from /lib/libc.so.6 #2 0x004ee07b in do_system () from /lib/libc.so.6 #3 0x00652ead in system () from /lib/libpthread.so.0 #4 0x56ff6b0d in TUnixSystem::Exec(char const*) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libCore.so #5 0x56ffbfab in TUnixSystem::StackTrace() () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libCore.so #6 0x56f28a16 in DefaultErrorHandler(int, bool, char const*, char const*) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libCore.so #7 0x56f28662 in ErrorHandler () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libCore.so #8 0x56f28733 in Fatal(char const*, char const*, ...) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libCore.so #9 0x555597cc in operator new(unsigned int) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libNew.so #10 0x55559c2d in operator new[](unsigned int) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libNew.so #11 0x555d9f98 in TBasket::Streamer(TBuffer&) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libTree.so #12 0x555da75d in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libTree.so #13 0x555dd6d2 in TBranch::GetBasket(int) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libTree.so #14 0x555ddee5 in TBranch::GetEntry(long long, int) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libTree.so #15 0x5562a94c in TTree::GetEntry(long long, int) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libTree.so #16 0x556005f2 in TChain::GetEntry(long long, int) () from /afs/cern.ch/cms/slc5_ia32_gcc434/cms/cmssw/CMSSW_3_8_3/external/slc5_ia32_gcc434/lib/libTree.so #17 0x08053ea2 in myPlot::Loop() () #18 0x0805bd2c in main () A debugging session is active.

is there a way to handle this?
Do I need to call setAutodelete(kTrue)?

Thanks for any help,
Shilpi

2

[quote]Fatal in : storage exhausted (failed to allocate 1 186 257 581 bytes)[/quote]This attempting to allocate 1Gb in one go which seems odd. I would suspect that the problem is that the file is corrupted. Try download the file to a local disk and see if the problem persist. (Alternatively try a different file and see if the problem is there too).

Cheers,
Philippe.

3

Hi Philippe,

Thanks for your reply.
I made smaller chunks of the same file and it worked.
could it be due to size?

Thanks,
Shilpi

4

also, if i copy that bigger file locally, the problem is gone.

Thanks,
Shilpi

5

[quote]also, if i copy that bigger file locally, the problem is gone.[/quote]So the problem is likely caused by an older version castor client library that was under some circumstances not sending the actual content of the file.

Cheers,
Philippe.

6

But it is happening to many files which are are very large in size.
I am having to copy all those files locally. Is there a way to make that
castor client library send the actual information of the file.
sorry, I am not much informed about it, but is there some new version
of castor client library?

Thanks,
Shilpi

7

[quote]sorry, I am not much informed about it, but is there some new version
of castor client library?[/quote]Yes, a priori the problem was fixed several months ago. So you should check that you are using the most recent castor client.

Philippe.

8

Hello Philippe,

apologies for my ignorance.
how do I check that and how do I update to the most recent one?

Thanks & Regards,
Shilpi

9

Hi,

Unfortunately, I do not know, you would need to ask the developer of castor or your experiment support for this information.

Philippe.

10

Thanks Philippe for helping me tracking down the
problem. I will then try to update my castor client
version to the latest.

Thanks & Regards,
shilpi