Size limit in t->BuildIndex() function?

rmadar · November 16, 2020, 10:59am

Hello,

I am trying to match entries between two trees (reco-level & truth-level), based on a run number and an event number. I am currently using the following logic:

// Before the event loop
if (tTruth->BuildIndex("runN", "evtN") < 0) {
 cout << "Issue with index building on Truth TTree" << endl; 
}

// Event loop over the nR reco-level events
for (int iR=0 ; iR<nR ; iR++) {              
                                                                              
    // Load reco values 
    tReco->GetEntry(iR);

    // Get the truth-level event based on run/event number
    tTruth->GetEntryWithIndex(eRreco->runN, eRreco->evtN)

    // Do stuff ...
}

This woks quite nicely but when the truth tree is large (typically above 10 million events), some random segmentation fault appear when building the indices before the event loop, without any message. Here is an example of the randomness of the situation: if I run over file A (4 millions events) and then file B (10 million events), it works. But if I run over file B first, it crashes.

Therefore I wanted to know if there are some know/possible reasons to have have such a limitation of the BuildIndex() function for large TTree's or if I have some bad hidden in my code.

It’s a bit difficult to prepare a “minimal” working example, given the nature of the problem, but let me know if any additional information can be useful.

Thanks for you help!
Romain

ROOT Version: v6.20.06
Platform: centos7
Compiler: gcc8
cvmfs path: /cvmfs/sft.cern.ch/lcg/releases/LCG_97a/ROOT/v6.20.06/x86_64-centos7-gcc8-opt/bin/root

jblomer · November 16, 2020, 1:30pm

It is strange to get a segfault but no stack trace (I understand this correctly?). Can you try if we get a stacktrace when running from /cvmfs/sft.cern.ch/lcg/releases/LCG_97a/ROOT/v6.20.06/x86_64-centos7-gcc8-dbg? @pcanal Do you have an idea how to debug it?

rmadar · November 16, 2020, 1:53pm

Thanks for your reply! Indeed, I don’t get any stack trace:

Trying with /cvmfs/sft.cern.ch/lcg/releases/LCG_97a/ROOT/v6.20.06/x86_64-centos7-gcc8-dbg I get, at least, an explicit error:

root.exe: /workspace/build/projects/ROOT-v6.20.06/src/ROOT/v6.20.06/tree/tree/src/TBasket.cxx:937: virtual void TBasket::Streamer(TBuffer&): Assertion `flag <= 40' failed.

Let me know which further test I can make or if the full source code would be useful. All data would be like 4Go, so I guess I can put them somewhere on cernbox if that helps?

Cheers,
Romain

pcanal · November 16, 2020, 2:33pm

Can you run the failing example under valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp ?

rmadar · November 16, 2020, 3:00pm

Here is the result of the execution of the failing example within valrgind:

[madar@clratlserv01]:match-trees>valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp /cvmfs/sft.cern.ch/lcg/releases/LCG_97a/ROOT/v6.20.06/x86_64-centos7-gcc8-dbg/bin/root
==257341== Memcheck, a memory error detector
==257341== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==257341== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==257341== Command: /cvmfs/sft.cern.ch/lcg/releases/LCG_97a/ROOT/v6.20.06/x86_64-centos7-gcc8-dbg/bin/root
==257341== 
   ------------------------------------------------------------------
  | Welcome to ROOT 6.20/06                        https://root.cern |
  | (c) 1995-2020, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Jun 10 2020, 06:10:57                 |
  | From tags/v6-20-06@v6-20-06                                      |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

root [0] .L createMatchedTree.C+
root [1] createMatchedTree("/AtlasDisk/user/madar/data/spincorr/ntuples/v01/ana_v0/ttbar13.root", "/AtlasDisk/user/madar/data/spincorr/ntuples/v01/ana_v0_truth/ttbar13.root", "test.root")

Matching will be done:
 -> for 1287221 reco events
 -> in  10492000 truth events
root.exe: /workspace/build/projects/ROOT-v6.20.06/src/ROOT/v6.20.06/tree/tree/src/TBasket.cxx:937: virtual void TBasket::Streamer(TBuffer&): Assertion `flag <= 40' failed.
==257341== 
==257341== HEAP SUMMARY:
==257341==     in use at exit: 85 bytes in 1 blocks
==257341==   total heap usage: 4 allocs, 3 frees, 72,959 bytes allocated
==257341== 
==257341== LEAK SUMMARY:
==257341==    definitely lost: 0 bytes in 0 blocks
==257341==    indirectly lost: 0 bytes in 0 blocks
==257341==      possibly lost: 0 bytes in 0 blocks
==257341==    still reachable: 85 bytes in 1 blocks
==257341==         suppressed: 0 bytes in 0 blocks
==257341== Rerun with --leak-check=full to see details of leaked memory
==257341== 
==257341== For lists of detected and suppressed errors, rerun with: -s
==257341== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Cheers,
Romain

sbinet · November 16, 2020, 3:20pm

hi,

it might be noteworthy to say here that one (both?) of the trees has been produced with groot

looking at what TTree::BuildIndex does, I believe it shouldn’t really matter (ie: I think groot produces the required informations for TTreePlayer and TTreeIndex to play well), but that could be valuable information nonetheless (to be sure “impossible” hypotheses are taken into account as well)

it could be useful to produce a slimmed down version (couple of events) of that rtree.Tree for others to inspect (if all is are dotted and ts crossed).

another possible thing to try out would be to read+copy the truth-tree with ROOT (sans using the optimized rootcp program), use that one instead, and see what gives.

cheers,
-s

pcanal · November 16, 2020, 3:47pm

I was not precise enough. In order for valgrind to give useful information you need to run it on the executable name root.exe rather than just root. I.e.

valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp /cvmfs/sft.cern.ch/lcg/releases/LCG_97a/ROOT/v6.20.06/x86_64-centos7-gcc8-dbg/bin/root.exe

rmadar · November 16, 2020, 4:02pm

@pcanal, I have no idea what’s important and what is not, so I put everything of the output (starting after the execution of the function in ROOT):

 root [0] .L createMatchedTree.C+
root [1] createMatchedTree("/AtlasDisk/user/madar/data/spincorr/ntuples/v01/ana_v0/ttbar13.root", "/AtlasDisk/user/madar/data/spincorr/ntuples/v01/ana_v0_truth/ttbar13.root", "test.root")

Matching will be done:
 -> for 1287221 reco events
 -> in  10492000 truth events
==262639== Warning: set address range perms: large range [0x59eaf040, 0x96eaf144) (undefined)
==262639== Warning: set address range perms: large range [0x3ee32040, 0x52092140) (undefined)
==262639== Warning: set address range perms: large range [0x96eb0040, 0x100110140) (undefined)
==262639== Warning: set address range perms: large range [0x100111040, 0x13d111144) (undefined)
==262639== Warning: set address range perms: large range [0x13d112040, 0x1ba372140) (undefined)
==262639== Warning: set address range perms: large range [0x1ba373040, 0x20d5d3140) (undefined)
==262639== Warning: set address range perms: large range [0x20ec68040, 0x24bc68144) (undefined)
==262639== Warning: set address range perms: large range [0x24e991040, 0x261c01140) (undefined)
==262639== Warning: set address range perms: large range [0x263296040, 0x2a0296144) (undefined)
==262639== Warning: set address range perms: large range [0x2a4653040, 0x2e1653144) (undefined)
==262639== Warning: set address range perms: large range [0x2e437c040, 0x3045ec140) (undefined)
==262639== Warning: set address range perms: large range [0x305c81040, 0x342c81144) (undefined)
==262639== Warning: set address range perms: large range [0x344316040, 0x391586140) (undefined)
==262639== Warning: set address range perms: large range [0x3942af040, 0x3d12af144) (undefined)
==262639== Warning: set address range perms: large range [0x3d2944040, 0x3ffbc4140) (undefined)
==262639== Warning: set address range perms: large range [0x4028ed040, 0x43f8ed144) (undefined)
==262639== Warning: set address range perms: large range [0x442616040, 0x47a886140) (undefined)
==262639== Warning: set address range perms: large range [0x47bf1b040, 0x4b8f1b144) (undefined)
==262639== Warning: set address range perms: large range [0x4bd2d8040, 0x4fa2d8144) (undefined)
==262639== Warning: set address range perms: large range [0x4fd001040, 0x53a271140) (undefined)
==262639== Warning: set address range perms: large range [0x53b906040, 0x578906144) (undefined)
==262639== Warning: set address range perms: large range [0x579f9b040, 0x5b41fb140) (undefined)
==262639== Warning: set address range perms: large range [0x5b5890040, 0x62fb00140) (undefined)
==262639== Warning: set address range perms: large range [0x631195040, 0x66e195144) (undefined)
==262639== Warning: set address range perms: large range [0x66f82a040, 0x6e3a8a140) (undefined)
==262639== Warning: set address range perms: large range [0x6e67b3040, 0x7237b3144) (undefined)
==262639== Warning: set address range perms: large range [0x724e48040, 0x7930a8140) (undefined)
==262639== Warning: set address range perms: large range [0x795dd1040, 0x7d2dd1144) (undefined)
==262639== Warning: set address range perms: large range [0x7d4466040, 0x82d6d6140) (undefined)
==262639== Warning: set address range perms: large range [0x8303ff040, 0x86d3ff144) (undefined)
==262639== Warning: set address range perms: large range [0x8717bc040, 0x8ae7bc144) (undefined)
==262639== Warning: set address range perms: large range [0x8afe51040, 0x8c60c1140) (undefined)
==262639== Warning: set address range perms: large range [0x8c7756040, 0x91b9c6140) (undefined)
==262639== Warning: set address range perms: large range [0x91d05b040, 0x95a05b144) (undefined)
==262639== Warning: set address range perms: large range [0x95e418040, 0x99b418144) (undefined)
==262639== Warning: set address range perms: large range [0x99caad040, 0x9eed0d140) (undefined)
==262639== Warning: set address range perms: large range [0x9f7ca7040, 0xa34ca7144) (undefined)
==262639== Warning: set address range perms: large range [0xa3e2d5040, 0xa7b2d5144) (undefined)
==262639== Warning: set address range perms: large range [0xa7c96a040, 0xabcbca140) (undefined)
==262639== Warning: set address range perms: large range [0xac1b64040, 0xafeb64144) (undefined)
==262639== Warning: set address range perms: large range [0xb001f9040, 0xb57459140) (undefined)
==262639== Warning: set address range perms: large range [0xb58aee040, 0xb88d5e140) (undefined)
==262639== Warning: set address range perms: large range [0xb8a3f3040, 0xbc73f3144) (undefined)
==262639== Warning: set address range perms: large range [0xbc8a88040, 0xbdbcf8140) (undefined)
==262639== Warning: set address range perms: large range [0xbdd38d040, 0xc055fd140) (undefined)
==262639== Warning: set address range perms: large range [0xc06c92040, 0xc43c92144) (undefined)
==262639== Warning: set address range perms: large range [0xc4804f040, 0xc8504f144) (undefined)
==262639== Warning: set address range perms: large range [0xc866e4040, 0xcd1944140) (undefined)
==262639== Warning: set address range perms: large range [0xcd466d040, 0xd1166d144) (undefined)
==262639== Warning: set address range perms: large range [0xd12d02040, 0xd76f62140) (undefined)
==262639== Warning: set address range perms: large range [0xd79c8b040, 0xdb6c8b144) (undefined)
==262639== Warning: set address range perms: large range [0xdbb048040, 0xdf8048144) (undefined)
==262639== Warning: set address range perms: large range [0xdfc405040, 0xe39405144) (undefined)
==262639== Warning: set address range perms: large range [0xe3aa9a040, 0xea1cfa140) (undefined)
==262639== Warning: set address range perms: large range [0xea4a23040, 0xee1a23144) (undefined)
==262639== Warning: set address range perms: large range [0xee474c040, 0xf5f9bc140) (undefined)
==262639== Warning: set address range perms: large range [0xf61051040, 0xf9e051144) (undefined)
==262639== Warning: set address range perms: large range [0xfa240e040, 0xfdf40e144) (undefined)
==262639== Warning: set address range perms: large range [0x1016f08040, 0x1056188140) (undefined)
==262639== Warning: set address range perms: large range [0x10561a1040, 0x10b7401140) (undefined)
==262639== Warning: set address range perms: large range [0x10b740e040, 0x10f440e144) (undefined)
==262639== Warning: set address range perms: large range [0x10f442b040, 0x111a68b140) (undefined)
==262639== Warning: set address range perms: large range [0x111a69c040, 0x115769c144) (undefined)
==262639== Warning: set address range perms: large range [0x11576c1040, 0x11bc921140) (undefined)
==262639== Warning: set address range perms: large range [0x11bc92e040, 0x11f992e144) (undefined)
==262639== Warning: set address range perms: large range [0x11f9953040, 0x1236953144) (undefined)
==262639== Warning: set address range perms: large range [0x1236960040, 0x1299bc0140) (undefined)
==262639== Warning: set address range perms: large range [0x1299be1040, 0x12d6be1144) (undefined)
==262639== Warning: set address range perms: large range [0x12d6c06040, 0x1313c06144) (undefined)
==262639== Warning: set address range perms: large range [0x1313c13040, 0x1326e83140) (undefined)
==262639== Warning: set address range perms: large range [0x1326ea0040, 0x1363ea0144) (undefined)
==262639== Warning: set address range perms: large range [0x1363ebd040, 0x13d012d140) (undefined)
==262639== Warning: set address range perms: large range [0x13d013e040, 0x140d13e144) (undefined)
==262639== Warning: set address range perms: large range [0x140fe8f040, 0x143b0ff140) (undefined)
==262639== Warning: set address range perms: large range [0x143c7a8040, 0x14797a8144) (undefined)
==262639== Warning: set address range perms: large range [0x147c4f5040, 0x149a765140) (undefined)
==262639== Warning: set address range perms: large range [0x149be06040, 0x14d8e06144) (undefined)
==262639== Warning: set address range perms: large range [0x14dd1ef040, 0x151a1ef144) (undefined)
==262639== Warning: set address range perms: large range [0x151b894040, 0x1544af4140) (undefined)
==262639== Warning: set address range perms: large range [0x154783d040, 0x158483d144) (undefined)
==262639== Warning: set address range perms: large range [0x1585ee2040, 0x15fe142140) (undefined)
==262639== Warning: set address range perms: large range [0x15ff7eb040, 0x1625a4b140) (undefined)
==262639== Warning: set address range perms: large range [0x16270f4040, 0x16640f4144) (undefined)
==262639== Warning: set address range perms: large range [0x1665795040, 0x16d69f5140) (undefined)
==262639== Warning: set address range perms: large range [0x16d9736040, 0x1716736144) (undefined)
==262639== Warning: set address range perms: large range [0x171ab17040, 0x1757b17144) (undefined)
==262639== Warning: set address range perms: large range [0x175a864040, 0x17c1ac4140) (undefined)
==262639== Warning: set address range perms: large range [0x17c3165040, 0x1800165144) (undefined)
==262639== Warning: set address range perms: large range [0x180454a040, 0x184154a144) (undefined)
==262639== Warning: set address range perms: large range [0x1845933040, 0x1882933144) (undefined)
==262639== Warning: set address range perms: large range [0x1892f8d040, 0x18cff8d144) (undefined)
==262639== Warning: set address range perms: large range [0x18d4376040, 0x1911376144) (undefined)
==262639== Warning: set address range perms: large range [0x1912a17040, 0x1953c87140) (undefined)
==262639== Warning: set address range perms: large range [0x19569cc040, 0x19939cc144) (undefined)
==262639== Warning: set address range perms: large range [0x1997db9040, 0x19d4db9144) (undefined)
==262639== Warning: set address range perms: large range [0x19d645e040, 0x1a156be140) (undefined)
==262639== Warning: set address range perms: large range [0x1a16d6b040, 0x1a80fcb140) (undefined)
==262639== Warning: set address range perms: large range [0x1a82674040, 0x1abf674144) (undefined)
==262639== Warning: set address range perms: large range [0x1ac23b9040, 0x1b3b619140) (undefined)
==262639== Warning: set address range perms: large range [0x1b3ccc2040, 0x1b79cc2144) (undefined)
==262639== Warning: set address range perms: large range [0x1b8b324040, 0x1bc8324144) (undefined)
==262639== Warning: set address range perms: large range [0x1bcc709040, 0x1c09709144) (undefined)
==262639== Warning: set address range perms: large range [0x1c0adae040, 0x1c8700e140) (undefined)
==262639== Warning: set address range perms: large range [0x1c89d5b040, 0x1cc6d5b144) (undefined)
==262639== Warning: set address range perms: large range [0x1ccb14c040, 0x1d0814c144) (undefined)
==262639== Warning: set address range perms: large range [0x1d0c535040, 0x1d49535144) (undefined)
==262639== Warning: set address range perms: large range [0x1d4c27a040, 0x1da84da140) (undefined)
==262639== Warning: set address range perms: large range [0x1da9b83040, 0x1de6b83144) (undefined)
==262639== Warning: set address range perms: large range [0x1de8224040, 0x1e64484140) (undefined)
==262639== Warning: set address range perms: large range [0x1e671cd040, 0x1ea41cd144) (undefined)
==262639== Warning: set address range perms: large range [0x1ea5872040, 0x1ec8ad2140) (undefined)
==262639== Warning: set address range perms: large range [0x1ecb81b040, 0x1f0881b144) (undefined)
==262639== Warning: set address range perms: large range [0x1f210d2040, 0x1f5e0d2144) (undefined)
==262639== Warning: set address range perms: large range [0x1f5f773040, 0x1f9c9d3140) (undefined)
==262639== Warning: set address range perms: large range [0x1f9e07c040, 0x1fea2dc140) (undefined)
**262639** new/new[] failed and should throw an exception, but Valgrind
**262639**    cannot throw exceptions and so is aborting instead.  Sorry.
==262639==    at 0x4C29C5C: VALGRIND_PRINTF_BACKTRACE (valgrind.h:6314)
==262639==    by 0x4C2AC83: operator new[](unsigned long) (vg_replace_malloc.c:433)
==262639==    by 0x52182A4: TBuffer::TBuffer(TBuffer::EMode, int) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libCore.so)
==262639==    by 0x7776418: TBufferIO::TBufferIO(TBuffer::EMode, int) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libRIO.so)
==262639==    by 0x776FF78: TBufferFile::TBufferFile(TBuffer::EMode, int) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libRIO.so)
==262639==    by 0x208529E4: TBasket::Streamer(TBuffer&) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so)
==262639==    by 0x208516DF: TBasket::ReadBasketBuffers(long long, int, TFile*) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so)
==262639==    by 0x2085CA1E: TBranch::GetEntry(long long, int) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so)
==262639==    by 0x2480B471: long double TTreeFormula::EvalInstance<long double>(int, char const**) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTreePlayer.so)
==262639==    by 0x2480F266: TTreeIndex::TTreeIndex(TTree const*, char const*, char const*) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTreePlayer.so)
==262639==    by 0x24823F4B: TTreePlayer::BuildIndex(TTree const*, char const*, char const*) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTreePlayer.so)
==262639==    by 0x208C3187: TTree::BuildIndex(char const*, char const*) (in /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so)
==262639== 
==262639== HEAP SUMMARY:
==262639==     in use at exit: 136,124,294,028 bytes in 509,902 blocks
==262639==   total heap usage: 822,269 allocs, 312,367 frees, 136,706,410,956 bytes allocated
==262639== 
==262639== LEAK SUMMARY:
==262639==    definitely lost: 4,189,734,960 bytes in 193 blocks
==262639==    indirectly lost: 0 bytes in 0 blocks
==262639==      possibly lost: 355,072,864 bytes in 65 blocks
==262639==    still reachable: 131,578,937,409 bytes in 502,152 blocks
==262639==                       of which reachable via heuristic:
==262639==                         stdstring          : 40 bytes in 1 blocks
==262639==                         newarray           : 27,896 bytes in 41 blocks
==262639==                         multipleinheritance: 15,544 bytes in 49 blocks
==262639==         suppressed: 548,795 bytes in 7,492 blocks
==262639== Rerun with --leak-check=full to see details of leaked memory
==262639== 
==262639== For lists of detected and suppressed errors, rerun with: -s
==262639== ERROR SUMMARY: 12062 errors from 6 contexts (suppressed: 3017 from 252)

@sbinet, thanks for mentioning this. Both trees are created with groot. I can produce two trees with a couple of event each (even if the matching will be difficult in that way, but maybe it’s OK with like 100 events - there is one reconstructed events every ~10 truth events). I will also try to use t->CopyTree() if this solve the issue.

pcanal · November 16, 2020, 4:12pm

It looks like somehow the index object is too large and/or the size is misinterpreted. Can you share the file so that @sbinet or I can take a look?

rmadar · November 16, 2020, 4:26pm

Sure, here is the (3.4 Go) file: https://cernbox.cern.ch/index.php/s/0TNG0h5xkkbXuO2 (password: rootforum).

In the meanwhile, does it mean that the results I get when the code doesn’t crash are probably corrupted (meaning the matching might be not correct)? I guess it does …

rmadar · November 17, 2020, 8:33am

Hello @sbinet,

In the meanwhile, I have prepared a shorter version of the file with 100 truth events:
https://cernbox.cern.ch/index.php/s/Mnd6Dtqw02y2wAI (same password as before).

I tried to use newTree = tTruth->CopyTree("") and run my initial code on newTree, but the command of CopyTree() failed with the following back trace (note that it worked well with the small version of tTruth):

root [1] copyTree("/AtlasDisk/user/madar/data/spincorr/ntuples/v01/ana_v0_truth/ttbar13.root", "/AtlasDisk/user/madar/data/spincorr/ntuples/v01/ana_v0_truth/ttbar13_copyTree.root")
Number of entries
10492000

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007fc7ac76446c in waitpid () from /lib64/libc.so.6
#1  0x00007fc7ac6e1f62 in do_system () from /lib64/libc.so.6
#2  0x00007fc7ad811663 in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libCore.so
#3  0x00007fc7ad813eb4 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libCore.so
#4  <signal handler called>
#5  0x00007fc7ab5442a1 in TBufferFile::ReadArray(int*&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libRIO.so
#6  0x00007fc79856f807 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so
#7  0x00007fc79857aa1f in TBranch::GetEntry(long long, int) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so
#8  0x00007fc7985e9448 in TTree::GetEntry(long long, int) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so
#9  0x00007fc79623f8b6 in TTreePlayer::CopyTree(char const*, char const*, long long, long long) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTreePlayer.so
#10 0x00007fc796503b43 in copyTree(TString, TString) () from /AtlasDisk/home2/madar/SM-SpinCorr/general-spincorr-studies/truth-reco/match-trees/createMatchedTree_C.so
#11 0x00007fc7ae08807d in ?? ()
#12 0x0000000000000000 in ?? ()
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007fc7ab5442a1 in TBufferFile::ReadArray(int*&) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libRIO.so
#6  0x00007fc79856f807 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so
#7  0x00007fc79857aa1f in TBranch::GetEntry(long long, int) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so
#8  0x00007fc7985e9448 in TTree::GetEntry(long long, int) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTree.so
#9  0x00007fc79623f8b6 in TTreePlayer::CopyTree(char const*, char const*, long long, long long) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.06-3f7fd/x86_64-centos7-gcc8-opt/lib/libTreePlayer.so
#10 0x00007fc796503b43 in copyTree(TString, TString) () from /AtlasDisk/home2/madar/SM-SpinCorr/general-spincorr-studies/truth-reco/match-trees/createMatchedTree_C.so
#11 0x00007fc7ae08807d in ?? ()
#12 0x0000000000000000 in ?? ()
===========================================================

sbinet · November 17, 2020, 9:22am

couldn’t reproduce your segfault on my machine with the following code:

// +build ignore

#include "TTree.h"
#include "TFile.h"

#include <iostream>

void issue_42318() {
	auto f = TFile::Open("./ttbar13_short.root");
	auto t = (TTree*)f->Get("truth");

	Long64_t evtNbr = 0;
	Int_t  runNbr = 0;

	Int_t d_nlep = 0;
	float *d_lep_pt = nullptr;

	Int_t d_njet = 0;
	float *d_jet_pt = nullptr;

	double truth_kvec[3];

	t->SetBranchAddress("runNumber", &runNbr);
	t->SetBranchAddress("eventNumber", &evtNbr);

	t->SetBranchAddress("d_nlep", &d_nlep);
	t->SetBranchAddress("d_lep_pt", &d_lep_pt);

	t->SetBranchAddress("d_njet", &d_njet);
	t->SetBranchAddress("d_jet_pt", &d_jet_pt);

	t->SetBranchAddress("truth_kvec", &truth_kvec);

	std::cout << "entries: " << t->GetEntries() << "\n";
	for (int i = 0; i < t->GetEntries(); i++) {
		t->GetEntry(i);
		std::cout << "entry: " << i 
				<< " run=" << runNbr 
				<< ", evt=" << evtNbr
				<< ", nlep=" << d_nlep
				<< ", njet=" << d_njet
				<< ", truth_kvec=[" << truth_kvec[0] <<", " << truth_kvec[1] <<", " << truth_kvec[2] << "]"
				<< "\n";
	}
}

even with:

[10:11 binet@zoidberg issue-42318]$ root ./ttbar13_short.root 
   ------------------------------------------------------------------
  | Welcome to ROOT 6.22/02                        https://root.cern |
  | (c) 1995-2020, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Aug 17 2020, 12:46:52                 |
  | From tags/v6-22-02@v6-22-02                                      |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

root [0] 
Attaching file ./ttbar13_short.root as _file0...
(TFile *) 0x559204750ab0
root [1] auto t = (TTree*)_file0->Get("truth")
(TTree *) @0x7ffdfa7bcf30
root [2] auto ff = TFile::Open("toto.root", "RECREATE")
(TFile *) @0x7ffdfa7bcf30
root [3] auto tt = t->CopyTree("")
(TTree *) @0x7ffdfa7bcf30
root [4] tt->Write()
(int) 9369
root [5] ff->Close()
root [6] .q

(and dumping that file with root-ls yielded the same values at first sight)

then, I noticed:

$> root-ls ./ttbar13_short.root 
=== [./ttbar13_short.root] ===
version: 62202
TTree   truth           (cycle=1)

whereas you mention:

ROOT Version: v6.20.06
Platform: centos7
Compiler: gcc8
cvmfs path: /cvmfs/sft.cern.ch/lcg/releases/LCG_97a/ROOT/v6.20.06/x86_64-centos7-gcc8-opt/bin/root

and mine:

$> root --version
ROOT Version: 6.22/02
Built for linuxx8664gcc on Aug 17 2020, 12:46:52
From tags/v6-22-02@v6-22-02

could you try with a ROOT-6.22 version?

here is a link to toto.root:

rmadar · November 17, 2020, 9:37am

Thanks for looking at this! I have actually no problem with ttbar13_short.root. The problem appears only for the big file (3.4 Go) linked here. So, I am not really surprised by what you see. Could you try with the big file and see if you can reproduce the crash?

sbinet · November 17, 2020, 9:41am

ah, ok.

do you have it on one of the LPC servers?
(my ADSL connexion is… not as fast as I’d want: 30mn and counting)

sbinet · November 17, 2020, 10:06am

the “good” news is that groot can’t dump back that file:

$> root-dump ./ttbar13.root 
panic: runtime error: makeslice: len out of range

goroutine 45 [running]:
go-hep.org/x/hep/groot/rtree.(*Basket).UnmarshalROOT(0xc00fdfcb90, 0xc0000cb400, 0x9fec, 0xa000)
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/basket.go:241 +0x425
go-hep.org/x/hep/groot/rtree.(*rbasket).inflate(0xc00fdfcb60, 0xc0032943e7, 0x7, 0x0, 0x0, 0xa01860, 0xdbef30e5, 0x9fec, 0x0, 0x3e8, ...)
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/rbasket.go:78 +0x55f
go-hep.org/x/hep/groot/rtree.(*bkreader).run(0xc00010eaf0, 0x3e8, 0x0, 0x1)
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/bkreader.go:156 +0x24f
created by go-hep.org/x/hep/groot/rtree.newBkReader
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/bkreader.go:125 +0x798

this happens as soon as “slices” are read (ex: d_lep_pt):

$> go run ./issue-42318.go ./ttbar13.root                                                           
entries: 10492000
>>> basket[d_lep_pt][16384000]: sz=-918617857
panic: runtime error: makeslice: len out of range

goroutine 11 [running]:
go-hep.org/x/hep/groot/rtree.(*Basket).UnmarshalROOT(0xc00007c9f0, 0xc006e1c000, 0x9fed, 0xa000)
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/basket.go:243 +0x50b
go-hep.org/x/hep/groot/rtree.(*rbasket).inflate(0xc00007c9c0, 0xc0031e6128, 0x8, 0x0, 0x0, 0xa01860, 0xdbee90f8, 0x9fed, 0x0, 0x3e8, ...)
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/rbasket.go:78 +0x55f
go-hep.org/x/hep/groot/rtree.(*bkreader).run(0xc00017a770, 0x3e8, 0x0, 0x1)
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/bkreader.go:156 +0x24f
created by go-hep.org/x/hep/groot/rtree.newBkReader
	/home/binet/work/gonum/src/go-hep.org/x/hep/groot/rtree/bkreader.go:125 +0x798
exit status 2

so… please file an issue against the go-hep/hep tracker, we’ll follow-up over there

rmadar · November 17, 2020, 10:06am

OK, I will! Thanks for checking!

EDIT: the opened issue can be found at https://github.com/go-hep/hep/issues/821

sbinet · November 17, 2020, 4:04pm

FTR, this was due to a var-len array that was empty over the 1e7 entries.
as it was empty, the auto-flush mechanism of groot would just not write it out until tree-close time.
but the slice of offsets wasn’t empty (1e7 entries full of 0).
and hell brok’ loose (as the reserved space for the buffer wouldn’t be big enough for the #offsets x 4bytes required length)

Axel · November 19, 2020, 10:26am

So the issue was caused by not writing the ROOT file with ROOT.

@rmadar a way more traditional and stable approach is to clone the input file, adding to it the derived (reco) data in new branches. That way your entries are guaranteed to be consistent, and no expensive post-processing is needed.

rmadar · November 19, 2020, 12:05pm

So the issue was caused by not writing the ROOT file with ROOT.

Yes, it seems so.

Thanks for the comment, event if I am not sure to fully understand what your propose:

adding to it the derived (reco) data in new branches

To do this, I’d have to find the reco event for each truth event (since they are stored in different trees with different number of entries). How exactly would you do it?

Also in my case, I have almost 400 millions truth events while I “only” have 40 million reco events, so I preferred to loop only on reco events (also because 340 millions of truth event are not reconstruced anyway, so no info to add)

Axel · November 19, 2020, 1:23pm

What I’m saying is that when generating the reco tree, that should just be added as an extension to the sim tree. This can ideally be done by friend trees and proper file names; or by writing the sim tree out as part of the (sim+reco) tree.

This has to happen as part of the reco tree generation, not once it has been generated. So please talk to the people creating these trees.