Dear ROOT users,
I am using OpenGATE to simulate PET scans and am a novice at ROOT. I am attempting to read the output root file and perform some data processing on the data. The root file I have generated is 100+GB, containing 1.2Billion events.
I recently profiled the application and found that 70% of computation time is spent on stream_ptr->GetEntry().
stream_ptr = new TChain(this->chain_name.c_str());
stream_ptr->Add(fullfilename.c_str());
// Turn off all branches
stream_ptr->SetBranchStatus("*",0);
// Branches are turned back on by SetBranchAddress()
stream_ptr->SetBranchAddress("time1", &time1);
stream_ptr->SetBranchAddress("time2", &time2);
stream_ptr->SetBranchAddress("eventID1",&eventID1);
stream_ptr->SetBranchAddress("eventID2",&eventID2);
stream_ptr->SetBranchAddress("energy1", &energy1);
stream_ptr->SetBranchAddress("energy2", &energy2);
stream_ptr->SetBranchAddress("comptonPhantom1", &comptonphantom1);
stream_ptr->SetBranchAddress("comptonPhantom2", &comptonphantom2);
In terms of reading the ROOT data we use:
if (current_position == nentries)
return Succeeded::no;
if (stream_ptr->GetEntry(static_cast<Long64_t>(current_position), 0) == 0)
return Succeeded::no;
current_position ++ ;
DoStuff();
We recently found that stream_ptr->SetBranchStatus("*",0); halved the time taken (I believe this is because there are many other branches in the root file that are not needed in the processing) but I am looking to accelerate the reading.
Are there any suggestions that might help me?
Regards,
Please read tips for efficient and successful posting and posting code
ROOT Version: 6.22/06Platform: MacOS/LinixCompiler: Not Provided
Hi Robert,
Indeed using SetBranchStatus("*", 0) is the correct thing to do: it turns off reading of all branches (aka dataset columns) by default and you can later only turn on the ones you actually need.
It’s not surprising that GetEntry takes a significant fraction of the computing time: it’s where all the reading work happens. Now: where does GetEntry itself spend time? One thing to check is whether you are simply hitting a cap in your I/O bandwidth, it can easily happen e.g. when reading from a network storage or from a spinning disk. If that’s the case, you might need faster hardware.R__unzip or similar, your bottleneck is decompression – you can consider switching to a different compression algorithm (it’s something that you can configure when writing the data out), e.g. zstd has a great balance between speed and compression factor. Another way to speed up a CPU-bound application is to run the processing in multiple threads or multiple processes. For a multi-thread approach, you can use RDataFrame , although that might require rewriting large parts of the logic. A less intrusive method could be to use TTreeProcessorMT (multi-thread) or TTreeProcessorMP (multi-process) to run your processing in parallel.
Hope this helps!
Hi Enrico,
Thank you for your comment. This is all being performed locally, on a 2019 Macbook Pro, for this testing testing/profiling.
I belive you are correct that the decompression is slowing the process, but it does not appear to the only limiting factor.
I have read around the RDataFrame and TTreeProcessor(MT/MP) links above. Thank you for providing them but as you mentioned, this will require large rewriting.
Regards,
Alright, @pcanal might have some suggestions to speed up many single-thread TBranch::GetEntry calls. What types are the branches? Can you post the output of tree->Print()?
pcanal
February 9, 2021, 4:08pm
6
In you case you can improve the single thread performance (too much time in spent in TTree::GetEntry itself) by calling TBranch::GetEntry directly.
TBranch *br_time1 = nullptr;
TBranch *br_time2 = nullptr;
... etc ...
stream_ptr->SetBranchAddress("time1", &time1, &br_time1);
stream_ptr->SetBranchAddress("time2", &time2, &br_time2);
... etc ...
and
if (current_position == nentries)
return Succeeded::no;
auto brentry = stream_ptr->LoadTree(static_cast<Long64_t>(current_position));
if (brentry < 0)
return Succeeded::no;
if (br_time1->GetEntry(brentry) == 0)
return Succeeded::no;
if (br_time2->GetEntry(brentry) == 0)
return Succeeded::no;
... etc ...
current_position ++ ;
DoStuff();
@eguiraud The output of stream_ptr->Print() is
******************************************************************************
*Chain :tree : Output/bigg.root *
******************************************************************************
******************************************************************************
*Tree :tree : tree *
*Entries : 10530000 : Total = 2369535749 bytes File Size = 1142040666 *
* : : Tree compression factor = 2.07 *
******************************************************************************
*Br 0 :eventID1 : eventID1/I *
*Entries : 10530000 : Total Size= 44970317 bytes File Size = 22800000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.95 *
*............................................................................*
*Br 1 :eventID2 : eventID2/I *
*Entries : 10530000 : Total Size= 44970317 bytes File Size = 22800000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.95 *
*............................................................................*
*Br 2 :sourceID1 : sourceID1/I *
*Entries : 10530000 : Total Size= 45000321 bytes File Size = 3270000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.58 *
*............................................................................*
*Br 3 :sourceID2 : sourceID2/I *
*Entries : 10530000 : Total Size= 45000321 bytes File Size = 3270000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.58 *
*............................................................................*
*Br 4 :sourcePosX1 : sourcePosX1/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42420000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 5 :sourcePosX2 : sourcePosX2/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42420000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 6 :sourcePosY1 : sourcePosY1/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42390000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 7 :sourcePosY2 : sourcePosY2/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42390000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 8 :sourcePosZ1 : sourcePosZ1/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42240000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 9 :sourcePosZ2 : sourcePosZ2/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42240000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 10 :globalPosX1 : globalPosX1/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42390000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 11 :globalPosX2 : globalPosX2/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42270000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 12 :globalPosY1 : globalPosY1/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42210000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 13 :globalPosY2 : globalPosY2/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42180000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.05 *
*............................................................................*
*Br 14 :globalPosZ1 : globalPosZ1/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 43110000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.03 *
*............................................................................*
*Br 15 :globalPosZ2 : globalPosZ2/F *
*Entries : 10530000 : Total Size= 45060329 bytes File Size = 42990000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.03 *
*............................................................................*
*Br 16 :time1 : time1/D *
*Entries : 10530000 : Total Size= 87000313 bytes File Size = 82980000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.04 *
*............................................................................*
*Br 17 :time2 : time2/D *
*Entries : 10530000 : Total Size= 87000313 bytes File Size = 82680000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.04 *
*............................................................................*
*Br 18 :energy1 : energy1/F *
*Entries : 10530000 : Total Size= 44940313 bytes File Size = 40380000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.10 *
*............................................................................*
*Br 19 :energy2 : energy2/F *
*Entries : 10530000 : Total Size= 44940313 bytes File Size = 40260000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.10 *
*............................................................................*
*Br 20 :comptVolName1 : comptVolName1/C *
*Entries : 10530000 : Total Size= 98010337 bytes File Size = 20790000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 4.69 *
*............................................................................*
*Br 21 :comptVolName2 : comptVolName2/C *
*Entries : 10530000 : Total Size= 98850337 bytes File Size = 22650000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 4.34 *
*............................................................................*
*Br 22 :RayleighVolName1 : RayleighVolName1/C *
*Entries : 10530000 : Total Size= 98100349 bytes File Size = 20910000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 4.66 *
*............................................................................*
*Br 23 :RayleighVolName2 : RayleighVolName2/C *
*Entries : 10530000 : Total Size= 98100349 bytes File Size = 20910000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 4.66 *
*............................................................................*
*Br 24 :comptonPhantom1 : comptonPhantom1/I *
*Entries : 10530000 : Total Size= 45180345 bytes File Size = 3450000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 12.92 *
*............................................................................*
*Br 25 :comptonPhantom2 : comptonPhantom2/I *
*Entries : 10530000 : Total Size= 45180345 bytes File Size = 3660000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 12.18 *
*............................................................................*
*Br 26 :comptonCrystal1 : comptonCrystal1/I *
*Entries : 10530000 : Total Size= 45180345 bytes File Size = 9450000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 4.72 *
*............................................................................*
*Br 27 :comptonCrystal2 : comptonCrystal2/I *
*Entries : 10530000 : Total Size= 45180345 bytes File Size = 9570000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 4.66 *
*............................................................................*
*Br 28 :RayleighPhantom1 : RayleighPhantom1/I *
*Entries : 10530000 : Total Size= 45210349 bytes File Size = 3480000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 12.82 *
*............................................................................*
*Br 29 :RayleighPhantom2 : RayleighPhantom2/I *
*Entries : 10530000 : Total Size= 45210349 bytes File Size = 3480000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 12.82 *
*............................................................................*
*Br 30 :RayleighCrystal1 : RayleighCrystal1/I *
*Entries : 10530000 : Total Size= 45210349 bytes File Size = 5370000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 8.31 *
*............................................................................*
*Br 31 :RayleighCrystal2 : RayleighCrystal2/I *
*Entries : 10530000 : Total Size= 45210349 bytes File Size = 5550000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 8.04 *
*............................................................................*
*Br 32 :gantryID1 : gantryID1/I *
*Entries : 10530000 : Total Size= 45000321 bytes File Size = 3270000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.58 *
*............................................................................*
*Br 33 :rsectorID1 : rsectorID1/I *
*Entries : 10530000 : Total Size= 45030325 bytes File Size = 16050000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 2.77 *
*............................................................................*
*Br 34 :moduleID1 : moduleID1/I *
*Entries : 10530000 : Total Size= 45000321 bytes File Size = 3270000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.58 *
*............................................................................*
*Br 35 :submoduleID1 : submoduleID1/I *
*Entries : 10530000 : Total Size= 45090333 bytes File Size = 12120000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 3.67 *
*............................................................................*
*Br 36 :crystalID1 : crystalID1/I *
*Entries : 10530000 : Total Size= 45030325 bytes File Size = 17760000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 2.50 *
*............................................................................*
*Br 37 :layerID1 : layerID1/I *
*Entries : 10530000 : Total Size= 44970317 bytes File Size = 3240000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.69 *
*............................................................................*
*Br 38 :gantryID2 : gantryID2/I *
*Entries : 10530000 : Total Size= 45000321 bytes File Size = 3270000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.58 *
*............................................................................*
*Br 39 :rsectorID2 : rsectorID2/I *
*Entries : 10530000 : Total Size= 45030325 bytes File Size = 16110000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 2.76 *
*............................................................................*
*Br 40 :moduleID2 : moduleID2/I *
*Entries : 10530000 : Total Size= 45000321 bytes File Size = 3270000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.58 *
*............................................................................*
*Br 41 :submoduleID2 : submoduleID2/I *
*Entries : 10530000 : Total Size= 45090333 bytes File Size = 12120000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 3.67 *
*............................................................................*
*Br 42 :crystalID2 : crystalID2/I *
*Entries : 10530000 : Total Size= 45030325 bytes File Size = 17520000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 2.54 *
*............................................................................*
*Br 43 :layerID2 : layerID2/I *
*Entries : 10530000 : Total Size= 44970317 bytes File Size = 3240000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 13.69 *
*............................................................................*
*Br 44 :sinogramTheta : sinogramTheta/F *
*Entries : 10530000 : Total Size= 45120337 bytes File Size = 42090000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.06 *
*............................................................................*
*Br 45 :sinogramS : sinogramS/F *
*Entries : 10530000 : Total Size= 45000321 bytes File Size = 43260000 *
*Baskets : 30000 : Basket Size= 32000 bytes Compression= 1.03 *
*............................................................................*
Here bigg.root is ~1.1GB.
Obviously there are many branches in here that are not needed in terms of the analysis above I would prefer to keep the structure of the root file untouched, there are other application which may need this.
@pcanal I think I will attempt to incorporate the suggested branch tests into the code, there are some checks in DoStuff() that will benifit from this but it unlikely any of the branches are “missing” from each entry.
Thank you for your suggestion.
pcanal
February 9, 2021, 6:29pm
8
[ Note, I updated the example to fix the check on the return value of LoadTree ]
1 Like
pcanal
February 10, 2021, 2:51am
9
Just for clarity the advantage is the code I propose is not in the checks (they are really ancillary), it is the unrolling and optimization of the TTree::GetEntry loop over the branch (essentially for each branch, check if enabled then call GetEntry)
Thank you for your help @pcanal .
Thanks the suggestion you presented here I have managed to accelerate the processing by appoximately a factor 2, as, for some cases, only a handful Branch GetEntry() calls are made before an event is rejected by DoStuff() due to the event failing to satisfy a condition and the code moves onto the next event.
You can find the PR discussion here
system
February 25, 2021, 10:41am
12
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
