How to merge ROOT files larger than 100 GB

ROOT
1

Dear ROOT users
How can I merge .root files larger than 100 GB?
Thank you

2

Hello,

I think this answer by @pcanal can be useful:

Cheers,
Enric

3

Thank you for your reply.
Should I create a shared library and set the LD_LIBRARY_PATH to it?
I am a Geant4/GATE Monte Carlo user, where do I create the shared library?
Thank you again.

4

Hello,

Actually what @pcanal suggests is to create a shared library that you preload before running hadd:

LD_PRELOAD=startup_C.so hadd output.root input_*

You can create the shared library in the same directory where the ROOT files to be merged are.

5

Hi Enric
Thank you for your reply. BUT I have a problem when I merge the root files:
When the size of all splits was about 420 GB it was merged to about 300 GB and then stopped. To override this, I reduced my run time and the size of all splits was about 250 GB but when I used this method to merge them it stopped at about 186 GB. How can I resolve this error? Did you face the same condition?
Cheers,
Hadi

6

I compiled directly the C++ program using “root.exe -b -l -q startup.C+” command in the directory where the root files are in and then used “LD_PRELOAD=startup_C.so hadd output.root input_*”. Should I do separate work to create a shared library and addressed it in .bashrc using LD_LIBRARY_PATH?
Thank you so much.

7

@pcanal can you comment on why it can be that the size of the merged result is smaller than the sum of the size of the individual fragments?

8

I compiled directly the C++ program using “root.exe -b -l -q startup.C+” command in the directory where the root files are in and then used “LD_PRELOAD=startup_C.so hadd output.root input_*”. Should I do separate work to create a shared library and addressed it in .bashrc using LD_LIBRARY_PATH?

No, it should be enough to preload that little shared library that sets the maximum tree size before hadd operates.

9

@pcanal I have a problem merging the root files larger than 100 GB that I described in previous messages. How can I resolve it? I used root_v6.14.00 and I addressed the startup_C.so directory in .bashrc using LD_LIBRARY_PATH.
Thank you so much.

10

@hadirezaei I am not sure. What is the result of calling:

file = TFile::Open(filename);
file->ls();

for both one of the input and the result.
and

tree->Print();

for your TTree (both for one input and the output).

Cheers,
Philippe.

11

@pcanal thank you for your response.
How can I use these commands?
and another point is that when I compile the startup.C program I find this warning: " Warning in cling::IncrementalParser::CheckABICompatibility():
Possible C++ standard library mismatch, compiled with GLIBCXX ‘20180415’
Extraction of runtime standard library version was: ‘20191114’
(int) 0"
Could the merging issue be the cause of this warning? and How can I resolve it?
Thank you so much.
Hadi

12

I used ROOT 6.14/00 and g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0.
Cheers,
Hadi

13

Dear ROOT users
which version of the gcc match to the ROOT 6.14/00?
Best
Hadi

14

This warning is fatal … and make the library unused. So we need to fix that problem first.

15

It depends where/when/how it was built. You indeed need to make sure that you use the same gcc version for your code as with the one use to build ROOT.

16

@pcanal
Thank you for your response.
How can I fix it?
Best
Hadi

17

By using the same compiler for ROOT and your code.

So the primary question is “where/when/how ROOT was built.” and/or where did you get it from? From that answer you will be able to tell whether you need to:

  • Select a different build from the same place
  • or Find another place to download ROOT from
  • or Use a different compiler
  • or Build ROOT yourself.
18

I got it from Release 61400 - ROOT and I installed it using CMake.
Thanks

19

So you downloaded the source tar file and built it using cmake and the default compiler?

20

@pcanal yes, I did it.