I am having trouble using the hadd command. The root files I am adding contain 3 output trees and several histograms as output from data12 and mc12. When I try to add 10 or more root files I get the following error:
hadd Target path: test4.root:/
terminate called after throwing an instance of 'std::bad_alloc’
what(): St9bad_alloc
Aborted
When I hadd 8 or 9 files or less it all works fine, but it is of course annoying when you have to hadd 500 files. I am using root version 5.34.00
Try to play with “user limits”:
[url]Upper limit of entries in one bin and in whole histogram
[url]No error when macro crashes
[url]File open limit?
[quote=“Wile E. Coyote”]Try to play with “user limits”:
[url]Upper limit of entries in one bin and in whole histogram
[url]No error when macro crashes
[url]File open limit?
Hi, thanks for the reply. I tried checking the limits and the output in the computer I am using is:
ulimit -n: unlimited
ulimit -s: 1024
I am not adding more than 1000 files and the memory is unlimited so I think that is not where the issue is.
[quote=“marc.bret.87”]
I am not adding more than 1000 files and the memory is unlimited so I think
that is not where the issue is.[/quote]
bad_alloc is an exception typically thrown when you process cannot allocate
any new memory. Your OS might not put any limits there, but there is a
physical limit just from the amount of RAM and swapped you have installed.
You could try to merge the files with hadd -T which will not merged trees
(likely to require a lot of memory). You can access them via TChains.
Also, if you use sparse histograms (THnSparse) merging them together will most
likely increase their size.
Note also … even if you have plenty of RAM and swap space … if you are using a 32-bit system, the max memory you can allocate per process is around 3 GB. If your program tries to allocate more RAM it will die with ‘std::bad_alloc’.
Try to run “top” (observe the “VIRT” / “RES” / “SHR” / “%MEM” fields), in another terminal window, in order to find how much resources your program is actually using while running.
Thanks, using hadd -T works fine. About using top here is the output I get
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32577 bret (my username) 18 0 977m 849m 14m R 74.7 3.5 0:16.77 hadd
So I guess its the problem you mentioned. I will try doing the same in a 64 bit computer but it sounds like I will just have to live with it.
Cheers
You just use “977m” VIRT, so it’s well below 3GB. Unless your machine has less than 1GB RAM (and no swap space), I see no problem here.
I see, so maybe the problem is not with allocating memory. that is annoying as I dont know very well what is causing the problem, thanks for the help
Just one more thing … I assume the “top” output comes from running “hadd” WITHOUT “-T” (well, I think the total RAM usage will depend on actual options that you use - note also that it may increase as the “hadd” opens new files, so you should “observe” it all the time when “hadd” runs - try something like “top -d 0.3”).