Hi ROOT-ers, I’m seeking advice for managing large TH2D objects of >1 GB size in ROOT v6.15. I’m successfully filling a (sparse) TH2D histogram with ~30k x ~700k bins (yes, you read that right), and applying 1 standard deviation Poisson error bars. However, the object appears to be <2 GB but over the ~1 GB size limit indicated in segfault errors I get at the level of calling an integral or writing the histogram to a root file, “Error in : bytecount too large (more than 1073741822)”. This is despite 16 GB working memory on my machine, so this isn’t inherently intractable. My main question is, 1) are there simple options to set a larger buffer size for a TH2D or TFile object to smoothly anticipate a size scale of 2 (even up to 10) GB? Is this limit really fixed in ROOT? If this is a firm limit I’d really encourage making it possible to increase this in future versions. Thanks in advance for any pointers!
Thanks for the post and welcome to the ROOT Community!
I am sorry you stumbled in this limitation.
Currently, ROOT’s IO cannot handle buffers which are larger than 1 GB, it’s a known issue, which will be addressed this year.
For your particular case, I can suggest to perhaps to divide your large histogram in 4 sub histograms to then save them separately. Please bear with me if I missed something and this solution is too simplistic for you.
Thank you for that confirmation, that saves me time. I’m considering your suggestion. I will likely cycle through my input data twice, to be able to intelligently partition the data in the second pass based on criteria I can only get from a first full pass. It should work. I’d still encourage the developers to increase this limit with some priority. Modern computing environments can manage substantially larger volumes and multiple histograms isn’t always a great solution. At least to make buffer size expandable would be very helpful, even if the same size is set as default.
Thank you for the welcome to the ROOT community, I am actually a PhD particle physicist and used ROOT extensively as a graduate student years ago. I’m applying it in a very different context now. What I’m really after is the integrated assessment of errors that the ROOT histogram objects enable! Fingers crossed