ROOT Object Memory Limits?

Hi,

I’m trying to implement a class for my analysis which is used to track statistical correlations across measurements using bootstrap replicas. This involves making replicas of some large histograms used for our unfolding, which creates a large object in memory we’d like to write out to a file.

In this way we’ve discovered that there seem to be limits on the size of objects ROOT can read/write successfully, I gather the limit is set to ~1 GB from reading old ROOT Forum posts (error message below)? Our question is whether this is an arbitrary limit, and if so whether there is some way to increase it on the our side? Or, is it a hard limit for some good reason? In that case, we’d like to know any suggestions you have.

We don’t include a working example here as I believe the problem is rather clear without it, and the objects we’re using are not public-facing but rather specific to ATLAS.

Many thanks,
Matt


ROOT Version: 6.20/06
Platform: CentOS-7
Compiler: Not Provided


Package.EventLoop INFO Processed 100000 events
TBufferFile::WriteByte… ERROR bytecount too large (more than 1073741822)
TBufferFile::WriteByte… ERROR bytecount too large (more than 1073741822)
TBufferFile::WriteByte… ERROR bytecount too large (more than 1073741822)
TFile::WriteBuffer ERROR error writing to file /home/mlb/workdir/out/histos_20220330/test_mc/fetch/data-out_InDetTrackParticles//test_mc-0.root (-1) (Bad address)
TFile::Seek ERROR cannot seek to position -1353651483 in file /home/mlb/workdir/out/histos_20220330/test_mc/fetch/data-out_InDetTrackParticles//test_mc-0.root, retpos=-1 (Invalid argument)
abort EL_JOBID=0

The limit is an implementation choice (made in the days when 1GB of RAM was very expensive :slight_smile: ) to save some space on internal pointers/reference (being 32 bits rather than 64 bits). Lifting this limitation is on our plan of work for this year but is not straightforward.

The one suggestion is to split the object over several I/O buffer. This can be done automatically if you store this object in a TTree (and the data is spread over more than one data member of the containing class).

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.