Error in <TBufferFile::CheckCount>: buffer offset too large (larger than 1073741822)

ROOT
, ,
1

Hello!
When I merged multiple root files with Hadd, I encountered this error.
I guess the reason for this issue may be that the merged root file is too large (128 partial files, each approximately 11GB, totaling 1.4TB), because I did not have this problem when I merged smaller root files ( totaling 100GB).
The hadd command I used is:
hadd -f -j 43 -d ${tmp_folder} ${fname}.root $(find ${data_folder} -name "*part.root" )
My OS is Ubuntu 20.04, and my ROOT version is 6.28.06.
Can someone give me some help？
Thanks a lot!

1 Like
2

Error in <TBufferFile::CheckCount>: buffer offset too large (larger than 1073741822)
Fatal in <TBufferFile::AutoExpand>: Request to expand to a negative size, likely due to an integer overflow: 0x8730079e for a max of 0x7ffffffe.
aborting

3

There is currently a limit on the number of baskets a TTree can hold (around 50 millions). You can do any of:

  1. Merge less files to stay under the limit
  2. Try a slow merge which might reduce the number of baskets ( hadd -O ... )
  3. Don’t merge the files and use them via a TChain (which itself could be persistent).
4

Hello @pcanal:
Thanks for your suggestion!
I am a ROOT beginner. I have read the official guide, but I still don’t understand what a Basket is. What is it related to the Branch? And, what is the unit for a Basket? not exceed 50 million bytes?

Thanks a lot!

5

The branches data is store in chunks that contains the data for that branch for several entries, those chunks are called ‘Basket’ (they are called pages in the TTree replacement called RNTuple)

If you do:

auto f = TFile::Open(filename, "READ");
auto t = f->GetObject<TTree>(treename);
t->Print();

For each branch you will see the number of baskets and the (rough) size of those baskets.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.