Limit on the size of objects in TFile

arekfu · October 8, 2020, 1:50pm

Hello ROOTers,

Is there a hard limit on the size of the objects that can be serialized to a TFile? The minimal example attached below fails with the following error message:

$ root -l
root [0] .L minimal_repr.cc+
Info in <TUnixSystem::ACLiC>: creating shared library /volatile/davide/src/root/bug/./minimal_repr_cc.so    
root [1] timing(152, 299)
Writing...
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

If there is a hard limit, can I somehow split up the objects in such a way that I can still write them on disk?

Thanks for your advice!
Davide

minimal_repr.cc (1.5 KB) minimal_repr.hh (1.4 KB)

ROOT Version: master branch
Platform: Ubuntu 16.04
Compiler: gcc 5.4.0

pcanal · October 8, 2020, 2:08pm

Yes, there is a limit of 1GB per single I/O storage. To work-around the limit store your object in a split (the default) branch in a TTree, even if it is just with one entry.

arekfu · October 8, 2020, 3:59pm

Philippe, thank you for your reply. I have tried to follow the instructions in your post here, but I run into this error message:

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TROOT::WriteTObject>: The current directory (PyROOT) is not associated with a file. The object (holder) has not been written.

My code reads (PyROOT)

# create large object m
rootf = TFile(output_file, "RECREATE")
t = TTree('holder', 'holder')
t.Branch('m.', m)
t.SetDirectory(rootf)
t.Fill()
t.Write()
rootf.Close()

Is there anything blatantly wrong with this?

Thanks again,
Davide

pcanal · October 8, 2020, 4:41pm

Did you generate a dictionary for the type?

Can you send the result of

t.Print();

and

ROOT.TClass.GetClass(name_of_your_type).GetStreamerInfo().ls()

arekfu · October 9, 2020, 7:22am

So I decided to reproduce this in the minimal C++ example above. I am attaching the new version of the code.

minimal_repr.cc (1.7 KB) minimal_repr.hh (1.4 KB)

Here is the output:

$ root -l
root [0] .L minimal_repr.cc+
root [1] minimal_repr()

StreamerInfo for class: A, version=1, checksum=0xef512161
  TObject        BASE            offset=  0 type=66 Basic ROOT object
  TString        name            offset= 16 type=65
  map<TString,double> map1            offset= 40 type=300 ,stl=4, ctype=61,
  map<TString,double> map2            offset= 88 type=300 ,stl=4, ctype=61,
  map<TString,double> map3            offset=136 type=300 ,stl=4, ctype=61,
  map<TString,double> map4            offset=184 type=300 ,stl=4, ctype=61,
   i= 0, TObject         type= 66, offset=  0, len=1, method=0
   i= 1, name            type= 65, offset= 16, len=1, method=0
   i= 2, map1            type=300, offset= 40, len=1, method=0
   i= 3, map2            type=300, offset= 88, len=1, method=0
   i= 4, map3            type=300, offset=136, len=1, method=0
   i= 5, map4            type=300, offset=184, len=1, method=0

StreamerInfo for class: B, version=1, checksum=0x10a16a9d
  TObject        BASE            offset=  0 type=66 Basic ROOT object
  vector<pair<double,double> > vec1            offset= 16 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec2            offset= 40 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec3            offset= 64 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec4            offset= 88 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec5            offset=112 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec6            offset=136 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec7            offset=160 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec8            offset=184 type=300 ,stl=1, ctype=61,
  vector<pair<double,double> > vec9            offset=208 type=300 ,stl=1, ctype=61,
  vector<map<TString,vector<A> > > big_vec         offset=232 type=300 ,stl=1, ctype=61,
  vector<TString> vec_str         offset=256 type=300 ,stl=1, ctype=61,
   i= 0, TObject         type= 66, offset=  0, len=1, method=0
   i= 1, vec1            type=300, offset= 16, len=1, method=0
   i= 2, vec2            type=300, offset= 40, len=1, method=0
   i= 3, vec3            type=300, offset= 64, len=1, method=0
   i= 4, vec4            type=300, offset= 88, len=1, method=0
   i= 5, vec5            type=300, offset=112, len=1, method=0
   i= 6, vec6            type=300, offset=136, len=1, method=0
   i= 7, vec7            type=300, offset=160, len=1, method=0
   i= 8, vec8            type=300, offset=184, len=1, method=0
   i= 9, vec9            type=300, offset=208, len=1, method=0
   i=10, big_vec         type=300, offset=232, len=1, method=0
   i=11, vec_str         type=300, offset=256, len=1, method=0
Writing...
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
******************************************************************************
*Tree    :holder    : holder                                                 *
*Entries :        1 : Total =      2031785835 bytes  File  Size =          0 *
*        :          : Tree compression factor =   1.00                       *
******************************************************************************
*Branch  :b.                                                                 *
*Entries :        1 : BranchElement (see below)                              *
*............................................................................*
*Br    0 :b.TObject.fUniqueID : UInt_t                                       *
*Entries :        1 : Total  Size=        741 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    1 :b.TObject.fBits : UInt_t                                           *
*Entries :        1 : Total  Size=        725 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    2 :b.vec1    : Int_t b.vec1_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    3 :b.vec1.first : Double_t first[b.vec1_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    4 :b.vec1.second : Double_t second[b.vec1_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    5 :b.vec2    : Int_t b.vec2_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    6 :b.vec2.first : Double_t first[b.vec2_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    7 :b.vec2.second : Double_t second[b.vec2_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    8 :b.vec3    : Int_t b.vec3_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br    9 :b.vec3.first : Double_t first[b.vec3_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   10 :b.vec3.second : Double_t second[b.vec3_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   11 :b.vec4    : Int_t b.vec4_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   12 :b.vec4.first : Double_t first[b.vec4_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   13 :b.vec4.second : Double_t second[b.vec4_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   14 :b.vec5    : Int_t b.vec5_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   15 :b.vec5.first : Double_t first[b.vec5_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   16 :b.vec5.second : Double_t second[b.vec5_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   17 :b.vec6    : Int_t b.vec6_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   18 :b.vec6.first : Double_t first[b.vec6_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   19 :b.vec6.second : Double_t second[b.vec6_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   20 :b.vec7    : Int_t b.vec7_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   21 :b.vec7.first : Double_t first[b.vec7_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   22 :b.vec7.second : Double_t second[b.vec7_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   23 :b.vec8    : Int_t b.vec8_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   24 :b.vec8.first : Double_t first[b.vec8_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   25 :b.vec8.second : Double_t second[b.vec8_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   26 :b.vec9    : Int_t b.vec9_                                          *
*Entries :        1 : Total  Size=       5154 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   27 :b.vec9.first : Double_t first[b.vec9_]                             *
*Entries :        1 : Total  Size=       1997 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   28 :b.vec9.second : Double_t second[b.vec9_]                           *
*Entries :        1 : Total  Size=       2003 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
*Br   29 :b.big_vec : vector<map<TString,vector<A> > >                       *
*Entries :        1 : Total  Size= 2031737423 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
*Br   30 :b.vec_str : vector<TString>                                        *
*Entries :        1 : Total  Size=       3087 bytes  One basket in memory    *
*Baskets :        0 : Basket Size=      32000 bytes  Compression=   1.00     *
*............................................................................*
Elapsed time for (152, 299): 28156 ms

I do not generate dictionaries for my types, I just load them with .L. Is that the problem? I am a bit suspicious about the fact that apparently big_vec did not get split.

Thanks again,
Davide

arekfu · October 12, 2020, 11:53am

Hello again,

I trimmed down the example some more to the point where it’s clear that the main problem is the attempt to serialize a large vector<map<TString, map<TString, double>>>. Going through TTree does not seem to help. Is there any other way I can persist the content of this object onto a TFile, short of writing my own format?

Thanks,
Davide

minimal_repr.hh (504 Bytes) minimal_repr.cc (1.5 KB)

system · October 26, 2020, 11:53am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.