I have daq program that writes a TTree and calls TTree::Autosave(“saveself”) every 30 seconds. The TTree contains only simple objects - shorts, ints, floats, doubles and a few arrays of shorts. I call TTree::SetMaxTreeSize(200000000000) (200 GBytes) though my trees never get larger than about 5 GB. The execution time for Autosave() appears to grow linearly with file size. Does this make sense? IS there a way to avoid this growth? Thanks.
I am using root v 5.06/00 on windows XP.
Ed
You will observe this behaviour if you create a memory-residen Tree.
The typical mistake is to do
TTree *T new TTree(...
TFile *f = new TFile(...
instead of
TFile *f = new TFile(...
TTree *T = new TTree(...
If this is not the case, post the output of mytree.Print() just before closing the file.
Rene
Rene,
I do the TFile *f = new TFile(…) before the TTree *t = new TTree(…)
I also create 2 TTree’s, in two separate files. Heres’t the Print() from each (They have different numbers of entries - thats normal)
[code]******************************************************************************
*Tree :nth : Leap Hit data *
*Entries : 56583959 : Total = 3518744399 bytes File Size = 633358108 *
: : Tree compression factor = 5.56 *
Br 0 :hreg : hit/S *
Entries : 56583959 : Total Size= 113513865 bytes File Size = 5855375 *
Baskets : 3544 : Basket Size= 32000 bytes Compression= 19.37 *
…
Br 1 :x : x/S *
Entries : 56583959 : Total Size= 113503218 bytes File Size = 92776238 *
Baskets : 3544 : Basket Size= 32000 bytes Compression= 1.22 *
…
Br 2 :y : y/S *
Entries : 56583959 : Total Size= 113503218 bytes File Size = 92752438 *
Baskets : 3544 : Basket Size= 32000 bytes Compression= 1.22 *
…
Br 3 :z : z/i *
Entries : 56583959 : Total Size= 227026864 bytes File Size = 79342909 *
Baskets : 7088 : Basket Size= 32000 bytes Compression= 2.86 *
…
Br 4 :tof : tof/F *
Entries : 56583959 : Total Size= 227041052 bytes File Size = 172402407 *
Baskets : 7088 : Basket Size= 32000 bytes Compression= 1.32 *
…
Br 5 :chi2 : chi2/s *
Entries : 56583959 : Total Size= 113513868 bytes File Size = 75571577 *
Baskets : 3544 : Basket Size= 32000 bytes Compression= 1.50 *
…
Br 6 :v : vs/F *
Entries : 56583959 : Total Size= 227026867 bytes File Size = 4794022 *
Baskets : 7088 : Basket Size= 32000 bytes Compression= 47.31 *
…
Br 7 :pulse : pulse/F *
Entries : 56583959 : Total Size= 227055240 bytes File Size = 15960904 *
Baskets : 7088 : Basket Size= 32000 bytes Compression= 14.21 *
…
Br 8 :erate : erate/F *
Entries : 56583959 : Total Size= 227055240 bytes File Size = 1971968 *
Baskets : 7088 : Basket Size= 32000 bytes Compression= 115.02 *
…
Br 9 :tstage : tstage/s *
Entries : 56583959 : Total Size= 113520968 bytes File Size = 863452 *
Baskets : 3544 : Basket Size= 32000 bytes Compression= 131.33 *
…
Br 10 :VMcpFront : Vf/F *
Entries : 56583959 : Total Size= 227083674 bytes File Size = 1966117 *
Baskets : 7089 : Basket Size= 32000 bytes Compression= 115.38 *
…
Br 11 :VMcpBias : Vb/F *
Entries : 56583959 : Total Size= 227076582 bytes File Size = 1964647 *
Baskets : 7089 : Basket Size= 32000 bytes Compression= 115.46 *
…
Br 12 :TargetErate : ErateT/F *
Entries : 56583959 : Total Size= 227097951 bytes File Size = 1893033 *
Baskets : 7090 : Basket Size= 32000 bytes Compression= 119.84 *
…
Br 13 :pulseDelta : pulseDelta/S *
Entries : 56583959 : Total Size= 113535248 bytes File Size = 75942896 *
Baskets : 3545 : Basket Size= 32000 bytes Compression= 1.49 *
…
Br 14 :laserx : xl/I *
Entries : 56583959 : Total Size= 227062398 bytes File Size = 1831331 *
Baskets : 7089 : Basket Size= 32000 bytes Compression= 123.86 *
…
Br 15 :lasery : yl/I *
Entries : 56583959 : Total Size= 227062398 bytes File Size = 1834489 *
Baskets : 7089 : Basket Size= 32000 bytes Compression= 123.64 *
…
Br 16 :laserz : zl/I *
Entries : 56583959 : Total Size= 227062398 bytes File Size = 1843142 *
Baskets : 7089 : Basket Size= 32000 bytes Compression= 123.06 *
…
Br 17 :laserpower : lp/F *
Entries : 56583959 : Total Size= 227090846 bytes File Size = 1961103 *
Baskets : 7090 : Basket Size= 32000 bytes Compression= 115.68 *
…
Br 18 :FractureGuard : fg/s *
Entries : 56583959 : Total Size= 113545868 bytes File Size = 868525 *
Baskets : 3545 : Basket Size= 32000 bytes Compression= 130.61 *
…
[/code]and here’s the other one[code]]*********************
*Tree :nt : Leap raw data *
*Entries : 56248424 : Total = 8024640204 bytes File Size = 1306291239 *
: : Tree compression factor = 6.15 *
*Br 0 :rhits : nx0/i:nx1:ny0:ny1:nu0:nu1 *
*Entries : 56248424 : Total Size= 1354041110 bytes File Size = 27421320 *
*Baskets : 42292 : Basket Size= 32000 bytes Compression= 49.34 *
…
*Br 1 :x0 : x0[nx0]/s *
*Entries : 56248424 : Total Size= 340948375 bytes File Size = 182762154 *
*Baskets : 17664 : Basket Size= 32000 bytes Compression= 1.86 *
…
*Br 2 :x1 : x1[nx1]/s *
*Entries : 56248424 : Total Size= 340506741 bytes File Size = 183909488 *
*Baskets : 17650 : Basket Size= 32000 bytes Compression= 1.85 *
…
*Br 3 :y0 : y0[ny0]/s *
*Entries : 56248424 : Total Size= 340883783 bytes File Size = 183116400 *
*Baskets : 17662 : Basket Size= 32000 bytes Compression= 1.86 *
…
*Br 4 :y1 : y1[ny1]/s *
*Entries : 56248424 : Total Size= 341182196 bytes File Size = 183636929 *
*Baskets : 17671 : Basket Size= 32000 bytes Compression= 1.86 *
…
*Br 5 :u0 : u0[nu0]/s *
*Entries : 56248424 : Total Size= 340661638 bytes File Size = 182748236 *
*Baskets : 17655 : Basket Size= 32000 bytes Compression= 1.86 *
…
*Br 6 :u1 : u1[nu1]/s *
*Entries : 56248424 : Total Size= 340740444 bytes File Size = 183604219 *
*Baskets : 17657 : Basket Size= 32000 bytes Compression= 1.85 *
…
*Br 7 :vs : v/F *
*Entries : 56248424 : Total Size= 225681743 bytes File Size = 4781104 *
*Baskets : 7046 : Basket Size= 32000 bytes Compression= 47.16 *
…
*Br 8 :pulse : pulse/D *
*Entries : 56248424 : Total Size= 451452284 bytes File Size = 140175608 *
*Baskets : 14093 : Basket Size= 32000 bytes Compression= 3.22 *
…
*Br 9 :erate : erate/F *
*Entries : 56248424 : Total Size= 225702902 bytes File Size = 1954126 *
*Baskets : 7046 : Basket Size= 32000 bytes Compression= 115.38 *
…
*Br 10 :xstage : xstage/S *
*Entries : 56248424 : Total Size= 112844777 bytes File Size = 834951 *
*Baskets : 3523 : Basket Size= 32000 bytes Compression= 135.02 *
…
*Br 11 :ystage : ystage/S *
*Entries : 56248424 : Total Size= 112844777 bytes File Size = 834951 *
*Baskets : 3523 : Basket Size= 32000 bytes Compression= 135.02 *
…
*Br 12 :zstage : zstage/S *
*Entries : 56248424 : Total Size= 112844777 bytes File Size = 834951 *
*Baskets : 3523 : Basket Size= 32000 bytes Compression= 135.02 *
…
*Br 13 :VMcpBias : Vbias/F *
*Entries : 56248424 : Total Size= 225724126 bytes File Size = 1946633 *
*Baskets : 7047 : Basket Size= 32000 bytes Compression= 115.83 *
…
*Br 14 :VMcpFront : VFront/F *
*Entries : 56248424 : Total Size= 225731179 bytes File Size = 1948115 *
*Baskets : 7047 : Basket Size= 32000 bytes Compression= 115.75 *
…
*Br 15 :TargetErate : ErateT/F *
*Entries : 56248424 : Total Size= 225745359 bytes File Size = 1874771 *
*Baskets : 7048 : Basket Size= 32000 bytes Compression= 120.29 *
…
*Br 16 :Thresh : tx0/s:tx1:ty0:ty1:tu0:tu1 *
*Entries : 56248424 : Total Size= 677200163 bytes File Size = 5540254 *
*Baskets : 21146 : Basket Size= 32000 bytes Compression= 122.10 *
…
*Br 17 :UpperThresh : tux/s:tuy:tuu *
*Entries : 56248424 : Total Size= 338618014 bytes File Size = 2590388 *
*Baskets : 10573 : Basket Size= 32000 bytes Compression= 130.60 *
…
*Br 18 :Walk : wx0/S:wx1:wy0:wy1:wu0:wu1 *
*Entries : 56248424 : Total Size= 677157865 bytes File Size = 4990460 *
*Baskets : 21146 : Basket Size= 32000 bytes Compression= 135.55 *
…
*Br 19 :laserx : xl/I *
*Entries : 56248424 : Total Size= 225709942 bytes File Size = 1813235 *
*Baskets : 7046 : Basket Size= 32000 bytes Compression= 124.34 *
…
*Br 20 :lasery : yl/I *
*Entries : 56248424 : Total Size= 225709942 bytes File Size = 1816368 *
*Baskets : 7046 : Basket Size= 32000 bytes Compression= 124.13 *
…
*Br 21 :laserz : zl/I *
*Entries : 56248424 : Total Size= 225709942 bytes File Size = 1824917 *
*Baskets : 7046 : Basket Size= 32000 bytes Compression= 123.55 *
…
*Br 22 :laserpower : lp/F *
*Entries : 56248424 : Total Size= 225738217 bytes File Size = 1942933 *
*Baskets : 7047 : Basket Size= 32000 bytes Compression= 116.06 *
…
*Br 23 :FractureGuard : fg/s *
*Entries : 56248424 : Total Size= 112869529 bytes File Size = 859856 *
*Baskets : 3524 : Basket Size= 32000 bytes Compression= 131.14 *
…[/code]
Your Tree looks OK.
Could you send the shortest possible version of your program (filling the branches with random numbers) ?
Rene
Rene,
I am having difficulty creating a stand-alone program that demonstrates the problem I reported. - no variable length arrays yet - I will add these and see if that makes a difference.
Ed
Hi,
Was there any further discoveries for this thread? We are using AutoSave and finding that the processing time does increase as our TFile size increases. Have not yet done any extensive experimentation to see when this precisely occurs - but this thread sounds like a similiar problem.
Thanks,
Heather
The time for AutoSave must be constant in time. If you have any evidence of the contrary, please post the shortest possible running code reproducing the problem.
In particular, before posting your code, check for possible memory leaks on your side.
Rene
Hi Rene,
Perhaps if I provide some background. This is using v5.10.00 on Linux. We are processing some rather large test data files and finding that the processing time increases as we move through the data file, as well as the memory usage…I will admit we need to look to our own code for possible memory leaks. By default I had been using AutoSave every thousand events - and that is perhaps far too often for very large numbers of events. At the end of the run, the processing terminates with a segmentation fault and here is the backtrace:
#2 0x03eb9e84 in TDirectory::WriteKeys () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#3 0x03eb8db8 in TDirectory::SaveSelf () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#4 0x03eb8c04 in TDirectory::Save () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#5 0x03eb5e4a in TDirectory::Close () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#6 0x03ec0b0e in TFile::Close () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#7 0x03ebf67c in TFile::~TFile () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#8 0x03f424bf in TCollection::GarbageCollect () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#9 0x03f5206c in TList::Delete () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#10 0x03efde3d in TROOT::~TROOT () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
#11 0x03f0273d in __tcf_0 () from /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/rh9_gcc32opt/ROOT/v5.10.00/root/lib/libCore.so
I found an old post from 2004…perhaps it is not relevant to this case:
root.cern.ch/phpBB2/viewtopic.ph … save+reset
but it does make me wonder - though I’m not sure why TTree::Reset would be helpful just after closing the TFile…would it potentially prevent problems when deleting the TFile -which deletes the TTree as well? In reading, I see a key is generated at each call of AutoSave - could I call so often that I create too many keys and that is causing the failure in WriteKeys?
I will try to create a small example…it might help me to see where the problem is in any case.
Thanks,
Heather
Heather,
The overhead induced by AutoSave may become noticeable if you AutoSave at a non-sense rate. By non-sense rate, I mean, for instance, calling AutoSave every 1000 events and your tree will contain one billion events, ie generation one million AutoSave and as many TKeys !!
If you believe that your program is so unstable that it may crash at such a high rate, you should replace the call to tree.autoSave() by
tree.Write("",TObject::kOverWrite) or TObject::WriteDelete.
see TObject::Write for more details.
root.cern.ch/root/htmldoc/TObjec … ject:Write
Rene