Reading histograms from TFile gets slower and slower

ROOT
1

Hi,

I currently observe the problem that reading histograms (TH1I) from a Root file gets slower and slower. The code snippet would be

void slowerAndSlower(TString fname)
{
  TFile *f = new TFile(fname);

  TDirectoryFile *traces = (TDirectoryFile*) f->Get("Strips");
  TList *trlist = traces->GetListOfKeys();
  int n = trlist->GetSize();
  
  TStopwatch w;
  double t0=0, t1=0;
  for (int i=0; i<n; ++i) {
    TString name = trlist->At(i)->GetName();
    TH1I *h = (TH1I*)traces->Get(name);
    if ((i+1)%1000==0) {
      t1 = w.CpuTime()*1000;
      printf("time for 1000 objects: %4.0f ms\n", t1-t0);
      t0 = t1;
    }
    w.Continue();
  }
}

with the output

time for 1000 objects:   60 ms
time for 1000 objects:  110 ms
time for 1000 objects:  170 ms
time for 1000 objects:  210 ms
time for 1000 objects:  280 ms
time for 1000 objects:  320 ms
time for 1000 objects:  390 ms
time for 1000 objects:  450 ms
time for 1000 objects:  510 ms
time for 1000 objects:  560 ms
time for 1000 objects:  620 ms
time for 1000 objects:  660 ms
time for 1000 objects:  750 ms
time for 1000 objects:  810 ms
time for 1000 objects:  880 ms
time for 1000 objects:  940 ms
time for 1000 objects: 1000 ms
time for 1000 objects: 1070 ms
time for 1000 objects: 1130 ms
time for 1000 objects: 1190 ms
time for 1000 objects: 1240 ms
time for 1000 objects: 1250 ms
time for 1000 objects: 1300 ms
time for 1000 objects: 1380 ms

When looking to the times, it looks as rather a quite constant additional time adds for each next bunch of 1000 objects, as if ROOT would always read the file from the beginning again and again.

The file contents look like (if this is important)

TFile**		file1.root	
 TFile*		file1.root	
  KEY: TDirectoryFile	Strips;1	Strips
  KEY: TDirectoryFile	QDC;1	QDC
  KEY: TDirectoryFile	VFTX;1	VFTX
  KEY: TDirectoryFile	scaler;1	scaler
  KEY: TDirectoryFile	mTDC;1	mTDC

I don’t know whether it’s an effect of the way I read the objects. Does anybody know how to fix this?

Best and thanks,
Klaus

Please read tips for efficient and successful posting and posting code

ROOT Version: v6-20-02
Platform: Linux Mint
Compiler: Not Provided

2

Try:
TH1I *h = (TH1I*)((TKey*)trlist->At(i))->ReadObj();

3

@pcanal Thanks for the quick reply. It gets better, but still looks like:

time for 1000 objects:   40 ms
time for 1000 objects:   40 ms
time for 1000 objects:   50 ms
time for 1000 objects:   70 ms
time for 1000 objects:   70 ms
time for 1000 objects:   70 ms
time for 1000 objects:  100 ms
time for 1000 objects:  110 ms
time for 1000 objects:  130 ms
time for 1000 objects:  120 ms
time for 1000 objects:  130 ms
time for 1000 objects:  150 ms
time for 1000 objects:  190 ms
time for 1000 objects:  230 ms
time for 1000 objects:  230 ms
time for 1000 objects:  250 ms
time for 1000 objects:  230 ms
time for 1000 objects:  270 ms
time for 1000 objects:  270 ms
time for 1000 objects:  310 ms
time for 1000 objects:  310 ms
time for 1000 objects:  250 ms
time for 1000 objects:  270 ms
time for 1000 objects:  260 ms
4

Inside of the “for” loop, do not create the “name” string and add: delete h;

5

The for loop now looks like

  TStopwatch w;
  double t0=0, t1=0;
  for (int i=0; i<n; ++i) {
    TH1I *h = (TH1I*)((TKey*)trlist->At(i))->ReadObj();
    if ((i+1)%1000==0) {
      t1 = w.CpuTime()*1000;
      printf("time for 1000 objects: %4.0f ms\n", t1-t0);
      t0 = t1;
    }
    delete h;
    w.Continue();
  }

and I basically don’t see any difference.

6

@pcanal I don’t really know why it is so slow with “trlist->At(i)” but I think I found a fast way:

  // ...
  Int_t i = 0;
  for (const auto && obj : *trlist) {
    auto h = (TH1*)((TKey*)obj)->ReadObj();
    i++;
    // ...

BTW. I guess one can get the same fast way when using a TList iterator TListIter (via the wrapper class TIter) as described e.g. in the TList class reference.

7

I don’t really know why it is so slow with “ trlist->At(i) ”

Since this is a list, calling At(i) to loop over a large list is a disaster. There is no caching involved so it always has to iterate over all the element from 0 to i thus the contstant time increase.

Using the an iterator is the right solution.

8

Hi guys,

thanks for the help and the explanations,

Actually I’m not a big expert using iterators but already found the suggestion in the TList documentation and tried with:

  TStopwatch w;
  double t0=0, t1=0;
  int i=0;
  TIter inext(trlist);
  while (TKey *key = (TKey*)inext()) {
    TString name = key->GetName();
    TH1I *h = (TH1I*)traces->Get(name);
    if ((i+1)%1000==0) {
      t1 = w.CpuTime()*1000;
      printf("time for 1000 objects: %4.0f ms\n", t1-t0);
      t0 = t1;
    }
    w.Continue();
    delete h;
    i++;
  }

which delivers again the very slow version from the top.

I think the problem is not only trlist->At(i), but also traces->Get(name). Since the TH1I are merged from many files, they don’t have unique names, but only the TKeys have unique names (no clue how that works inside a ROOT file). So there is no list connecting the histograms, but only the TKeys, right? And this means, there is no way to get rid of both the Get(name) and ->At(i), or?

Best,
Klaus

9

Ah, now with Wile’s method

  for (const auto && obj : *trlist) {
    TString name = ((TKey*)obj)->GetName();
    TH1I *h = (TH1I*)((TKey*)obj)->ReadObj();

it is fast! Thanks a lot!!

10

Try your iterator’s example with: TH1I *h = (TH1I*)key->ReadObj();

11

Works as well with the same speed, thanks for improving my understanding of TKey and iterators! :slight_smile:

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.