Controlling TTree buffering during readback

Hi,
I’m reading a series of large ROOT files (well, a few variables from them) over the network from a disk server. If I copy the file straight up I see about 22Mb/sec. When I run ROOT I see about 5-6 Mb/sec. The CPU of the ROOT process is running at about 1-2% (12% == 1 core on this machine). The difference in rates makes me think that the buffering/read ahead stradegy used by ROOT is non-optimal for this configuration. Are there any parameters I can play with to convince it to do better? I don’t have control, btw, over the format of the ROOT files, unfortunately, just their read back.

Cheers,
Gordon.

Hi Gordon,

Did you enable the TTreeCache (i.e. the read-ahead mechanism)? It is off by default.

Cheers,
Philippe.

No! :slight_smile: And thanks, I think I’d heard of this in the past, but had forgotten it. I will attempt it later this evening.

BTW, the docs in TTreeCache specify doing “AddBranch(”"…" not “AddBranchToCache(”"". That source code set of comments should probably be fixed. I assume that is all I need to do to enable the cache, right?

I think I want to use example 2a in the TTreeCache header docs. Now, I only read a small sub-set of the branches, and I know exactly which branches I need to read. How important is it for me to put those names in instead of the “*”? How much of a difference in performance will it make?

Many thanks!

Cheers, Gordon.

Two more things about the use of this:

  1. Lets say I do training for 200 events (with “*” set) and a new branch only gets accessed on event #310 because I have such efficient cuts. What happens?

  2. What governs the setting of teh cache size to 10 MB? Is that something I should muck with? If I know I’m on a computer with boat loads of memory, can I just set it to 100 MB? Any performance penalties expected (I’m doing testing as well on my own).

  3. Say I specify the branches I’m looking at (my tool is like TTree::Draw in that it knows exactly what is referenced). Lets say I have a bug and I don’t get one of the branches right. What happens then? I suppose this is the same as question #1.

  4. It wasn’t clear from your instructions in TTreeCache. Lets say I know exactly what branches will be used ahead of time. So I put them in with AddBranchToCache - the complete list. Shoudl I call SetLearnEntries after that and set it to zero? I went hunting for calls to AddBranchToCache in the tree player but failed to find them (I did see some calls to caching calls).

  5. What clean up do I need to do? Lets say I process only chains, and a new Chain is created for each query even if the files are the same. The input files are only opened by the TChain (I never open them in a TFile otehrwise). At the end of my run do I need to do something to the cache to “reset” it for the next query? And I’m always processing files start to finish (so I never set the entries the cache should be active over).

Many thanks! -Gordon.

Last qusetion (I think) for this evening. I turned on the cache both using the “*” and also by using specific named branchs. In each case it looks to me like I get the same improvement - about 10-15% or so. So that is real, especially on the longer runs! Thanks, and that was pretty easy!

Next, when in learn mode does it start to apply what it learns right away? For example, if it sees one branch is getting hit on every event, it can just start reading that ahead? :slight_smile: What I’m worried about is the setting for the # of learned entries - at 100. I have no idea how good that is, and it may well be good for one query, but no good for the next…

Cheers,
Gordon.

[quote]BTW, the docs in TTreeCache specify doing “AddBranch(”"…" not “AddBranchToCache(”"". That source code set of comments should probably be fixed.[/quote]Indeed. I update the doc.

[quote]I assume that is all I need to do to enable the cache, right?[/quote]Yes, this is the minimal requirement.

[quote]I think I want to use example 2a in the TTreeCache header docs. Now, I only read a small sub-set of the branches, and I know exactly which branches I need to read.[/quote]Then it is no longer 2a) you need by 3a)

[quote]How important is it for me to put those names in instead of the “*”? How much of a difference in performance will it make?[/quote]It really depends on the ratio of the total size of the branches you need compares to the total size all the branches. The smaller the ratio the more important it is as the TTreeCache will only read the data for the branches requested. (and thus if you use * with a small ration you end up reading much more than you need).

Cheers,
Philippe.

[quote]1) Lets say I do training for 200 events (with “*” set) and a new branch only gets accessed on event #310 because I have such efficient cuts. What happens?[/quote]Then the branch is not cached and its baskets will be read individual resulting in lower performance (especially if the file is remote).

[quote]2) What governs the setting of teh cache size to 10 MB? Is that something I should muck with? If I know I’m on a computer with boat loads of memory, can I just set it to 100 MB? Any performance penalties expected (I’m doing testing as well on my own).[/quote]The initial value is set by the size of a cluster which is determine by the value of AutoFlush as set when writing the file (the default value is 30Mb). In ROOT v5.30, when enabling the TTreeCache you can set it to any size you want and it will read that much data ahead (hence reducing the number of reads).

[quote]3) Say I specify the branches I’m looking at (my tool is like TTree::Draw in that it knows exactly what is referenced). Lets say I have a bug and I don’t get one of the branches right. What happens then? I suppose this is the same as question #1.[/quote]Yes, it is the same question and result :slight_smile:. The file will be read correctly but (slightly) slower than wanted.

[quote]4) It wasn’t clear from your instructions in TTreeCache. Lets say I know exactly what branches will be used ahead of time. So I put them in with AddBranchToCache - the complete list. Shoudl I call SetLearnEntries after that and set it to zero? I went hunting for calls to AddBranchToCache in the tree player but failed to find them (I did see some calls to caching calls).[/quote]No, you should call StopCacheLearningPhase.

[quote]5) What clean up do I need to do? [/quote]None as far as the TTreeCache is concerned (well unless the set of branches changes and the TTree/TChain is the same, in which case you need to disable and re-enable the cache.

Cheers,
Philippe.

[quote]Next, when in learn mode does it start to apply what it learns right away? [/quote]No.

[quote]For example, if it sees one branch is getting hit on every event, it can just start reading that ahead? What I’m worried about is the setting for the # of learned entries - at 100. I have no idea how good that is, and it may well be good for one query, but no good for the next…[/quote]It only start reading ahead after the 100th entry before that (currently) all the read are individual reads.

Cheers,
Philippe.

Hi,
Thanks for all the answers. From this I conclude that I should do the following things (for my situation):

  1. Use AddBranchToCache on all the branches I’m going to be accessing during the query
  2. Set the cache size to be 100 MB
  3. StopCacheLearningPhase
  4. call TChain::Process

This is because of my situation:

  • I know ahead of time exactly what branches my TSelector will access

  • I am on a machine with a lot of memroy

    I’m running on 5.28 right now but will upgrade to 5.30 shortly. Hopefully nothing above will cause unhappy things to happen there.

    Thanks a lot for your help!

    Cheers,
    Gordon.