I’m using proof with a TChain with 345 files for a total of 44 Gb. During the validation (when proof says: “Validating files x out y”) the memory used grows too much until 50 Gb!!! How to avoid this?
Hi,
Where? On the master? On the client?
Gerri
[quote=“ganis”]Hi,
Where? On the master? On the client?
Gerri[/quote]
50 Gb is the total used memory summed on a system of 3 machines. Evey machines has 16 Gb of RAM for a total of 16*3 = 48 Gb.
ROOT 5.28/00
Hi,
I do not see what could leak, unless there is something leaking while getting the entries of the tree.
Could you try by creating subchains with 10, 20, 30 files and see if you get a linear increase?
What happens if you run TChain::GetEntries() locally (in the ROOT session on the client)? Do you see the leak also there?
Gerri
[quote=“ganis”]Hi,
I do not see what could leak, unless there is something leaking while getting the entries of the tree.
Could you try by creating subchains with 10, 20, 30 files and see if you get a linear increase?
What happens if you run TChain::GetEntries() locally (in the ROOT session on the client)? Do you see the leak also there?
Gerri[/quote]
What do you mean by subchains?
I changed system, using a desktop computer with 6 Gb RAM and 4 core, to look more in the detail at the memory used.
I tried with a chain with all the files (~350 for a total of 44 Gb), when I call GetEntries from an interactive console I can see at the beginning the memory used going up and down with variation of 100 Mb, not too much (see init1.png). At a certain point the variation becomes bigger probably because files are very different in size (see init2.png). After 3 minutes I killed the command.
With proof the memory used is bigger, maybe because I’ve 4 core it process 4 files at the same time (see init_proof.png). At some point proof use 70% ~ 4 Gb only to get the number of entries! Isn’t it too much?
Hi,
With which version of ROOT where the file written? Do you really need to call GetEntries on all the files?
Philippe.
[quote=“pcanal”]Hi,
With which version of ROOT where the file written? Do you really need to call GetEntries on all the files?
Philippe.[/quote]
ROOT 5.28/00. I don’t call GetEntries, I call Proof::Process, and I suppose it call GetEntries
[quote]ROOT 5.28/00. I don’t call GetEntries, I call Proof::Process, and I suppose it call GetEntries[/quote]Fair enough. What does you Process function do? Do you see the same problem with an ‘empty’ selector’? Do you see a similar problem when running with Proof?
Philippe.
[quote=“pcanal”][quote]ROOT 5.28/00. I don’t call GetEntries, I call Proof::Process, and I suppose it call GetEntries[/quote]Fair enough. What does you Process function do? Do you see the same problem with an ‘empty’ selector’? Do you see a similar problem when running with Proof?
Philippe.[/quote]
I think that the problem is when I (or proof) call GetEntries, as you can see from my previous post on Jan 05.
[quote=“pcanal”][quote]ROOT 5.28/00. I don’t call GetEntries, I call Proof::Process, and I suppose it call GetEntries[/quote]Fair enough. What does you Process function do? Do you see the same problem with an ‘empty’ selector’? Do you see a similar problem when running with Proof?
Philippe.[/quote]
My question is related with this one: Why proof use all my memory (during process)
where you can find an example file. Ganis told that the problem is during GetEntries()
Fair enough. I will continue the discussion on this other thread.
Philippe.