At a quick look it seems to me that SCycleOutput and SCycleBaseHist do what they have to do.
This said, merging of objects other then histograms - e.g. SCycleOutput - has always been done one-by-one (I mean, in all ROOT versions).
The object merged is deleted by PROOF right-after.
What happens to the object cloned in SCycleBaseHist::Book ?
I should perhaps setup a simple SFrame job just doing histogram booking and merging …
please find in attachment a tar file with a release of SFrame where i added some more histograms
to the example code. Setup ROOT, then from Sframe/ do “make”. To run do “sframe_main user/config/FirstCycle_config.xml”. By default it runs PROOF-lite, to run on a PROOF cluster change
in the xml file where you find “lite” with “username@proof_master:port”.
The code runs on ~12000 events from a single file and does 1000 TH1F. This is done inside a simple loop
so you can increase this number easily for testing.
best,
Max
p.s the input file is on cern afs. SFrame.tgz (310 KB)
i just noticed that i forgot to tell you that before compiling SFrame you need to source setup.sh from SFrame/ dir. Please let me know in case you have problems trying this example code.
This way it’s obvious to the user that they don’t have to care about memory management here. The Clone() function is used to put the histograms into the right directory.
anyway the memory problem persists even in 5.28.00f, where merging improvements were expected.
If also the SCycleOutput are merged 1-by-1 and deleted then i don’t understand why during merging
the proofserv.exe application grows in memory up to more than 2.5gb giving then the usual errors.
Any idea? Gerardo, did you find the time to look at the example code?
i finally managed to test the code with ROOT 5.32.02 and the i found again the same behavior.
The proofserv.exe job starts to eat a lot of memory (more that 20 GB!) in the merging phase when
i setup many histograms (order of 60000 TH1D).
Since, afaiu, Gerardo checked with many TH3 and didn’t observe this effect, i must conclude
this is a “feature” of SFrame, anyway i don’t know how to test this. Any suggestion is more that welcome!
looking at release notes for latest ROOT version 5.34, i saw:
“In TProofOutputFile, add switch to control the way histograms are merged by TFileMerger, i.e. one-by-one or all-in-one-go. The default is one-by-one which requires much less memory. Merging in one-go (the previous default) can be activated by passing ‘H’ in the constructor options.”
I was a bit surprised, since i thought one-by-one was already the default, but ok i probably misunderstood. Anyway i then tested my code in PROOF with this ROOT version but again i found a huge memory consumption
when merging (~30 GB).
I tried to get a feeling to how much is the memory filled my histograms. I took a file made running in steps
and loaded in ROOT:
I’ve 55402 histograms and they fill about 600 MB of memory. So if the merging would keep in memory only the sum and the next-to-be-merged histograms then it should require about 1.2 GB. So i’m a bit puzzled about the effective way the merging is working… or is working within SFrame that i know it’s not a ROOT-only related question.
I posted here again also to keep a track of the tests i’m doing but if you (Gerri, Attila) prefer we can continue the discussion offline, of course it would be fine.
To me it’s just important we continue the discussion…