SWAP memory and roofit

ROOT
1

Dear all,
I’m interested in the usage of SWAP memory by RooFit @ generation time.
Currently I have troubles running jobs on large statistics because they
fail using a too large amount of SWAP memory.
To generate 40k Events with my PDFs I use 500Mb of swap memory.
If I go to 400k I do exceed the 3Gb limit and the job is killed.

How can I contain the usage of SWAP memory when generating the dataset?
What is taking all that memory?
[the use seems to scale nearly linearly with the stat. to be generated]

Cheers.
alessio

2

Hi Alessio,

I general memory consumed during generation is used to hold
the events you generate. For multidimensional p.d.fs that
do not provide efficient internal generator implementations a
large number of initial samples need to be generated to find the
maximum of the function

Looking at your numbers, I see some odd behaviour though. It adds up to about 13K per events which sounds like too much, which might indicate a memory leak somewhere in the generator code. The actual generator code used depends very much on what p.d.f exactly you’re using. Can you send a more details on your p.d.f?

Wouter

3

Dear all,
months passed since last posting.
I was trying to understand what causes such big SWAP memory usage on my laptop when running my fit.
I remind you the numbers: 13kB memory for each event. I need to generate 400k and this means I have no chance…( > 5GB )…

Wouter suggested a memory leak but I did run the code trough valgrind with memcheck tool. And the output is rather puzzling.
I generated 5, 15, 25, 50 and 100 events. But the summary does not change significantly (see below).
While I do see a significant change in the resources taken from the executable with the system monitor running…

Anybody has a suggestion on how to debug this problem. I really need to fix this problem to make decent studies…

I attach a full zipped log for futher investigations…

Cheers,
alessio

5ev:
memgrind.out.6423:==6586== definitely lost: 60,386 bytes in 1,279 blocks.
memgrind.out.6423:==6586== indirectly lost: 3,476,531 bytes in 16,409 blocks.
memgrind.out.6423:==6586== possibly lost: 1,499,182 bytes in 19,754 blocks.

15 ev:
memgrind_15.out.6423:==6680== definitely lost: 60,034 bytes in 1,288 blocks.
memgrind_15.out.6423:==6680== indirectly lost: 3,522,564 bytes in 16,661 blocks.
memgrind_15.out.6423:==6680== possibly lost: 1,501,194 bytes in 19,983 blocks.

25 ev:
memgrind_25.out.6423:==6764== definitely lost: 60,054 bytes in 1,288 blocks.
memgrind_25.out.6423:==6764== indirectly lost: 3,469,652 bytes in 16,306 blocks.
memgrind_25.out.6423:==6764== possibly lost: 1,518,541 bytes in 20,064 blocks.

50 ev:
memgrind_50.out.6423:==6823== definitely lost: 60,362 bytes in 1,302 blocks.
memgrind_50.out.6423:==6823== indirectly lost: 3,485,915 bytes in 16,427 blocks.
memgrind_50.out.6423:==6823== possibly lost: 1,525,903 bytes in 20,036 blocks.

100 ev:
memgrind_100.out.6423:==7041== definitely lost: 59,898 bytes in 1,278 blocks.
memgrind_100.out.6423:==7041== indirectly lost: 3,493,495 bytes in 16,448 blocks.
memgrind_100.out.6423:==7041== possibly lost: 1,515,555 bytes in 20,045 blocks.
memgrind_100.out.6423.gz (42.6 KB)