RooFit memory increase per fitTo iteration

[lan13005]

For some context, I would like to do up to 100M unbinned maximum likelihood fits on relatively small datasets. I saw a drastic memory increase as the program ran and I created a program which I thought reproduced this (RooFit for loop fitTo explodes in memory usage). I was using the wrong command to pull memory information but was told to use ProcInfo_t. Its a short program so I appended it here. So I ran this program with Root version 6.23 and 6.24.

I measured the average memory increase(KB) per iteration with various options included in the fitTo command shown below. I would also like to access the fitResults so I use Save and I can choose to delete that pointer later in these tests.

Root 6.23
No fitting - 0.1
Save, Minos, BatchMode, PrintLevel - 10.7
Save, Minos, BatchMode, PrintLevel + delete fitPtr - 2.72
Save, PrintLevel + delete fitPtr - 2.57

Root 6.24
No fitting - 0.1
Save, Minos, BatchMode, PrintLevel - 72.7
Save, Minos, BatchMode, PrintLevel + delete fitPtr - 63.53
Save, BatchMode + delete fitPtr - 21.2
Save, Minos + delete fitPtr - 0.312
Save, PrintLevel + delete fitPtr - 0.096

Clearly doing 10**8 fits is not really feasible. What can I do to reduce the memory usage and complete these fits as fast as possible? I would like to use BatchMode but it seems to consume a lot of memory and not release it. Minos seems to increase usage by a factor of 3 which grows significantly if using BatchMode. Is there a way I can release the memory after I am done with each iteration? If not I could always restart root…

Thanks for your time

#include "RooRealVar.h"
#include "RooGaussian.h"
#include "RooPlot.h"
#include "RooTrace.h"

void test() {
    RooTrace::active(kTRUE);
    RooTrace::mark();
    RooRealVar x("x","x",-10,10);
    RooDataSet d("d","d",RooArgSet(x));
    RooRealVar s("s","s",1,1,10);
    RooRealVar m("m","m",0,-10,10);
    RooGaussian g("gauss","gauss(x,m,s)",x,m,s);
    RooPlot *f=x.frame();

    for (Int_t j=0; j<300; ++j){
        x.setVal(gRandom->Gaus(0,1));
        d.add(x);
    }

    ProcInfo_t pinfo;
    int startMemResident;
    int endMemResident;
    int niters=1000;
    RooArgSet input=RooArgSet(x);
    for (Int_t i=0; i<niters; i++) {
        RooFitResult* roo_result = g.fitTo(d,RooFit::PrintLevel(-1),RooFit::Save(),RooFit::Minos(kTRUE),RooFit::BatchMode(true),RooFit::Save());//,RooFit::SumW2Error(true));//,RooFit::BatchMode(true));
        x.setVal(0);
        g.getVal(input);

         gSystem->GetProcInfo(&pinfo);
         std::cout << i << "  memory usage " << pinfo.fMemResident << "  " << pinfo.fMemVirtual << std::endl;
         if (i==0)
             startMemResident=pinfo.fMemResident;
         if (i==(niters-1))
             endMemResident=pinfo.fMemResident;
         //delete roo_result;
     }

    int deltaMem=endMemResident-startMemResident;
    double avgMem=(double)deltaMem/niters;
    cout << endl << "Memory increase = " << deltaMem << endl;
    cout << "Avg increase per iteration = " << avgMem << endl;
}

[bellenot]

Maybe @jonas can help you on this

[jonas]

Hi @Ian13005, thanks for this report! Indeed, there was a memory leak in the batch mode that I fixed now in 6.24 and the 6.26 development branch (see PR #8142 and #8160).

Concerning the remaining memory increase, I have to investigate a bit more because it is related to the MemPoolForRooSets mechanism. What happens here is that there is a memory increase once in a while because a new memory pool arena is created. This is done to avoid reusing memory locations for RooArgSets, so we can be sure the memory address is like a unique ID.

However, when there are no pointers to a memory arena anymore it should free it’s memory, which doesn’t seem to happen for some reason. I will come back to you when I fully understand what’s going on, but rest assured these memory issues are worked on.

Cheers,
Jonas