For some context, I would like to do up to 100M unbinned maximum likelihood fits on relatively small datasets. I saw a drastic memory increase as the program ran and I created a program which I thought reproduced this (RooFit for loop fitTo explodes in memory usage). I was using the wrong command to pull memory information but was told to use ProcInfo_t. Its a short program so I appended it here. So I ran this program with Root version 6.23 and 6.24.
I measured the average memory increase(KB) per iteration with various options included in the fitTo command shown below. I would also like to access the fitResults so I use Save and I can choose to delete that pointer later in these tests.
Root 6.23
No fitting - 0.1
Save, Minos, BatchMode, PrintLevel - 10.7
Save, Minos, BatchMode, PrintLevel + delete fitPtr - 2.72
Save, PrintLevel + delete fitPtr - 2.57
Root 6.24
No fitting - 0.1
Save, Minos, BatchMode, PrintLevel - 72.7
Save, Minos, BatchMode, PrintLevel + delete fitPtr - 63.53
Save, BatchMode + delete fitPtr - 21.2
Save, Minos + delete fitPtr - 0.312
Save, PrintLevel + delete fitPtr - 0.096
Clearly doing 10**8 fits is not really feasible. What can I do to reduce the memory usage and complete these fits as fast as possible? I would like to use BatchMode but it seems to consume a lot of memory and not release it. Minos seems to increase usage by a factor of 3 which grows significantly if using BatchMode. Is there a way I can release the memory after I am done with each iteration? If not I could always restart root…
Thanks for your time
#include "RooRealVar.h"
#include "RooGaussian.h"
#include "RooPlot.h"
#include "RooTrace.h"
void test() {
RooTrace::active(kTRUE);
RooTrace::mark();
RooRealVar x("x","x",-10,10);
RooDataSet d("d","d",RooArgSet(x));
RooRealVar s("s","s",1,1,10);
RooRealVar m("m","m",0,-10,10);
RooGaussian g("gauss","gauss(x,m,s)",x,m,s);
RooPlot *f=x.frame();
for (Int_t j=0; j<300; ++j){
x.setVal(gRandom->Gaus(0,1));
d.add(x);
}
ProcInfo_t pinfo;
int startMemResident;
int endMemResident;
int niters=1000;
RooArgSet input=RooArgSet(x);
for (Int_t i=0; i<niters; i++) {
RooFitResult* roo_result = g.fitTo(d,RooFit::PrintLevel(-1),RooFit::Save(),RooFit::Minos(kTRUE),RooFit::BatchMode(true),RooFit::Save());//,RooFit::SumW2Error(true));//,RooFit::BatchMode(true));
x.setVal(0);
g.getVal(input);
gSystem->GetProcInfo(&pinfo);
std::cout << i << " memory usage " << pinfo.fMemResident << " " << pinfo.fMemVirtual << std::endl;
if (i==0)
startMemResident=pinfo.fMemResident;
if (i==(niters-1))
endMemResident=pinfo.fMemResident;
//delete roo_result;
}
int deltaMem=endMemResident-startMemResident;
double avgMem=(double)deltaMem/niters;
cout << endl << "Memory increase = " << deltaMem << endl;
cout << "Avg increase per iteration = " << avgMem << endl;
}