There seems to be some discussion about a similar error before but was quite some time ago and was resolved (2012)
https://root-forum.cern.ch/t/memleak-when-fitting/13745
Everytime I call fitTo there seems to be various roofit objects that are created and are not cleaned after every iteration. I cannot get RooTrace::dump() to output anything but I am able to use RooTrace::printObjectCounts() to output the object counts. It seems fitTo creates a bunch of RooAbsCollection objects that are not cleaned which starts to consume all the memory. I can see the resident memory usage go up when looking at the top command. I put together a small program that I think shows this problem.
Tried with root 6.08 and 6.23.
testMemoryUsage.zip (761 Bytes)