Thanks, indeed I was missing a Write(). I also noticed that using Close() before deleting the TDirectory was also enough, but perhaps it is not for each possible case.
However, aggressively closing and deleting every possible pointer does not seem to decrease the memory usage a lot. I have prepared a reproducer that can be run with just ROOT. I am sorry it’s a bit long, but it reflects what I am actually trying to achieve in my application. It basically creates some histograms and stores them in a file, but if they already exist there, it merges them.
#include <iostream>
#include <iomanip>
#include <TH1F.h>
#include <TFile.h>
#include <TSystem.h>
void reportMemory(std::string prefix) {
sleep(2);
ProcInfo_t pi;
gSystem->GetProcInfo(&pi);
std::cout << std::setw(40) << prefix << ": resident " << std::setw(6) << pi.fMemResident / 1000 << "MB, virtual " << std::setw(6) << pi.fMemVirtual / 1000 << "MB" << std::endl;
}
void test2()
{
reportMemory("test2 start");
{
auto file = new TFile("out.root", "UPDATE");
if (file->IsZombie() || !file->IsOpen() || !file->IsWritable()) {
std::cout << "failed to open the file" << std::endl;
return;
}
TObjArray* arr = new TObjArray();
arr->SetName("array");
arr->SetOwner(false);
reportMemory("created array");
// this aims to create histograms worth 1000 * 25000 * 4B = 100MB of memory
for (auto i = 0; i < 1000; i++) {
auto name = std::string("histo") + std::to_string(i);
TH1F* histo = new TH1F(name.c_str(), name.c_str(), 25000, 0, 10000);
histo->FillRandom("gaus");
arr->Add(histo);
}
reportMemory("filled array");
auto stored = file->Get<TObjArray>("array");
if (stored != nullptr) {
reportMemory("read another array from file");
TObjArray* coll = new TObjArray();
coll->SetOwner(false);
coll->Add(stored);
reportMemory("prepared a collection for merging");
arr->Merge(coll);
reportMemory("merged");
delete coll;
reportMemory("deleted wrapper collection");
stored->SetOwner(true);
delete stored;
reportMemory("deleted the stored array");
} else {
reportMemory("did not find another array in file");
}
file->WriteObject(arr, arr->GetName(), "Overwrite");
reportMemory("wrote array to file");
arr->SetOwner(true);
delete arr;
reportMemory("deleted array");
file->Write();
reportMemory("did file->write");
file->Close();
reportMemory("did file->close");
delete file;
reportMemory("deleted file");
}
reportMemory("test2 end");
}
int main(int argc, char **argv)
{
test2();
return 0;
}
What I observed is that Writing, Closing and deleting does not decrease the memory usage, as you can see in the output below:
$> root -l
root [0] .x test2.C
test2 start: resident 245MB, virtual 406MB
created array: resident 245MB, virtual 406MB
filled array: resident 347MB, virtual 505MB
did not find another array in file: resident 347MB, virtual 505MB
wrote array to file: resident 363MB, virtual 520MB
deleted array: resident 363MB, virtual 520MB
did file->write: resident 363MB, virtual 520MB
did file->close: resident 363MB, virtual 520MB
deleted file: resident 363MB, virtual 520MB
test2 end: resident 363MB, virtual 520MB
root [1] .x test2.C
test2 start: resident 363MB, virtual 520MB
created array: resident 363MB, virtual 520MB
filled array: resident 363MB, virtual 520MB
read another array from file: resident 450MB, virtual 607MB
prepared a collection for merging: resident 450MB, virtual 607MB
merged: resident 450MB, virtual 607MB
deleted wrapper collection: resident 450MB, virtual 607MB
deleted the stored array: resident 450MB, virtual 607MB
wrote array to file: resident 451MB, virtual 607MB
deleted array: resident 451MB, virtual 607MB
did file->write: resident 451MB, virtual 607MB
did file->close: resident 451MB, virtual 607MB
deleted file: resident 451MB, virtual 607MB
test2 end: resident 451MB, virtual 607MB
The memory usage stays the same with later executions of the same script. Also, it does depend on the object size, so it cannot be just libraries. I performed the same test by compiling this macro and the results are roughly the same - the initial usage is much lower, but then reaches similar values.
Would you know why the memory usage does not fall despite nicely deleting and closing the objects and the file? Is there a way to avoid it?