Storing larger data in TClonesArray?

[scharf]

Hi,
I’m trying to read large data files with many individual measurements, storing each individual measurement in a TH1F. The TH1F are arranged in a TClonesArray.
I would like to store the data in memory because I have to first determine quantities from the combined dataset before I can analyse the individual measurements and I would like to avoid reading the data (<1 gb) in several times. I’m am not sure if my approach is at all suitable to achieve this goal.

Currently I use a class for reading which looks like this:

ReadData::ReadData(string path) {
	rundata = new TClonesArray("TH1F", 100000);

	FILE* pFILE = fopen(path.c_str(), "rb");
	for (int i = 0; i < 100000; i++) {
		TH1F* hCh = (TH1F*)rundata.ConstructedAt(i);
		hCh->SetBins(1024, 0, 1);

		float values = 0.;
		int nitem;
		for (int j = 0; j < 1024; j++) {
			nitem = fread(&values, sizeof(float), 1, pFILE);
			hCh->SetBinContent(j + 1, val);
		}
	}
	fclose(pFILE);
}

ReadData::~ReadData() {
	rundata->Clear();
}

TH1F* ReadData::Getmeasurement(int eventnr) {
	TH1F* his;
	his = (TH1F*)rundata->At(eventnr);
	return his;
}

In reality it loops over many smaller files with a total of currently 100,000 individual measurements with 1024 values each. It works fine for up to a few 10,000s of measurements but crashes for more. I assume the memory usage gets too high (crashes when memory usage is ~1.7 gb). If I add errors to the histograms the maximum number of measurements is significantly reduced.

My questions:

1. Am I making some stupid mistakes here?

2. Is there a better way to do this?

Thanks!

ROOT Version: 6.20/02
Platform: windows 10

[moneta]

Hi,
This could be related to the 1Gb limit on the buffer size. @pcanal should give you a more precise answer.

Cheers

Lorenzo

[scharf]

Hi,

is there a way to increase the buffer size? Since the binary data is less than 1 Gb I’d assume it should be possible to store it all in the buffer?

Sometimes I get the following error:

CreateFileMapping: returned 8: Not enough memory resources are available to process this command.
Error in <TInterpreter::TCling::AutoLoad>: failure loading library libWin32gdk.dll for TGWin32
CreateFileMapping: returned 8: Not enough memory resources are available to process this command.
No stack trace: cannot find (functions in) dbghelp.dll!