Error while writing RooWorkspace on a disk

Roofit and RooStats
1

Dear RooFit experts,

I’m trying to store RooWorkspace in TFile on a disk but it keeps failing with an error message [1]. Let me elaborate a bit on the situation.

In my workspace, there are pdfs with roughly 5k parameters and the size of data and template histograms is about 400 MB. I have no problem to make it in memory, but when I call writeToFile, an output file appears after several minutes but it only contains ProcessID. And roughly more than 10 hours later it starts to print out the error message. It keeps repeating the same message and never terminates.

We checked that the problem doesn’t lie in individual histograms but there seems to be a threshold on a certain quantity proportional to the number of histograms. In addition, we are now using PyROOT, version 6.12/07 but my colleague checked that the same error occurs at 6.22.

Any comments or suggestions for further investigation would be really appreciated.

[1] Error in TBufferFile::WriteByteCount: bytecount too large (more than 1073741822)

2

Your problem seems related to Roofit and/or I/O. Our experts in these to fields are @moneta and @pcanal . They may have some ideas about this issue.

3

@moneta @pcanal - can any of you please help? Thank you!

4

ROOT Forum → Search → 1073741822

5

Just putting a bit more on the situation, any entity in my workspace and the workspace itself are smaller than 1GB. So I think the TBufferFile::WriteByteCount error is just a consequence of something occurring at writeToFile. We will provide more details if you could tell us how to examine writeToFile in runtime at lower levels.

6

Hi,
Please can you post your code and data so we can reproduce it. there could be something we can improve to avoid TBuffer size limit of 1GB.

Best regards

Lorenzo

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

8

Hello,

After investigating the issue using the provided inputs, there is no simple solution in the current ROOT version to avoid this 1GB limit.
The best solution is optimise the class to be imported in the RooWorkspace and minimise the memory usage. For example the FastHisto class of the CMS combine, used for the morphing, stores in addition to bin content values, also bin edges and bin widths. These could be re-computed at run time for the common case of fixed bins. Also in case of variable bins, one stores in the pdf class a vector of FastHisto and I presume one does not need to repeat the bin edges and bin widths.

Otherwise a possible workaround would be to store the pdf’s at one level down in the tree in separate workspace’s and importing them later in a larger workspace that is kept only in memory.

It is however planned to remove this 1GB limit in future ROOT versions

Lorenzo