Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822): TMVA

TMVA
1

Hi,

My name is Marta Ferrario and I am currently working on my thesis project using TMVA in Root. As a first step, I am preparing the DataLoaders to be used then to train a BDT: I split both signal and background samples in two folds, so I need to produce 2 DataLoaders. With the smaller samples everything runs smoothly, but when switching to the larger samples, I start encountering issues: first I fill the DataLoader with all signal/background training/test events, then I do PrepareTrainingAndTestTree, and when I try to save to a file (dataloader->Write()) I get repeated errors like:

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

Error in <TBufferFile::CheckCount>: buffer offset too large (larger than 1073741822)

Error in <TBufferFile::CheckCount>: buffer offset too large (larger than 1073741822)

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

prepare_dataloader : INFO : now prepare Training and Test tree ...

And at the end:

Error in <TRint::HandleTermInput()>: std::bad_alloc caught: std::bad_alloc

From what I understand, this seems related to memory limits or buffer size limitations when handling very large data sets. During the processing of the first fold, I already see multiple TBufferFile errors (bytecount too large, buffer offset too large), but the dataloader is still successfully saved. However, when the program proceeds to the second fold, it crashes with: Error in <TRint::HandleTermInput()>: std::bad_alloc caught: std::bad_alloc

Is there a way to solve this problem without reducing the number of events?

Thanks in advance for any suggestions.

2

Welcome to the ROOT Forum!
Did you try to search the forum for the same kind of problem? See for example:
https://root-forum.cern.ch/search?q=Error%20in%20%3CTBufferFile%3A%3AWriteByteCount%3E%3A%20bytecount%20too%20large%20order%3Alatest
And in particular: Writing large trees into files
If it doesn’t help, feel free to ask for more help