Writing large trees into files

[meyerma]

Hello,

I have been saving large trees recently and I am facing a memory issue while writing them.
My tree are around 1.8GB. (100M events)

I am used to create a tree fill it. So far, so good.
After playing with it I open a TFile, select/mkdir a TDirectory inside and I use TDirectory::WriteTObject to save at the right location.

For some reason, I managed to use TTree::ChangeFile and writing the file successfully, but this is not what I want. Would you have any suggestion how to proceed ?

Here is some

Fatal in <TBufferFile::AutoExpand>: Request to expand to a negative size, likely due to an integer overflow: 0x80007cf5 for a max of 0x7ffffffe.
aborting
[./6.28.04/lib/libCore.so] ErrorHandler (no debug info)
[./6.28.04/lib/libCore.so] TObject::Fatal(char const*, char const*, ...) const (no debug info)
[./6.28.04/lib/libCore.so] TBuffer::AutoExpand(int) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::WriteFastArray(char const*, int) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::WriteObjectClass(void const*, TClass const*, bool) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferIO::WriteObjectAny(void const*, TClass const*, bool) (no debug info)
[./6.28.04/lib/libCore.so] TObjArray::Streamer(TBuffer&) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) (no debug info)
[./6.28.04/lib/libRIO.so] int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) (no debug info)
[./6.28.04/lib/libRIO.so] TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::WriteClassBuffer(TClass const*, void*) (no debug info)
[./6.28.04/lib/libTree.so] TBranch::Streamer(TBuffer&) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::WriteObjectClass(void const*, TClass const*, bool) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferIO::WriteObjectAny(void const*, TClass const*, bool) (no debug info)
[./6.28.04/lib/libCore.so] TObjArray::Streamer(TBuffer&) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) (no debug info)
[./6.28.04/lib/libRIO.so] int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, TStreamerInfo::TCompInfo* const*, int, int, int, int, int) (no debug info)
[./6.28.04/lib/libRIO.so] TStreamerInfoActions::GenericWriteAction(TBuffer&, void*, TStreamerInfoActions::TConfiguration const*) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::ApplySequence(TStreamerInfoActions::TActionSequence const&, void*) (no debug info)
[./6.28.04/lib/libRIO.so] TBufferFile::WriteClassBuffer(TClass const*, void*) (no debug info)
[./6.28.04/lib/libRIO.so] TKey::TKey(TObject const*, char const*, int, TDirectory*) (no debug info)
[./6.28.04/lib/libRIO.so] TFile::CreateKey(TDirectory*, TObject const*, char const*, int) (no debug info)
[./6.28.04/lib/libRIO.so] TDirectoryFile::WriteTObject(TObject const*, char const*, char const*, int) (no debug info)

Cheers, M.

[meyerma]

Hi @pcanal , I have been able to reduce the size of my tree and I end up now with errors like:

		Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)
		Error in <TBufferFile::WriteByteCount>: bytecount too large (more than 1073741822)

Would you maybe have some idea how to fix this ?
I couldn’t figure out based on previous similar threads.

[pcanal]

Hi,

This is unfortunately a hard limit (that we are currently working on lifting) and the only work-around is to use ChangeFile (or equivalent) to split the data in multiple files (well technically multiple TTree).

Cheers,
Philippe.