The error message tells you the fix: you should create your TFile BEFORE creating the TTrees. If the TTrees are created first, they exist in memory (hence the “memory-resident”) part, but because TTrees can get quite large, your program can essentially run out of memory.
The solution is to create the TFile to which you want to save the TTrees before creating the TTrees, then they will be automatically attached and become non-memory-resident. When the TTrees get large enough, the data will be dumped to disk rather than being held in memory.
Jean-François