TBranch errors when deserializing TTree

Dear experts,

I have a problem related to the deserialization of a TTree from a coral::Blob container. This I implemented some time ago following the discussion in Serialize and deserialize TTree into coral::Blob. Back then I tested it with a big TTree object and things seemed fine to me, though I cannot remember exactly if I looped through all events or not – something I’m doing now with the implementation of the function in the final code.

So to recap, the coral::Blob object, when retrieved from Oracle, merely fills some space in memory with a byte array which, interpreted as a TMemFile, makes it possible to retrieve the encoded TTree. The function to retrieve looks like this:

bool readBlobAsTTree(const coral::Blob &blob, TTree*& out){
    std::string sb = reinterpret_cast<const char*>(blob.startingAddress());
    std::vector<unsigned char> bdata = base64_decode(sb);
    TMemFile f("buffer", reinterpret_cast<char*>(bdata.data()), static_cast<uLongf>(blob.size()));
    TTree* t = (TTree*) f.Get("tree");
    TTree* tt = t->CloneTree();
    tt->SetDirectory(0);
    f.Close();
    out = tt;
    return true;
}

The line TTree* tt = t->CloneTree(); actually was not in the first implementation following the thread linked above, which is strange because without it, the returned TTree does not make any sense. It contains the correct number of entries (1048576) but the values of the branches change with every iteration, there’s clearly a memory problem. Cloning the tree, gives the correct values per event, however, for selected branches and a number of entries (and different branches per entry, it’s not always the same few branches), it gives me the following errors:

Error in <TBranch::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranch::TBranch::Fill>: Failed to write out basket.

and also

Error in <TTree::Fill>: Failed filling branch:tree.vmm, nbytes=-1, entry=1045642
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)

Reading the TTree with pyROOT does not give me the errors and all seems to work fine.

Now sharing the code is a bit difficult since this is part of a whole framework. Hence I would like to ask if there’s something obvious which I’m doing wrong and which you can spot already from the function itself? If not, I’ll try to figure out how I can make the code accessible to you.

Thank you very much and regards,
heico

Hi @heico,

IIUC, you retrieve a base64-encoded blob, which you then decode. To me, it seems that the size of the TMemFile -see below- should be that of the decoded data, i.e. bdata.size(). In any case, blob.size() should be larger and should not be causing any issues besides larger memory usage, I think.

OTOH, I don’t know why CloneTree() is needed, but is maybe related to the error you are getting. I am going to invite @pcanal to this topic, as his knowledge will be useful here.

Also, although not important for your issue, the std::string sb = ... assignment might imply an additional memory copy.

Cheers,
J.

Dear @jalopezg,

you are right, it should be bdata.size(). I changed it but the error still persists, and the errors appear when adding the call to CloneTree(). If I don’t add this line, I get a seg fault because the TTree obviously is tried to be accessed outside the function afterwards, but the function does not return a valid TTree. So maybe that’s the original issue?

Thank you!
heico

When doing:

The clone get associated (and attempted to be written to) with the TMemFile (the ‘current directory’ at creation time). Since the TMemFile was open read only, you get the expected write error. But even if the TMemFile was open in update mode, the CloneTree would succeed but eventhough TTree object would not be delete when the file is close (thanks to the SetDirectory(0);) The data is points to would be gone when the TMemFile goes away.

the returned TTree does not make any sense. It contains the correct number of entries (1048576) but the values of the branches change with every iteration, there’s clearly a memory problem.

Yes the TTree object (returned by Get) is deleted when executing f.Close(). If you call SetDirectory(nullptr) on that object, you prevent its deletion at file closure but the memory holding the data would still disappear as the TMemFile is deleted.

To solve this problem try:

    TMemFile f("buffer", reinterpret_cast<char*>(bdata.data()), static_cast<uLongf>(blob.size()));
    TTree* t = (TTree*) f.Get("tree");
    t->LoadBaskets(); // or (large_number_greater_than_2gb) if you more than 2GB of data.
    t->SetDirectory(0);
    f.Close();
    out = t;

The call to LoadBaskets will reads of the (compressed) data from the TMemFile and thus the TTree becomes independent from it (mostly, don’t call DropBasket and they (of course) wont be able to be re-retrieved).

Cheers,
Philippe.

Dear Philippe,

this is such a great answer and explanation, thank you so very much!
I tried your solution and it seems to work perfectly on first sight. Later today I will integrate the new function in a few other calls in my framework and test it better. So unless I come back whining about something new I don’t understand, I think this is solved.

Thank you again!
heico

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.