Hi !
I have a little problem handling TClonesArrays in Python.
In my own Event class I have a TClonesArray that contains a list of Candidate Objects.
I am creating a new TFile in Python with a new TTree that holds my custom events and I can copy all events from the old file to the one I created in python. No Problem so far.
I can also write a fancy filter that writes out less events than in the original sample.
What I would like to do, however, is to write events with a modified TClonesArray, i.e. one where only one of the original entries survives.
I can interactively look at a random event, and do a TClonesArray.RemoveAt() and even iterate ofer the resulting TClonesArray and it gives me only the surviving items.
However, trying to write an event after performing a RemoveAt on some members of the list seems to fail.
Unfortunately, the TMatrixD magic doesn’t seem to work with TClonesArray. I would like to perform something like a Clear followed by a list[0] = survivingCandidate.
What would be a good way to do that ?
The traceback is attached. You can see that it fails at TFile::Fill()
not sure why you get a crash, but there may be some problems with object ownership. I think the (default) memory heuristics that PyROOT uses work just fine for TClonesArray, but maybe your script proves me wrong. Can you give me a simple example that reproduces the problem?
ev = JansEvent()
inf = TFile(‘x.root’)
int = inf.Get(‘postCut’)
int.SetBranchAddress(‘event’, AddressOf(ev))
f = TFile(‘new.root’, ‘recreate’)
t = TTree(‘t’, ‘Test’)
t.Branch(‘event’, ‘JansEvent’, ev, 32000, 99)
int.GetEvent(0) #ev.bList.RemoveAt(1)
t.Fill()
t.Write()
f.Close()
[/code]
Commenting out the RemoveAt line results in the crash.
Hmmm. Should have posted to the Bug database probably. Sorry.
[edit] You may have to comment out the GClonesArray.hh include.
The example should work fine without it [/edit]
ok, back at this one: it took me a while to realize that not only the python, but also the equivalent .C macro crashes. I.e., I shouldn’t be searching for a problem in PyROOT, but in ROOT itself. It’s here (src/Tree/TBranchElement.cxx), as per your traceback:
TClonesArray *clones = (TClonesArray*)fObject;
if (!clones) return;
Int_t n = clones->GetEntriesFast();
fInfo->WriteBufferClones(b,clones,n,fID,fOffset);
Iow, once you do a RemoveAt(), (or python del, since that subsquently calls RemoveAt() again), GetEntriesFast() gives the wrong size and writing the TClonessArray en bloc causes a segfault.
A solution is to call ev.bList.Compress() just before writing (or after each RemoveAt()). After that, the writing of the TClonesArray as a block of course works.
As for “list[0] = survivingCandidate” or something similar: it’ll need to call placement new, since we’re dealing with a TClonessArray. I’m unsure how to do that generally.
interesting fact: Calling Compress() on the TClonesArray gets rid of the crash, but the next call to an entry in the array in the next event fails to cast it down.
This happens when I use setBranchAddress to my custom class as well as when using TChain.event
that error I can only reproduce by accessing an unoccupied slot in the TClonesArray. Not sure how I can handle the error more appropriately: it is, after all, not quite an our of range error. Maybe an IndexError exception will do?
I am not sure I understand what you mean.
The problem I am having is that calling Compress() causes this Error Message on a later event. Without Compress() everything runs well. I can investigate further, but what I suspect naively is that Compress reduces the available number of slots and when I read an event with more than the changed number of available slots, I get an error. But I know nothing about the internals of TClonesArray, so this is just wild speculation. I can try to investigate further when I find some time.
Cheers,
Jan
PS: Is there another function in TCLonesArray that I should call after writing the event to disk and before reading in the next event or before accessing the candidates in the next event that could remedy this ?
You should never call TClonesArray::Compress while writing a Tree (or you have to set addresses again).
At the end or beginning of each event, call TClonesArray::Clear
Hm.
Calling TClonesArray::Clear() before the fill causes the same problem as TClonesArray::Compress(). The TObject isn’t cast down in the next event.
Traceback (most recent call last):
...
AttributeError: 'TObject' object has no attribute 'gamma'
Calling Clear() after the fill causes a crash. *** Break *** segmentation violation
Generating stack trace...
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
/usr/bin/addr2line: python: No such file or directory
0x011f9ed9 in TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) + 0x41 from /u1/local/root/lib/root/libCore.so
0x007c4aa2 in TBranchElement::FillLeaves(TBuffer&) + 0x4b4 from /u1/local/root/lib/root/libTree.so
0x007b509d in TBranch::Fill() + 0x32d from /u1/local/root/lib/root/libTree.so
0x007c4526 in TBranchElement::Fill() + 0x1f0 from /u1/local/root/lib/root/libTree.so
0x007c44bb in TBranchElement::Fill() + 0x185 from /u1/local/root/lib/root/libTree.so
0x007c44bb in TBranchElement::Fill() + 0x185 from /u1/local/root/lib/root/libTree.so
0x007efdd5 in TTree::Fill() + 0xa7 from /u1/local/root/lib/root/libTree.so
0x00814e71 in <unknown> from /u1/local/root/lib/root/libTree.so
0x0178ad72 in G__CallFunc::Execute(void*) + 0x94 from /u1/local/root/lib/root/libCint.so
0x0064f046 in PyROOT::TIntExecutor::Execute(G__CallFunc*, void*) + 0x24 from /u1/local/root/lib/root/libPyROOT.so
0x00654b70 in PyROOT::TMethodHolder::CallFast(void*) + 0x2c from /u1/local/root/lib/root/libPyROOT.so
0x00654c5b in PyROOT::TMethodHolder::CallSafe(void*) + 0x4d from /u1/local/root/lib/root/libPyROOT.so
0x00654370 in PyROOT::TMethodHolder::Execute(void*) + 0xce from /u1/local/root/lib/root/libPyROOT.so
0x00654592 in PyROOT::TMethodHolder::operator()(PyROOT::ObjectProxy*, _object*, _object*) + 0x17a from /u1/local/root/lib/root/libPyROOT.so
0x00654e2f in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) + 0x5b from /u1/local/root/lib/root/libPyROOT.so
0x0805bd60 in PyObject_Call + 0x1c from python
0x080a927f in <unknown> from python
0x080a8d37 in <unknown> from python
0x080a73b3 in PyEval_EvalFrame + 0x223b from python
0x080a91b9 in <unknown> from python
0x080a8da7 in <unknown> from python
0x080a73b3 in PyEval_EvalFrame + 0x223b from python
0x080a7e6e in PyEval_EvalCodeEx + 0x40a from python
0x080aa8ca in PyEval_EvalCode + 0x22 from python
0x080d22a5 in <unknown> from python
0x080d1b09 in PyRun_SimpleFileExFlags + 0x191 from python
0x080554cb in Py_Main + 0x56b from python
0x08054f5b in main + 0x17 from python
0x0048c78a in __libc_start_main + 0xda from /lib/tls/libc.so.6
0x08054eb9 in ldexp + 0x61 from python
Traceback (most recent call last):
File "cutAnalysis.py", line 71, in ?
writeFilteredTuple(iSam, cuts)
File "/afs/slac.stanford.edu/u/br/jstrube/BaBar/StandaloneRoot/cutLibrary.py", line 117, in writeFilteredTuple
t.Fill()
SystemError: problem in C++; program state has been reset
I’m open to other ideas
Jan
ev = JansEvent()
inf = TFile(‘x.root’)
int = inf.Get(‘postCut’)
int.SetBranchAddress(‘event’, AddressOf(ev))
f = TFile(‘new.root’, ‘recreate’)
t = TTree(‘t’, ‘Test’)
t.Branch(‘event’, ‘JansEvent’, ev, 32000, 99)
int.GetEvent(0) #ev.bList.RemoveAt(1) #ev.bList.Clear() #ev.bList.Compress()
t.Fill()
int.GetEvent(1)
ev.bList[0].gamma
t.Write()
f.Close() [/code][/ul]
You may have to change the inf = TFile line. It’s the example from this thread, but the files from the other thread.
I hope this runs. If it doesn’t reproduce the problem for you, let me know, then I come up with a better example.
In summary:
[ol]
Removing items and writing to file crashes the program.
Calling Clear() or Compress() on the TClonesArray any time before TFile::Fill() causes the event to be written, but in the next event the TClonesArray entries aren’t cast down any more.
Just writing the correct candidate to position 0 in the Array and ignoring the rest (instead of RemoveAt’ing them) isn’t yet implemented in python.
Getting rid of the TClonesArray and using an STL vector doesn’t (seem to) work from the python side, either. I can create an STL vector in python, but I am not able to read it from a file.
[/ol]On this last part I only did a naive test, though. Do STL vectors need any special treatment when they are persisted ? Like extra streamer info somewhere ?
Cheers,
Jan
Somehow the information got lost somwhere but I just fixed a problem with the generation of the TTree in the cases of your classes. In order to test, you should try with the HEAD of the cvs repository. (The object were corrupted).
Also note that the file you mention in your last mail are insufficient to compiler JanEvent.cc (missing files).
So please try with the latest version of ROOT and if it still does not work, please upload one tar file containing all the necessary files to reproduce the crash.
Hmm. It works some way I don’t understand, but not how it’s supposed to.
Please run the attached bug.py. x.root must be in the same directory.
Please have a look at the file.
If you remove the first Clear(), you will see the failure to downcast
if you remove the second Clear() you will see the crash
if you leave in both clears, it runs without any messages, but it’s clearly wrong.
The reason why it took me a while to reproduce the problem is this:
Direct access to the contents of the TClonesArray works !
It’s when I loop over them that it shows problems.
So:
looping over them with python iterators doesn’t work, because it doesn’t cast down.
looping over them with direct access in the range (0…GetEntries()) doesn’t work, because after the Clear(), GetEntries() reports 0…
Removing the Clear() and then accessing by number again fails to cast down.
Hope that helps.
Jan pyroot.tar (30 KB)
<airdrie> ls -l
total 0
JansEvent.cc -> ../SkimR16/BetaMiniUser/JansEvent.cc
JansEvent.hh -> ../SkimR16/BetaMiniUser/JansEvent.rdl
Please upload a working copy.
Could please fully try your example before uploading it (by that I mean, go to a fresh new directory and do all the steps you expect me to have to do and see if they all do what you expect!)? There is yet another problem :
root [0] .L JansEvent.cc+
Info in <TWinNTSystem::ACLiC>: creating shared library c:\Download\jan\JansEvent_cc.dll
2072772110_cint.cxx
c:\download\jan\JansEvent.hh(37) : fatal error C1083: Cannot open include file: 'GClonesArray.hh': No such file or direc
tory
Error: external preprocessing failed. FILE: LINE:0
!!!Removing c:\Download\jan\s3bc_.cxx c:\Download\jan\s3bc_.h !!!
Error: rootcint: error loading headers...
Error in <ACLiC>: Dictionary generation failed!
Info in <ACLiC>: Invoking compiler to check macro's validity
JansEvent.cc
c:\download\jan\JansEvent.hh(37) : fatal error C1083: Cannot open include file: 'GClonesArray.hh': No such file or directory
I’m sorry. I mentioned earlier in this thread that this include is not necessary for the example. Please remove it. I should have wrapped it in an #ifdef block.
Apologies.
I will try again. However I noted:struct JansEvent {
#if !defined(__CINT__)
typedef GClonesArray<B_Parameters> B_List;
#else
typedef TClonesArray B_List;
#endif
B_List bList;
};Besides the fact that this is yet another thing I need to change to compile, is actually technically wrong and possibly very damaging. In essense this lies to CINT (and thus the ROOT I/O) on the nature of bList and
JansEvent. This has no chance of working unless GClonesArray<B_Parameters> are strictly the same in memory.
GClonesArray is a wrapper around TClonesArray that just provides STL-style accessors and iterators. The data structure should be the same in memory. (I haven’t explicitly checked, which I probably should, but never noticed any problems with it)
Thanks for noting this, though.
Well … fix my fixed up version of the library and the ROOT from cvs, I have absolutely no problem. I even translated (see attachement) your python script into C++ and ran with valgrind with no problem.
So either this problem comes from your GClonesArray either there is a more complex issue. If removing the GClonesArray does not solve the issue for you please provide me with[ul]
a single tested tar file containing of the files you need
the exact instruction to follow to execute YOUR output from following this instruction
include a stack trace in case of core dump.[/ul]
Note that to wrap a TClonesArray with a different interface we recommend to simply do (with any #ifdef):
class GClonesArray {
private: TClonesArray data; // or *data
public: void push_back(...);.... };In particular the ROOT I/O framework does not (completely) support class inherting from TClonesArray.
: