Retroactively handling multiple Class versions in ROOT files

pcanal · April 17, 2019, 8:58pm

, but of course ‘MyClassB’ is in fact ‘MyClassB_v1’ (simply by a different name)

Is it always? I thought the original issue is that you had different class layout with the same version number … (if the wasn’t any class layout change then we might be able to employ different tricks).

ROOT will complain about creating a TClonesArray of objects of a class with no in-memory definition…

It probably won’t … but the ‘type’ of the instances with MyClassB (in particular the virtual table will not be the same as MyClassB_vers0 it will ‘just’ be the one from the closest compiled base).

TObjArray only allows access to it’s owning parent file to determine the contained class

It won’t able you determine the content just the version numbers. A TObjArray can contains an heterogeneous set of object and thus it knows the actual type only after reading it (and/or it could pick at the byte stream).

A TClonesArray on the other hand can only contains one kind of objects and records that information in one of its data member and thus you can determine at run-time before reading the objects which kind of object needs to be read.

Cheers,
Philippe.

marc1uk · April 17, 2019, 9:45pm

Is it always? I thought the original issue is that you had different class layout with the same version number

Sorry, that was merely an example; MyClassB could be a MyClassB_v1, or MyClassB_v2 etc, i merely meant that when MyClassA_v1 is loaded and the TObjArray is populated, there is an appropriate definition of the contained class in memory (differing by name. e.g. MyClassB_v1 instead of MyClassB … and vtable, which i hadn’t thought of), and that the parent MyClassA::GetTObjArrayElement method involves a pointer cast from TObject to the specific MyClassB_vx anyway, so i wondered if it could ‘just work’.

Thanks for the clarification on the TObjArray/TClonesArray, but i’m not sure it answers my confusion on why multiple TObjArray derivations are needed, implementation wise. In both cases I’m overriding the Streamer methods to manually determine the contained type, so whether the normal automatic type detection is active before or after reading is irrelevant; in neither case is it being used.

pcanal · April 18, 2019, 7:58pm

Only if the in-memory layout are indeed the same and that the virtual table are essentially the same.

For the TObjArray, you can’t know at run-time which set of classes to use (whether it is MyClassB_ver_* or MyClassC_ver_*).

Cheers,
Philippe.

PS. If the onfile-layout of all the versions are the same, I am still confused why those complications are needed.

marc1uk · April 23, 2019, 5:18pm

Hi Philippe,
I gave this a go; I have defined a TObjArray_wrapper class which inherits from TObjArray and overrides the Streamer method with one that determines the class type and passes a suitable TClassRef as you described. I changed the TObjArray member of MyClassA_v0 to be a TObjArray_wrapper member. Finally I also added

#pragma link C++ class TObjArray_wrapper-;
#pragma read sourceClass="TObjArray" targetClass="TObjArray_wrapper";

to my LinkDef file.
However, i get the following errors when compiling the root dictionary:

./include/TObjArray_wrapper.hh:21:10: error: class member cannot be redeclared
    void Streamer(TBuffer &b);
         ^
./include/TObjArray_wrapper.hh:20:5: note: previous declaration is here
    ClassDef(TObjArray_wrapper,1)
    ^
.../root-6.06.08/build/include/Rtypes.h:255:4: note: expanded from macro 'ClassDef'
   _ClassDef_(name,id,virtual,)   \
   ^
.../root-6.06.08/build/include/Rtypes.h:248:25: note: expanded from macro '_ClassDef_'
   virtual_keyword void Streamer(TBuffer&) overrd; \

If I omit the ClassDef line in TObjArray_wrapper, I instead get

Error in <TObjArray_wrapper>: TObjArray_wrapper inherits from TObject but does not have its own ClassDef

pcanal · April 23, 2019, 6:43pm

The ClassDef is necessary. The Streamer is declared in the ClassDef and thus you commit your own from the class declaration (i.e delete line number 21).

Cheers,
Philippe.

marc1uk · April 23, 2019, 7:59pm

Ah right, I see, thankyou.
That compiles without any errors, but unfortunately I still get the same errors are before;

Error in <TBufferFile::ReadObject>: trying to read an emulated class (MyClassB) to store in a compiled pointer (TObject)

I’ve put some print statements in the TObjArray_wrapper::Streamer method, but they don’t seem to be printed, so it looks as if my custom streamer isn’t being used.

pcanal · April 23, 2019, 11:27pm

Did you use

#pragma link C++ class  TObjArray_wrapper-; // The - request a custom streamer.

?

marc1uk · April 23, 2019, 11:29pm

Yep, i did
[filling minimum character limit]

pcanal · April 24, 2019, 2:08pm

Fair enough … I have a couple of guess why this is not working. Could you provide a running/failing example so that I can try out a few possible work-around?

Thanks,
Philippe.

marc1uk · April 29, 2019, 2:22pm

OK, I’ve attached the set of files i’m using to define the classes (the various versions together with a wrapper) and a makefile to build the dictionary. I’ve also included a small file with 5 events generated with what ought to be the latest version of all the classes; in particular it should contain a WCSimRootEvent_v2 object in the wcsimrootevent branch. That is to say, the members of WCSimRootEvent_v2 should match the members in the WCSimRootEvent class in the file.
So if you

make -f GNUmakefile.FNAL

to build the dictionary, then

gSystem->Load("libWCSimRoot.so")
WCSimRootEvent_v2* ev = new WCSimRootEvent_v2()
wcsimT->SetBranchAddress("wcsimrootevent", &ev)
wcsimT->GetEntry(0);

Ideally it would retrieve successfully, but in fact it returns the previously described errors.

example.tar.gz (59.5 KB)
wcsim_0.root (185.5 KB)

marc1uk · May 10, 2019, 7:46pm

Keep alive bump.
[filling character limit]

marc1uk · May 22, 2019, 1:50pm

Any chance to take a look at the example files, @pcanal? This clearly isn’t a terribly urgent task, but i’d like to see it through to completion.

pcanal · May 22, 2019, 4:19pm

I have started looking into it and have not yet found a good solution. I will let you know as I find out more.

marc1uk · May 22, 2019, 4:39pm

Thanks Philippe..

pcanal · May 30, 2019, 9:17pm

I am making some progress. I have finally found a non-trivial way to have TObjArray_wrapper::Streamer actually called. However it currently does not work properly, likely due to the hierarchy of WCSimRootTrigger*.

I’ll let you know when I get that hurdle passed.

marc1uk · May 30, 2019, 9:24pm

Alright, that’s good to hear. Thanks for the continuing support.

pcanal · June 4, 2019, 9:44pm

Hi,

I finally found a combination that works. It adds code to both instrument the StreamerInfo with a ‘data member specific streamer’ and a work-around the fact it is not propagated properly in some circumstances.

This runs without any problem for me (just some informational messages):


Processing script.C...
Warning in <TClass::Init>: no dictionary for class WCSimRootEvent is available
Warning in <TClass::Init>: no dictionary for class WCSimRootTrigger is available
Warning in <TClass::Init>: no dictionary for class WCSimRootGeom is available
Warning in <TClass::Init>: no dictionary for class WCSimRootPMT is available
Warning in <TClass::Init>: no dictionary for class WCSimRootOptions is available
Warning in <TClass::Init>: no dictionary for class WCSimRootEventHeader is available
Warning in <TClass::Init>: no dictionary for class WCSimRootPi0 is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCapture is available
Warning in <TClass::Init>: no dictionary for class WCSimRootTrack is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCherenkovHit is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCherenkovHitTime is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCherenkovDigiHit is available
Info in <TMacOSXSystem::ACLiC>: creating shared library /Users/pcanal/root_working/test/2019-misc/example/./streamer_cc.so
Info in <TBufferFile::ReadObjectAny>: Using Converter StreamerInfo from TObjArray to TObjArray_wrapper
Info in <TBufferFile::ReadObjectAny>: Using Converter StreamerInfo from TObjArray to TObjArray_wrapper
Info in <TBufferFile::ReadObjectAny>: Using Converter StreamerInfo from TObjArray to TObjArray_wrapper

Could you verify that the data is actually read properly?

Thanks,
Philippe.

marc1uk · June 5, 2019, 9:31am

Great news!
I can certainly give it a try and check the data read out is as expected. I’ll need a copy of the modifications, though?

pcanal · June 5, 2019, 11:30am

Sorry. The upload had failed. Let’s try again.example.update.001.tar.gz (238.5 KB)

marc1uk · June 10, 2019, 4:35pm

H’mm, unfortunately it seems like this isn’t quite working. I built the library with your modifications and appended a simple check to the end of the script file: calling Print() on the WCSimRootEvent. This should scan over it’s contained triggers, digits, photons, tracks etc, and print out a summary. It ought to be a good first check everything is being read OK.
Unfortunately the script segmentation faults shortly into the print call:

$ root script.C 
root [0] 
Processing script.C...
Warning in <TClass::Init>: no dictionary for class WCSimRootEvent is available
Warning in <TClass::Init>: no dictionary for class WCSimRootTrigger is available
Warning in <TClass::Init>: no dictionary for class WCSimRootGeom is available
Warning in <TClass::Init>: no dictionary for class WCSimRootPMT is available
Warning in <TClass::Init>: no dictionary for class WCSimRootOptions is available
Warning in <TClass::Init>: no dictionary for class WCSimRootEventHeader is available
Warning in <TClass::Init>: no dictionary for class WCSimRootPi0 is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCapture is available
Warning in <TClass::Init>: no dictionary for class WCSimRootTrack is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCherenkovHit is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCherenkovHitTime is available
Warning in <TClass::Init>: no dictionary for class WCSimRootCherenkovDigiHit is available
Info in <TBufferFile::ReadObjectAny>: Using Converter StreamerInfo from TObjArray to TObjArray_wrapper
Info in <TBufferFile::ReadObjectAny>: Using Converter StreamerInfo from TObjArray to TObjArray_wrapper
Info in <TBufferFile::ReadObjectAny>: Using Converter StreamerInfo from TObjArray to TObjArray_wrapper
This entry had 1 triggers
Trigger time : 0
Number Primary Vertices : 1
will loop over 1 primaries (fNvtxs=1), c.f. size(fVtxs)=2700
Primary vertex 0 was at (-1.58819e-23, -1.58819e-23, -1.58819e-23)
looping over 1935 digits
getting digihit 0
adding photon count
calling digihit->GetPhotonIds()

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f5abbc89ffa in __GI___waitpid (pid=11719, stat_loc=stat_loc
entry=0x7ffc786d2160, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x00007f5abbc110ab in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2  0x00007f5abccf9c54 in TUnixSystem::Exec (shellcmd=<optimized out>, this=<optimized out>) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/unix/src/TUnixSystem.cxx:2096
#3  TUnixSystem::StackTrace (this=0x224e4f0) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/unix/src/TUnixSystem.cxx:2324
#4  0x00007f5abccfbdcc in TUnixSystem::DispatchSignals (this=0x224e4f0, sig=kSigSegmentationViolation) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/unix/src/TUnixSystem.cxx:3562
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x00007f5ab1cec481 in WCSimRootTrigger_v2::Print (this=0x4fa12d0, verbosity=10, maxprimariestoprint=10, maxtrackstoprint=10, maxdigitstoprint=10, maxphotonsperdigittoprint=10, maxphotonstoprint=10) at src/WCSimRootTrigger_v2.cc:264
#8  0x00007f5ab1cf70d5 in WCSimRootEvent_v2::Print (this=0x49a8080, verbosity=10, maxtriggerstoprint=10, maxprimariestoprint=10, maxtrackstoprint=10, maxdigitstoprint=10, maxphotonsperdigittoprint=10, maxphotonstoprint=10) at src/WCSimRootEvent_v2.cc:26
#9  0x00007f5abd4794e4 in __cling_Un1Qu30(void*) ()
#10 0x00007f5ab8ef5628 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/build/lib/libCling.so
#11 0x00007f5ab8efb8fe in cling::Interpreter::EvaluateInternal(std::string const&, cling::CompilationOptions, cling::Value*, cling::Transaction**) () from /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/build/lib/libCling.so
#12 0x00007f5ab8efbae5 in cling::Interpreter::process(std::string const&, cling::Value*, cling::Transaction**) () from /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/build/lib/libCling.so
#13 0x00007f5ab8f9a50b in cling::MetaProcessor::process(char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/build/lib/libCling.so
#14 0x00007f5ab8f9abe5 in cling::MetaProcessor::readInputFromFile(llvm::StringRef, cling::Value*, bool) () from /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/build/lib/libCling.so
#15 0x00007f5ab8e5e2bc in TCling::ProcessLine (this=0x22af0a0, line=<optimized out>, error=0x7ffc786d63fc) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/meta/src/TCling.cxx:1980
#16 0x00007f5ab8e52087 in TCling::ProcessLineSynch (this=0x22af0a0, line=0x325cba0 ".x /home/marc/LinuxSystemFiles/ToolAnalysis/ToolAnalysis/ToolDAQ/WCSimLib/./script.C", error=0x7ffc786d63fc) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/meta/src/TCling.cxx:2812
#17 0x00007f5abcbee8dd in TApplication::ExecuteFile (file=<optimized out>, error=0x7ffc786d63fc, keep=<optimized out>) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/base/src/TApplication.cxx:1129
#18 0x00007f5abcbef89e in TApplication::ProcessLine (this=0x229beb0, line=<optimized out>, sync=<optimized out>, err=0x7ffc786d63fc) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/base/src/TApplication.cxx:978
#19 0x00007f5abd048695 in TRint::ProcessLineNr (this=this
entry=0x229beb0, filestem=filestem
entry=0x7f5abd057133 "ROOT_cli_", line=line
entry=0x7ffc786d6400 ".x script.C", error=error
entry=0x7ffc786d63fc) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/rint/src/TRint.cxx:745
#20 0x00007f5abd049aa7 in TRint::Run (this=0x229beb0, retrn=<optimized out>) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/core/rint/src/TRint.cxx:420
#21 0x0000000000400f80 in main (argc=1, argv=0x7ffc786d8588) at /home/marc/LinuxSystemFiles/ROOT/root-6.06.08/sauce/main/src/rmain.cxx:30
===========================================================


Root >

Running with the normal dictionaries the output I would expect is:

root [4] ev->Print()
This entry had 1 triggers
Trigger time : 0
Trigger type : PromptTrigger
Number Primary Vertices : 1
Primary vertex 0 was at (0, 168.1, 168.1)
Num Tracks: 3
Num Digits: 1935
Num Photons: 1935
    Track 0{ Flag: -1 | PDG: 11 | ParentPDG: 0 | sProc:  | eProc: NuIntx }
    Track 1{ Flag: -2 | PDG: 2212 | ParentPDG: 0 | sProc:  | eProc: NuIntx }
    Track 2{ Flag: 0 | PDG: 11 | ParentPDG: 0 | sProc:  | eProc: Cerenkov }
      digit 0 at time -0.417573ns has charge 1.49149 from 1 true photons
        digit 0, photon 0 has truetime 6.05443
      digit 1 at time 1.67687ns has charge 0.694754 from 1 true photons
        digit 1, photon 0 has truetime 5.41446
      digit 2 at time 2.0762ns has charge 1.27331 from 1 true photons
        digit 2, photon 0 has truetime 5.39229
      digit 3 at time 2.35291ns has charge 0.099146 from 1 true photons
        digit 3, photon 0 has truetime 5.22537
      digit 4 at time 3.14034ns has charge 0.767604 from 1 true photons
        digit 4, photon 0 has truetime 5.27153
      digit 5 at time 3.20434ns has charge 1.63413 from 1 true photons
        digit 5, photon 0 has truetime 5.44086
      digit 6 at time 3.57768ns has charge 1.11406 from 1 true photons
        digit 6, photon 0 has truetime 5.40951
      digit 7 at time 3.92938ns has charge 1.46568 from 1 true photons
        digit 7, photon 0 has truetime 5.27193
      digit 8 at time 4.32783ns has charge 1.11076 from 1 true photons
        digit 8, photon 0 has truetime 5.3451
      digit 9 at time 4.37274ns has charge 1.14471 from 1 true photons
        digit 9, photon 0 has truetime 5.18689

So it seems that even those values printed before the segfault aren’t correct either. Sorry!