Variable getting mangled during retrieval

Hey experts,

I am posting this in Newbie because I am pretty sure it isn’t appropriate for ROOT Forum, but I really hope someone can chime in with past experience looking at a similar issue. This is a big problem for me, and really I am just hoping to get some next steps out of this post.

I am working with ROOT in art (FNAL). When I retrieve values from a TBranch storing an int[3], they come out mangled (all entries are {268374015,268374015,268374015}. (My retrieval is as follows)

Int_t pctag[3];
TBranch* tr_PCTag     ;
priv_LightFile = TFile::Open(light_file_name.c_str(), "READ");
if(! priv_LightFile->IsOpen()){return;}
tr = (TTree*)priv_LightFile->Get("midas_data");
priv_nlightentries = tr->GetEntries();
tr->SetBranchAddress("PCTimeTag",   pctag      , &tr_PCTag     );
tr->GetEntry(0);
std::cout<<std::fixed<<"priv_LightTimeFirst:       "<<pctag[0]<<", "<<pctag[1]<<", "<<pctag[2]<<"\n";
//output
priv_LightTimeFirst: 268374015, 268374015, 268374015

The result of a scan from inside this module is as follows:

415         tr->Scan("PCTimeTag","","",10);
>>> n   
***********************************
*    Row   * Instance * PCTimeTag *
***********************************
*        0 *        0 * 1.504e+09 *
*        0 *        1 * 1.504e+09 * 
*        0 *        2 *       778 *
*        1 *        0 * 1.504e+09 *
*        1 *        1 * 1.504e+09 *
*        1 *        2 *       962 *

As you can see, the result of scan is different than the result of what I get from standard retrieval with getEntry(i).

My question isn’t really a typical question (Not here is exactly my code, why doesn’t it work), because this is a very large piece of software. Rather, I am trying to find out where to look for debugging this symptom (since I am clearly not down to the broken code yet). Of course, it complicates things that this works fine for me without our framework (just quickly parsing the file in ROOT6 standalone).

Edit: Please disregard the below. I have managed to correct the issue with a segfault in ROOT by updating part of our interface. I do not know exactly which change fixed it (major version increment), but it is fixed.


I am retrieving a value from a ROOT file inside of the art framework (FNAL) and seeing VERY strange behaviour. If I load the file in root and scan the tree I am trying to load, the values in it make sense, but when I open the file in my compiled code, it fails in a really strange way.

Int_t type times I extract from the TTree in my file come out the same value. (268374015), when I am supposed to get an epoch time (150XXXXXXX).

tr->SetBranchAddress("TimeStamp",   &time_stamp, &tr_TimeStamp );
tr->GetEntry(i);

I tried adding a scan to my compiled art code, tr->Scan("MyBranch","","",10);, but the result is a segfault in TEnv.cxx:455. Somehow I end up inside the if statement without a defined gROOT (er and gSystem can both be inspected with gdb print statements, gdb tells me gROOT is undeclared) (er=0, gSystem=$1 = (TSystem *) 0x645540, No symbol "gROOT" in current context.)

I am really confused here, and not really sure what to do in debugging this. My actual root code in this art module is pretty simple, but it doesn’t respond the same way as when run in ROOT itself. Can anyone offer some insight into what kind of failure I might be looking at (So I can try to figure out what part of our code is interfering and fix it). I would start with out stuff, but the only errors I am getting are from ROOT code, and I am looking at a 400K+ line project. I need to narrow in a lot more.

And what does tr->Print(); say about the “PCTimeTag” branch?

BTW. For “gROOT”, you need to #include "TROOT.h" (and, in your own standalone executable, you might need to create a TApplication).

The TTree print is below.

417         tr->Print();                                                      
>>> n                                                                         
******************************************************************************
*Tree    :midas_data: MIDAS data                                             *
*Entries :   104848 : Total =    329921688194 bytes  File  Size = 84092884646 
*        :          : Tree compression factor =   3.92                       *
******************************************************************************
*Br    0 :TimeStamp : TimeStamp/I                                            *
*Entries :   104848 : Total  Size=     668495 bytes  File Size  =     401508 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.54     *
*............................................................................*
*Br    1 :TriggerTimeTag : TriggerTimeTag/I                                  *
*Entries :   104848 : Total  Size=     680710 bytes  File Size  =     270744 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.33     *
*............................................................................*
*Br    2 :PCTimeTag : PCTimeTag[3]/I                                         *
*Entries :   104848 : Total  Size=    1507285 bytes  File Size  =     794504 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.84     *
*............................................................................*
*Br    3 :event     : event/I                                                *
*Entries :   104848 : Total  Size=     658723 bytes  File Size  =     412035 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.48     *
*............................................................................*
*Br    4 :nchannels : nchannels/I                                            *
*Entries :   104848 : Total  Size=     668495 bytes  File Size  =     258534 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.40     *
*............................................................................*
*Br    5 :nsamples  : nsamples/I                                             *
*Entries :   104848 : Total  Size=     666052 bytes  File Size  =     256095 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.41     *
*............................................................................*
*Br    6 :TimeSample : TimeSample/I                                          *
*Entries :   104848 : Total  Size=     670938 bytes  File Size  =     260973 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.38     *
*............................................................................*
*Br    7 :channel_nsat : channel_nsat[8])/I                                  *
*Entries :   104848 : Total  Size=    3611576 bytes  File Size  =     322937 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=  11.03     *
*............................................................................*
*Br    8 :channel_pedestal : channel_pedestal[8])/F                          *
*Entries :   104848 : Total  Size=    3621348 bytes  File Size  =    2242516 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.59     *
*............................................................................*
*Br    9 :channel_charge : channel_charge[8])/F                              *
*Entries :   104848 : Total  Size=    3616462 bytes  File Size  =    2798006 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.27     *
*............................................................................*
*Br   10 :crt_daq_match : crt_daq_match/I                                    *
*Entries :   104848 : Total  Size=     678267 bytes  File Size  =     260973 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.41     *
*............................................................................*
*Br   11 :crt_match_id : crt_match_id/I                                      *
*Entries :   104848 : Total  Size=     675824 bytes  File Size  =     258534 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.42     *
*............................................................................*
*Br   12 :crt_adc   : crt_adc[4][32]/I                                       *
*Entries :   104848 : Total  Size=   54170107 bytes  File Size  =     753385 *
*Baskets :     4876 : Basket Size=      21914 bytes  Compression=  71.77     *
*............................................................................*
*Br   13 :crt_SE_time : crt_SE_time[2]/I                                     *
*Entries :   104848 : Total  Size=    1092779 bytes  File Size  =     260971 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   4.00     *
*............................................................................*
*Br   14 :crt_TS_time : crt_TS_time[2]/I                                     *
*Entries :   104848 : Total  Size=    1092779 bytes  File Size  =     260971 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   4.00     *
*............................................................................*
*Br   15 :crt_nHitAllTot : crt_nHitAllTot/I                                  *
*Entries :   104848 : Total  Size=     680710 bytes  File Size  =     263412 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.40     *
*............................................................................*
*Br   16 :crt_nHitTot : crt_nHitTot/I                                        *
*Entries :   104848 : Total  Size=     673381 bytes  File Size  =     256095 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.44     *
*............................................................................*
*Br   17 :crt_nHitSatTot : crt_nHitSatTot/I                                  *
*Entries :   104848 : Total  Size=     680710 bytes  File Size  =     263412 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.40     *
*............................................................................*
*Br   18 :crt_charge : crt_charge/F                                          *
*Entries :   104848 : Total  Size=     670938 bytes  File Size  =     253656 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.45     *
*............................................................................*
*Br   19 :crt_ToF   : crt_ToF/F                                              *
*Entries :   104848 : Total  Size=     663609 bytes  File Size  =     246339 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.49     *
*............................................................................*

Thanks for the notice about TROOT. I am guessing my update pulled the appropriate headers in somewhere in the framework then.

There are some strange “)” characters in “channel_nsat[8])/I”, “channel_pedestal[8])/F” and “channel_charge[8])/F”.

You do not show the full source code of your macro but … right after the tr->GetEntry(0);, try to add:

tr->ResetBranchAddresses(); // disconnect from local variables

and then try tr->Scan(...);

BTW. You mention the “epoch time”. This is usually a “time_t” type, which is usually a 64 bit (signed) integer, so a “Long64_t” in ROOT (try sizeof(time_t)).

Yes, I know epoch is usually a Long64_t, but this file stores the value in a Int_t (If the experiment were still running, I would probably try to patch the source to store it as a long).

As to the use of ResetBranchAddresses(), the Scan and Print are as seen below.

//Scan
***********************************
*    Row   * Instance * PCTimeTag *
***********************************
*        0 *        0 * 1.504e+09 *
*        0 *        1 * 1.504e+09 *
*        0 *        2 *       778 *
*        1 *        0 * 1.504e+09 *
*        1 *        1 * 1.504e+09 *
*        1 *        2 *       962 *
*        2 *        0 * 1.504e+09 *
*        2 *        1 * 1.504e+09 *
*        2 *        2 *       781 *
*        3 *        0 * 1.504e+09 *
*        3 *        1 * 1.504e+09 *
*        3 *        2 *       834 *
//Print
******************************************************************************
*Tree    :midas_data: MIDAS data                                             *
*Entries :   104848 : Total =    329921688194 bytes  File  Size = 84092884646 
*        :          : Tree compression factor =   3.92                       *
******************************************************************************
*Br    0 :TimeStamp : TimeStamp/I                                            *
*Entries :   104848 : Total  Size=     668495 bytes  File Size  =     401508 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.54     *
*............................................................................*
*Br    1 :TriggerTimeTag : TriggerTimeTag/I                                  *
*Entries :   104848 : Total  Size=     680710 bytes  File Size  =     270744 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.33     *
*............................................................................*
*Br    2 :PCTimeTag : PCTimeTag[3]/I                                         *
*Entries :   104848 : Total  Size=    1507285 bytes  File Size  =     794504 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.84     *
*............................................................................*
*Br    3 :event     : event/I                                                *
*Entries :   104848 : Total  Size=     658723 bytes  File Size  =     412035 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.48     *
*............................................................................*
*Br    4 :nchannels : nchannels/I                                            *
*Entries :   104848 : Total  Size=     668495 bytes  File Size  =     258534 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.40     *
*............................................................................*
*Br    5 :nsamples  : nsamples/I                                             *
*Entries :   104848 : Total  Size=     666052 bytes  File Size  =     256095 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.41     *
*............................................................................*
*Br    6 :TimeSample : TimeSample/I                                          *
*Entries :   104848 : Total  Size=     670938 bytes  File Size  =     260973 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   2.38     *
*............................................................................*
*Br    7 :channel_nsat : channel_nsat[8])/I                                  *
*Entries :   104848 : Total  Size=    3611576 bytes  File Size  =     322937 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=  11.03     *
*............................................................................*
*Br    8 :channel_pedestal : channel_pedestal[8])/F                          *
*Entries :   104848 : Total  Size=    3621348 bytes  File Size  =    2242516 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.59     *
*............................................................................*
*Br    9 :channel_charge : channel_charge[8])/F                              *
*Entries :   104848 : Total  Size=    3616462 bytes  File Size  =    2798006 *
*Baskets :     2439 : Basket Size=      21914 bytes  Compression=   1.27     *
*............................................................................*

The values retrieved in Int_t[3] pctag in entry 0 are still

268374015, 268374015, 268374015

Edit: I should add. I do reset the addresses right after I call tr->GetEntry(0)

I think you need to provide a small root file for inspection.

If you don’t have any small file, you could try:

rooteventselector -l 100 FullFile.root:midas_data First100Events.root

But this assumes that your ROOT if working correctly (I have some doubts about it from what you show).

Will do. I will cut the first 2 events out of my test file and post it. (This was made on a different system than I am reading it from.) Thank you for the help so far.


The event selector is having issues.

<MACHINE:DIRECTORY> rooteventselector -l 2 /PATH/
FILE.root:midas_data midas_test_2events.root                                    
"WARNING: Same name objects aren't supported: 'midas_data' of '/PATH/FILE.root' won't be processed                                            "

Well, I guess @pcanal would need to have a look at it (I suspect the erroneous “)” in these branches 7, 8 and 9 generate problems).

Thanks. Sorry I wasn’t able to get you better information.

@pcanal, do you have access to the dunegpvms at FNAL?

I did check the file in ROOT without our framework, and the branches do still have the tailing ) in the name, even outside of art.

I don’t know if it helps for not, but directly from ROOT I can access the information correctly with GetEntry.

root [1] new TBrowser()
(TBrowser *) 0x4eb0bc0
root [2] TBranch* br;
root [3] int val[3]
(int [3]) { 0, 0, 0 }
root [4] midas_data->SetBranchAddress("PCTimeTag", val, &br)
(int) 0
root [5] midas_data->GetEntry(0)
(int) 3146520
root [6] val
(int [3]) { 1504264671, 1504264671, 778 }

Well, can it be that you mix object files and / or libraries built with different ROOT versions when compiling / linking your standalone application?

It’s possible. My application is a LArSoft module, and LArSoft has gotten very large. I was hoping to find out what kind of failure I was looking at so that I can narrow down which part of LArSoft is failing. I just don’t know enough about ROOTs internals to know what could cause GetEntry and Scan to return different values, which makes it hard to figure out which part of our code is misbehaving. I will have to look at the linking and see if someone hard linked a specific version of some ROOT library (I hope not).

In any case. Thank you for the time you have already shared. I greatly appreciate it, and always learn quite a bit when I end up here.

If you change the ROOT version, you are expected to rebuild everything that depends on it. This includes your object files and libraries and you are also required to recreate all “dictionaries”, i.e. the source code that was created by calls to “rootcint” and / or “rootcling” (which you then need to compile, of course, too). And do remember that you should use the same compiler for your own source code as the one that was used to build your ROOT.

No apparent library linking errors in CMake.

I am digging more with GDB (kind of just pseudo documenting my work here at this point, and who knows, maybe someone else will have a similar problem later.

Setting a watchpoint on the memory address of pctag[0] stops while GetEntry is running (expected) during the assignment of p[0] (expected). The call to Bytes is from TBufferFile::ReadFastArray where frombuf has the correct value for pctag[0] (1.5*10^9).

444     inline void frombuf(char *&buf, Int_t *x)    { frombuf(buf, (UInt_t *) x); }
//Values: 
>>> p buf                                              
$9 = (char *&) @0xf33e378: 0xf3deade "Y\252SrY\252Sr" 
>>> p x                                  
$10 = (Int_t *) 0x7fffd7727aa4           
>>> p *buf     
$11 = 89 'Y'   
>>> p *x                   
$12 = 1504334706           

Backing out of that function, I still have the correct value (Now in TBufferFile::ReadFastArray) in ii[0]. ii[1] and ii[2] have not yet been assigned (as expected).

>>> p *(ii+2)           
$10 = 934               
>>> p *(ii+1)           
$11 = 1504334706        
>>> p *ii               
$12 = 1504334706        

So far so good, but the confusing this is that it stops again 2 more times during the same invocation of GetEntry.

Lets look at what happens in the stack every time this value is written. (Breaking whenever pctag[0] is written to, and going up the stack to look at what entry we are on).

Hardware watchpoint 2: *0x7fffd7727aa4                  
                                                        
Old value = 0                                           
New value = 1504334706                                  

ReadFastArray i=0
GetEntry entry=104847
GetEntry seqprocessing i=2
>>> p branch->GetName()            
$15 = 0xf01cac9 "PCTimeTag"        

Right. Nothing is there to start, so we go from 0 to the value we want. Next

Hardware watchpoint 2: *0x7fffd7727aa4    
                                          
Old value = 1504334706                    
New value = 1504317404                    

ReadFastArray i=10
GetEntry entry=104847
GetEntry seqprocessing i=35
>>> p branch->GetName()                  
$20 = 0xf588bb9 "adc_value_5"            

adc_value_5 should not be going to this address. Next

Hardware watchpoint 2: *0x7fffd7727aa4
                                                    
Old value = 1504317404                              
New value = 265949148                               

ReadFastArray i=11
GetEntry entry=104847
GetEntry i=35 (branch35?)
>>> p branch->GetName()                      
$12 = 0xf588bb9 "adc_value_5"

So for some reason, I am overwriting this value with information from adc_value_5.

I have my answer (and it is an unbelievably foolish one). The array that I fill for the adc values is declared as Short_t adc[5][30000] instead of Short_t adc[6][300000].

>>> p &adc[5][10]                               
$12 = (short *) 0x7fffd7727aa4                  
>>> p &pctag[0]                                 
$13 = (Int_t *) 0x7fffd7727aa4                  

Other things may still be strange, and the ROOT installation integrated into our framework still seems to be a bit tricky, but this specific issue is solved.