Root 6.24/08 :: breakage accessing/hadd files produced with v6-30-01

ROOT
1

Hi! while trying to merge some files produced with v6-30-01 with epel root 6.24/08
i get these errors

Error in <THashList::Delete>: A list is accessing an object (0x1d61840) already deleted (list name = THashList)
Error in <TList::Clear>: A list is accessing an object (0x1d68a80) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d68a80) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d68a80) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d692e0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d692e0) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d692e0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6cae0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6cae0) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6cae0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6d3a0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6d3a0) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6d3a0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6dc60) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6dc60) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6dc60) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6e520) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6e520) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6e520) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d69850) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d69850) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d69850) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6a0b0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6a0b0) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6a0b0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6a910) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6a910) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6a910) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6b170) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6b170) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6b170) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6ba30) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d6ba30) already deleted (list name = TList)
Error in <TList::Delete>: A list is accessing an object (0x1d6ba30) already deleted (list name = TList)
Error in <THashList::Delete>: A list is accessing an object (0x1d64810) already deleted (list name = THashList)
Error in <TList::Clear>: A list is accessing an object (0x1d3b230) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d47660) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1d4d5e0) already deleted (list name = TList)

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
    gdb.printing.register_pretty_printer(gdb.current_objfile(),
#0  0x00007f7869ac560c in __libc_waitpid (pid=2019856, stat_loc=stat_loc
entry=0x7fff9ed5dd60, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:31
#1  0x00007f7869a42f62 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2  0x00007f786b0c03fc in TUnixSystem::StackTrace (this=0xb35960) at /usr/src/debug/root-6.24.08/core/unix/src/TUnixSystem.cxx:2411
#3  0x00007f786b0c231a in TUnixSystem::DispatchSignals (this=0xb35960, sig=kSigSegmentationViolation) at /usr/src/debug/root-6.24.08/core/unix/src/TUnixSystem.cxx:3649
#4  <signal handler called>
#5  0x00007f786b024e7a in GetHashValue (this=0x1c2ce10, obj=0x1d4cb60) at /usr/src/debug/root-6.24.08/core/cont/inc/THashTable.h:101
#6  THashTable::Remove (this=0x1c2ce10, obj=0x1d4cb60) at /usr/src/debug/root-6.24.08/core/cont/src/THashTable.cxx:419
#7  0x00007f786b0246a5 in THashList::Delete (this=0x1c2cd90, option=<optimized out>) at /usr/src/debug/root-6.24.08/core/cont/src/THashList.cxx:224
#8  0x00007f786b71d266 in TDirectoryFile::~TDirectoryFile (this=0x1c2cc70, __in_chrg=<optimized out>) at /usr/src/debug/root-6.24.08/io/io/src/TDirectoryFile.cxx:195
#9  0x00007f786b71d2e9 in TDirectoryFile::~TDirectoryFile (this=0x1c2cc70, __in_chrg=<optimized out>) at /usr/src/debug/root-6.24.08/io/io/src/TDirectoryFile.cxx:202
#10 0x00007f786b725201 in TFileMerger::MergeOne (this=0x7fff9ed61300, target=0x1c20490, sourcelist=0x7fff9ed61358, type=12, info=..., oldkeyname=..., allNames=..., status=
0x7fff9ed60a8e: true, onlyListed=
0x7fff9ed60a8f: false, path=..., current_sourcedir=0x1c20890, current_file=0x1c20890, key=0x1c21560, obj=<optimized out>, nextkey=...) at /usr/src/debug/root-6.24.08/io/io/src/TFileMerger.cxx:780
#11 0x00007f786b727c4d in TFileMerger::MergeRecursive (this=0x7fff9ed61300, target=0x1c20490, sourcelist=0x7fff9ed61358, type=12) at /usr/src/debug/root-6.24.08/io/io/src/TFileMerger.cxx:881
#12 0x00007f786b7280b9 in TFileMerger::PartialMerge (this=0x7fff9ed61300, in_type=12) at /usr/src/debug/root-6.24.08/io/io/src/TFileMerger.cxx:968
#13 0x0000000000405efa in operator() (merger=..., __closure=0x7fff9ed60f40) at /usr/src/debug/root-6.24.08/main/src/hadd.cxx:473
#14 operator() (nFiles=1, start=<optimized out>, merger=..., __closure=0x7fff9ed60ef0) at /usr/src/debug/root-6.24.08/main/src/hadd.cxx:501
#15 main (argc=8, argv=0x7fff9ed61638) at /usr/src/debug/root-6.24.08/main/src/hadd.cxx:543
===========================================================


The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  0x00007f786b024e7a in GetHashValue (this=0x1c2ce10, obj=0x1d4cb60) at /usr/src/debug/root-6.24.08/core/cont/inc/THashTable.h:101
#6  THashTable::Remove (this=0x1c2ce10, obj=0x1d4cb60) at /usr/src/debug/root-6.24.08/core/cont/src/THashTable.cxx:419
#7  0x00007f786b0246a5 in THashList::Delete (this=0x1c2cd90, option=<optimized out>) at /usr/src/debug/root-6.24.08/core/cont/src/THashList.cxx:224
#8  0x00007f786b71d266 in TDirectoryFile::~TDirectoryFile (this=0x1c2cc70, __in_chrg=<optimized out>) at /usr/src/debug/root-6.24.08/io/io/src/TDirectoryFile.cxx:195
#9  0x00007f786b71d2e9 in TDirectoryFile::~TDirectoryFile (this=0x1c2cc70, __in_chrg=<optimized out>) at /usr/src/debug/root-6.24.08/io/io/src/TDirectoryFile.cxx:202
#10 0x00007f786b725201 in TFileMerger::MergeOne (this=0x7fff9ed61300, target=0x1c20490, sourcelist=0x7fff9ed61358, type=12, info=..., oldkeyname=..., allNames=..., status=
0x7fff9ed60a8e: true, onlyListed=
0x7fff9ed60a8f: false, path=..., current_sourcedir=0x1c20890, current_file=0x1c20890, key=0x1c21560, obj=<optimized out>, nextkey=...) at /usr/src/debug/root-6.24.08/io/io/src/TFileMerger.cxx:780
#11 0x00007f786b727c4d in TFileMerger::MergeRecursive (this=0x7fff9ed61300, target=0x1c20490, sourcelist=0x7fff9ed61358, type=12) at /usr/src/debug/root-6.24.08/io/io/src/TFileMerger.cxx:881
#12 0x00007f786b7280b9 in TFileMerger::PartialMerge (this=0x7fff9ed61300, in_type=12) at /usr/src/debug/root-6.24.08/io/io/src/TFileMerger.cxx:968
#13 0x0000000000405efa in operator() (merger=..., __closure=0x7fff9ed60f40) at /usr/src/debug/root-6.24.08/main/src/hadd.cxx:473
#14 operator() (nFiles=1, start=<optimized out>, merger=..., __closure=0x7fff9ed60ef0) at /usr/src/debug/root-6.24.08/main/src/hadd.cxx:501
#15 main (argc=8, argv=0x7fff9ed61638) at /usr/src/debug/root-6.24.08/main/src/hadd.cxx:543
===========================================================


Error in <TList::Clear>: A list is accessing an object (0x19ff6e0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x19ffb90) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1a2b980) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1a2be50) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1a2c220) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1a2c6c0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x1a2d5b0) already deleted (list name = TList)

trying a simple rootls i get:

rootls -lt /alice/flow3_apass1_weights/flowCenW_lhc23zzg_544032_v2_phiMultTightRofCuts_isPVcont_wghtPt_20240118_0849/job_000070/AnalysisResults.root
Error in <TList::Clear>: A list is accessing an object (0x53f6480) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5442ec0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x544af80) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x544b2f0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x544b660) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x544b9d0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x544bf00) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x544c3b0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5472ef0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x54733c0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5473790) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5473c30) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5474b20) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x54757d0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5475c00) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5475e60) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x54768a0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5476d50) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5477490) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5477910) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5477d90) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5478910) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5478b70) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5478fc0) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5479440) already deleted (list name = TList)
Error in <TList::Clear>: A list is accessing an object (0x5479eb0) already deleted (list name = TList)
TDirectoryFile  Jan 19 08:16 2024 bc-selection-task     "bc-selection-task"
TDirectoryFile  Jan 19 08:16 2024 event-selection-task  "event-selection-task"
TDirectoryFile  Jan 19 08:16 2024 flow_base             "flow_base"

with the same root (that produced the files) i get:

[O2Physics/daily-20240115-0100-1] flowCenW_lhc23zzg_544032_v2_phiMultTightRofCuts_isPVcont_wghtPt_20240118_0849 > rootls -lt /alice/flow3_apass1_weights/flowCenW_lhc23zzg_544032_v2_phiMultTightRofCuts_isPVcont_wghtPt_20240118_0849/job_000101/AnalysisResults.root
TDirectoryFile  Jan 19 08:39 2024 bc-selection-task;1    "bc-selection-task" 
TDirectoryFile  Jan 19 08:39 2024 event-selection-task;1 "event-selection-task" 
TDirectoryFile  Jan 19 08:39 2024 flow_base;1            "flow_base"

beside the error i noticed that the old root does not see the cycle of the object whereas
the correct root does…

is this expected? i was almost certain that for plain root objects i could use and manipulate root files in a forward compatible way
Thanks a lot!

2

Dear @adrian_sev ,

Indeed forward-compatibility should be guaranteed or otherwise properly handled. I believe @pcanal will be able to point us towards a possible cause for your issue.

Cheers,
Vincenzo

3

Dear @adrian_sev ,

I saw that this error was already encountered in this other post and it’s due to a missing back port fix. In the 6.24 release cycle, you should use 6.24/10 to get your application working again. The full list of releases that support this case is updated at Fix forward compatibility with v6-30 by pcanal · Pull Request #12845 · root-project/root · GitHub

Best,
Vincenzo

1 Like
4

Oh, cool, nice that it’s fixed! I will wait then for the update of the EPEL package :slight_smile:
Thanks a lot!

5

You may want to ping them to update sooner rather than later as this is a kinda ‘important’ update.

6

@pcanal ping who? the first step would be to have a ROOT release with the mentioned PR… backported to 6.24 as i see that this PR is in v6-28-00-patches only (i apologize if i’m wrong). Moreover EPEL7 (for EL7) most probably is stuck to 6.24 as the EL7 gcc does not have cxx17, so the last ROOT version to be present in EPEL7 is 6.24…
So, is it possible to have this backported and a 6.24/10 tagged?
Moreover, just looking at commit list maybe the following would make sense to be backported?
e48ba0494b fBits read: preserve kIsOnHeap, always set kNotDeleted.
and any PyROOT and other bugfixes that could be relevant?
Thanks a lot!!

7

ping @pcanal :slight_smile: Could you please express a view/opinion/ack on the subject of back-porting the PR to 6-24-00-patches and about tagging 6.24/10 ?
I apologize for insistence but i would love a clear status for the future :slight_smile: (if these are not possible then i should remove any usage of system/epel ROOT usage and use a built one)

L.E. actually i now see the fBits read: preserve kIsOnHeap, always set kNotDeleted (i apologize if i did not see this before) on top of v6-24-00-patches, so my only question remains about tagging of 6.24/10 :slight_smile:

Thanks a lot!!

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.