Hello!
We have recently upgraded our ROOT version from v6-26-04
to v6-26-10
. Since then, we are observing intermittent problems with deleting certain objects. Before we were getting the following warning, which was difficult to understand (not sure which TList was the culprit):
Error in <TList::Clear>: A list is accessing an object (0x3bbab60) already deleted (list name = TList)
Now we are having segfaults while deleting these, here is the related stack trace:
/opt/o2/lib/libCore.so.6.26: TCollection::GarbageCollect(TObject*) task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
/opt/o2/lib/libCore.so.6.26: TList::Clear(char const*) task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
/opt/o2/lib/libGpad.so.6.26: TPad::Close(char const*) task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
/opt/o2/lib/libGpad.so.6.26: TPad::~TPad() task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
/opt/o2/lib/libGpad.so.6.26: TCanvas::~TCanvas() task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
Objects are serialized and deserialized before this deletion, it could play a role here. We suspect that this commit could be related to the segfaults: core: Disable check for already deleted object. by pcanal · Pull Request #11334 · root-project/root · GitHub
Here is where these objects are created: QualityControl/TrendingTask.cxx at master · AliceO2Group/QualityControl · GitHub
The issue can be also reproduced by opening the attached file correlation_mean_stddev.root with a TBrowser
, opening and then closing the canvas. Upon closing it, the root
instance experiences a crash in a similar code area:
#6 0x00007fe735135cb0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x00007fe71f91ee28 in TPad::Clear (this=this
entry=0x3f9ea00, option=option
entry=0x7fe71ec44ec4 "") at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/graf2d/gpad/src/TPad.cxx:644
#8 0x00007fe71f8f89d3 in TCanvas::Clear (this=0x3f9ea00, option=0x7fe71ec44ec4 "") at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/graf2d/gpad/src/TCanvas.cxx:746
#9 0x00007fe71ebe77f8 in TRootCanvas::ReallyDelete (this=0x58c3560) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/gui/gui/src/TRootCanvas.cxx:704
#10 0x00007fe73103dba1 in TClingCallFunc::exec (this=0x58ffd00, address=<optimised out>, ret=<optimised out>) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:127
#11 0x00007fe73569ebe8 in TQConnection::SendSignal (this=0x5f81af0) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TQConnection.h:76
#12 0x00007fe735a099d9 in TQObject::EmitVA<>(char const*, int) (this=0x2b187a0, signal_name=<optimised out>) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TQObject.h:137
#13 0x00007fe7356d05ad in TQObject::Emit (signal=0x7fe7358b9483 "Timeout()", this=0x2b187a0) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TQObject.h:173
#14 TTimer::Timeout (this=0x2b18790) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TTimer.h:97
#15 TTimer::Notify (this=0x2b18790) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TTimer.cxx:145
#16 0x00007fe7356cffe1 in TTimer::CheckTimer (this=this
entry=0x2b18790, now=...) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TTimer.cxx:132
#17 0x00007fe735791aa4 in TUnixSystem::DispatchTimers (this=0x203f800, mode=true) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/unix/src/TUnixSystem.cxx:2940
#18 0x00007fe7357929ed in TUnixSystem::DispatchOneEvent (this=0x203f800, pendingOnly=<optimised out>) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/unix/src/TUnixSystem.cxx:1080
#19 0x00007fe7356bf8b9 in TSystem::InnerLoop (this=0x203f800) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TSystem.cxx:406
#20 TSystem::Run (this=0x203f800) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TSystem.cxx:356
#21 0x00007fe735658b33 in TApplication::Run (this=this
entry=0x2099320, retrn=retrn
entry=false) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TApplication.cxx:1609
#22 0x00007fe735a08929 in TRint::Run (this=this
entry=0x2099320, retrn=retrn
entry=false) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/rint/src/TRint.cxx:501
#23 0x00000000004011ad in main (argc=<optimised out>, argv=0x7fff02388b58) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/main/src/rmain.cxx:84
If we use ROOT v6-26-04, the same objects can be correctly opened and closed.
Would you have any hints what could be our problem here? It is likely that we were creating the concerned object in a wrong way, which could have worked by chance before, but it we are struggling to understand where the issue is exactly. Many thanks for your continuous support!
_ROOT Version: v6-26-10
_Platform: CentOS Stream 8
_Compiler: GCC v7.3.0