Segmentation faults when deleting certain objects since v6-26-10

pkonopka · December 13, 2022, 9:41am

Hello!

We have recently upgraded our ROOT version from v6-26-04 to v6-26-10. Since then, we are observing intermittent problems with deleting certain objects. Before we were getting the following warning, which was difficult to understand (not sure which TList was the culprit):

Error in <TList::Clear>: A list is accessing an object (0x3bbab60) already deleted (list name = TList)

Now we are having segfaults while deleting these, here is the related stack trace:

 /opt/o2/lib/libCore.so.6.26: TCollection::GarbageCollect(TObject*) task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
 /opt/o2/lib/libCore.so.6.26: TList::Clear(char const*) task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
 /opt/o2/lib/libGpad.so.6.26: TPad::Close(char const*) task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
 /opt/o2/lib/libGpad.so.6.26: TPad::~TPad() task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"
 /opt/o2/lib/libGpad.so.6.26: TCanvas::~TCanvas() task="alio2-cr1-hv-gw01.cern.ch:/opt/git/ControlWorkflows/tasks/jit-3aff17ad70b2670179a7114f29a2837cfaf8d812-qc-check-sink-QC_CalibrationTr-mo_0@842f2f51c04da4626c5b6c7d97d8727659cc85fb#2cDTwyee58d"

Objects are serialized and deserialized before this deletion, it could play a role here. We suspect that this commit could be related to the segfaults: core: Disable check for already deleted object. by pcanal · Pull Request #11334 · root-project/root · GitHub

Here is where these objects are created: QualityControl/TrendingTask.cxx at master · AliceO2Group/QualityControl · GitHub

The issue can be also reproduced by opening the attached file correlation_mean_stddev.root with a TBrowser, opening and then closing the canvas. Upon closing it, the root instance experiences a crash in a similar code area:

#6  0x00007fe735135cb0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x00007fe71f91ee28 in TPad::Clear (this=this
entry=0x3f9ea00, option=option
entry=0x7fe71ec44ec4 "") at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/graf2d/gpad/src/TPad.cxx:644
#8  0x00007fe71f8f89d3 in TCanvas::Clear (this=0x3f9ea00, option=0x7fe71ec44ec4 "") at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/graf2d/gpad/src/TCanvas.cxx:746
#9  0x00007fe71ebe77f8 in TRootCanvas::ReallyDelete (this=0x58c3560) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/gui/gui/src/TRootCanvas.cxx:704
#10 0x00007fe73103dba1 in TClingCallFunc::exec (this=0x58ffd00, address=<optimised out>, ret=<optimised out>) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/interpreter/llvm/src/include/llvm/ADT/SmallVector.h:127
#11 0x00007fe73569ebe8 in TQConnection::SendSignal (this=0x5f81af0) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TQConnection.h:76
#12 0x00007fe735a099d9 in TQObject::EmitVA<>(char const*, int) (this=0x2b187a0, signal_name=<optimised out>) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TQObject.h:137
#13 0x00007fe7356d05ad in TQObject::Emit (signal=0x7fe7358b9483 "Timeout()", this=0x2b187a0) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TQObject.h:173
#14 TTimer::Timeout (this=0x2b18790) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/inc/TTimer.h:97
#15 TTimer::Notify (this=0x2b18790) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TTimer.cxx:145
#16 0x00007fe7356cffe1 in TTimer::CheckTimer (this=this
entry=0x2b18790, now=...) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TTimer.cxx:132
#17 0x00007fe735791aa4 in TUnixSystem::DispatchTimers (this=0x203f800, mode=true) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/unix/src/TUnixSystem.cxx:2940
#18 0x00007fe7357929ed in TUnixSystem::DispatchOneEvent (this=0x203f800, pendingOnly=<optimised out>) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/unix/src/TUnixSystem.cxx:1080
#19 0x00007fe7356bf8b9 in TSystem::InnerLoop (this=0x203f800) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TSystem.cxx:406
#20 TSystem::Run (this=0x203f800) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TSystem.cxx:356
#21 0x00007fe735658b33 in TApplication::Run (this=this
entry=0x2099320, retrn=retrn
entry=false) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/base/src/TApplication.cxx:1609
#22 0x00007fe735a08929 in TRint::Run (this=this
entry=0x2099320, retrn=retrn
entry=false) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/core/rint/src/TRint.cxx:501
#23 0x00000000004011ad in main (argc=<optimised out>, argv=0x7fff02388b58) at /mnt/mesos/sandbox/sandbox/jenkins/workspace/DailyBuilds/DailyO2-ubuntu2004/daily-tags.Lw31gYHUjm/SOURCES/ROOT/v6-26-10-alice5/v6-26-10-alice5/main/src/rmain.cxx:84

If we use ROOT v6-26-04, the same objects can be correctly opened and closed.

Would you have any hints what could be our problem here? It is likely that we were creating the concerned object in a wrong way, which could have worked by chance before, but it we are struggling to understand where the issue is exactly. Many thanks for your continuous support!

_ROOT Version: v6-26-10
_Platform: CentOS Stream 8
_Compiler: GCC v7.3.0

moneta · December 13, 2022, 2:43pm

Hi,
@pcanal, author of the commit, can maybe comment on this

Lorenzo

pcanal · December 13, 2022, 6:46pm

This problem is fixed by Avoid TPad::fFrame double delete when read from file by pcanal · Pull Request #11763 · root-project/root · GitHub which will be included in the next set of patch releases (for example v6-26/12).

A work around for the issue:

auto ccdb_object = file->Get<TCanvas>("ccdb_object");
ccdb_object->GetFrame()->ResetBit(kCanDelete);

pkonopka · December 14, 2022, 9:29am

Thank you! We tested this, it indeed helps with closing these objects.

We also discovered that we incorrectly use Draw() on objects which are already drawn (to update them), which actually duplicates some pointers in a certain list. So instead of:

        title->Draw();

apparently we should have done:

        c->Modified();
        c->Update();

Where title is inside the canvas c.

Here is the PR for the reference.

system · December 28, 2022, 9:30am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.