Crash when creating RDataFrame graph

Continuing the discussion from RDataFrame MT performance running on remote files:

I create several Defines and Aggregates in loop(s), always in this format

latestDF = std::make_unique<RNode>(latestDF->Define(...))
histo = latestDF->Aggregate(...))

It seems to work correctly as the histograms are filled and saved after the analysis. But the ROOT::RDF::SaveGraph on the main RDataFrame fails:

I’m not sure if this could narrow what I might do wrong in my graph - the graph is created in the last line of code, the RDataFrame analysis seems to finish ok, histograms are saved etc…

===========================================================
#6  0x000000000073d0d1 in std::__shared_ptr<ROOT::Internal::RDF::GraphDrawing::GraphNode, (__gnu_cxx::_Lock_policy)2>::operator= (this=0x80) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/shared_ptr_base.h:1078
#7  0x000000000073d119 in std::shared_ptr<ROOT::Internal::RDF::GraphDrawing::GraphNode>::operator= (this=0x80) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/shared_ptr.h:103
#8  0x000000000073d14a in ROOT::Internal::RDF::GraphDrawing::GraphNode::SetPrevNode (this=0x0, node=...) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDF/GraphNode.hxx:75
#9  0x0000000000735d8a in ROOT::Internal::RDF::RActionCRTP<ROOT::Internal::RDF::RAction<ROOT::Internal::RDF::AggregateHelper<main(int, char**)::<lambda(TH2D&, const ZHCuts::Helper2DObservable&)>, main(int, char**)::<lambda(TH2D, TH2D)>, void, ZHCuts::Helper2DObservable, TH2D, false>, ROOT::Detail::RDF::RNodeBase, ROOT::TypeTraits::TypeList<ZHCuts::Helper2DObservable> > >::GetGraph(void) (this=0x97da9f0) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDF/RAction.hxx:187
#10 0x00007fdc71502ee5 in ROOT::Internal::RDF::GraphDrawing::GraphCreatorHelper::RepresentGraph[abi:cxx11](ROOT::Detail::RDF::RLoopManager*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/lib/libROOTDataFrame.so
#11 0x00007fdc715032f4 in ROOT::Internal::RDF::GraphDrawing::GraphCreatorHelper::RepresentGraph[abi:cxx11](ROOT::RDataFrame&) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/lib/libROOTDataFrame.so
#12 0x000000000074df37 in ROOT::Internal::RDF::GraphDrawing::GraphCreatorHelper::operator()<ROOT::RDataFrame> (this=0x7ffd6f98594f, node=...) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDF/GraphUtils.hxx:162
#13 0x00000000007447cf in ROOT::RDF::SaveGraph<ROOT::RDataFrame> (node=..., outputFile=...) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDFHelpers.hxx:128
#14 0x000000000070f224 in main (argc=2, argv=0x7ffd6f9884c8) at src/first_test.cxx:671
===========================================================


Thanks,
Zdenek

ROOT Version: 6.22/00
Platform: Centos7
Compiler: g++ 8.3.0


Hi,
this looks like a problem in SaveGraph (sorry about that!), but I could not reproduce the crash with simple combinations of Defines and Aggregates. Could you please share a reproducer that we can use to debug?

Cheers,
Enrico

Hello,
I still can’t figure out how to reproduce it with a simple code, but when writing an example, I notice some strange features from the graph -
test3.C (3.2 KB) testtree2.root (6.8 KB)
it runs on a simple tree with float and vector<float> branches
it produces
new
which is what I wanted to achieve, however when

  //latestDF = std::make_unique<RNode>(latestDF->Define("r42",[](){return 42;}));

is uncommented, the graph changes to


However, the output root stays the same/the v_float part gets filled.
I also noticed when I define multiple columns

latestDF = std::make_unique<RNode>(latestDF->Define("A",...);
latestDF = std::make_unique<RNode>(latestDF->Define("B",...);
latestDF = std::make_unique<RNode>(latestDF->Define("C",...);

this gets plotted as T->C->B->A instead of T->A->B->C?


test4.C (751 Bytes)
Cheers,
Zdenek

Uhm, probably related to https://sft.its.cern.ch/jira/browse/ROOT-9722 .

I’ll take a look asap.

Found the problem, it’s a bug in SaveGraph, but not ROOT-9722. Work in progress :smile:

Thanks for reporting!

Ok so, the issue is now reported as https://github.com/root-project/root/issues/6853 and I should have a fix at https://github.com/root-project/root/pull/6861 .

Do you by chance have a setup with which you can test nightly ROOT builds?

Cheers,
Enrico

Yes, I think I could set up the nightlies -
I setup

export LCGENV_PATH=/cvmfs/sft.cern.ch/lcg/releases
export PATH=/cvmfs/sft.cern.ch/lcg/releases/lcgenv/latest:${PATH}
eval "`lcgenv x86_64-centos7-gcc8-opt all`"

for gcc and then

source /cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/thisroot.sh
which root
/cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/root
root-config --version
6.23/01

At the moment, it gives the same graph as before, should I wait?
Cheers,
Zdenek

Yes :smile: the pull request is still not merged – after it’s merged the next nightlies will contain the change. I’ll ping you here!

Hi,
I just merged the fix in master, tomorrow’s nightlies should contain the fix.

Cheers,
Enrico

Hi,
Is there an easy way for me to see to which nightly it was propagated?
When I setup the nigtly in the way above, I seem to get root from Mon here

ls -hl /cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/root
lrwxrwxrwx. 1 cvmfs cvmfs 101 Nov  2 07:27 /cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/root ->
 /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3python3/Mon/ROOT/HEAD/x86_64-centos7-gcc8-opt/bin/root

I tried to check for Tue area but it seems to be empty
Cheers,
Zdenek

Hi Zdenek,
argh sorry about that – the ROOT team does not directly maintain the LCG releases, I’ll ask the maintainers and get back to you asap.

Thank you for your patience!
Enrico

Hi,
turns out that the dev3python3 nightlies are deprecated and now simply called dev3 (i.e. Python 3 is now the default). I suggested they symlink dev3python3 to dev3 rather than leaving it behind like that, let’s see what happens :grinning_face_with_smiling_eyes:

[eguiraud@lxplus747 ~]$ source /cvmfs/sft.cern.ch/lcg/views/dev3/latest/x86_64-centos7-gcc8-opt/setup.sh 
which root
[eguiraud@lxplus747 ~]$ which root
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Fri/x86_64-centos7-gcc8-opt/bin/root
[eguiraud@lxplus747 ~]$ root 
   ------------------------------------------------------------------
  | Welcome to ROOT 6.23/01                        https://root.cern |
  | (c) 1995-2020, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Dec 04 2020, 02:37:00                 |
  | From heads/master@v6-23-01-RF-binSampling-258-ga88a734           |
  | With g++ (GCC) 8.3.0                                             |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

In the meanwhile I updated the ROOT website accordingly.

Let me know how that goes, and sorry this was not more straightforward.

Cheers,
Enrico

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.