Crash when creating RDataFrame graph

zhubacek · November 17, 2020, 10:40pm

Continuing the discussion from RDataFrame MT performance running on remote files:

I create several Defines and Aggregates in loop(s), always in this format

latestDF = std::make_unique<RNode>(latestDF->Define(...))
histo = latestDF->Aggregate(...))

It seems to work correctly as the histograms are filled and saved after the analysis. But the ROOT::RDF::SaveGraph on the main RDataFrame fails:

I’m not sure if this could narrow what I might do wrong in my graph - the graph is created in the last line of code, the RDataFrame analysis seems to finish ok, histograms are saved etc…

===========================================================
#6  0x000000000073d0d1 in std::__shared_ptr<ROOT::Internal::RDF::GraphDrawing::GraphNode, (__gnu_cxx::_Lock_policy)2>::operator= (this=0x80) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/shared_ptr_base.h:1078
#7  0x000000000073d119 in std::shared_ptr<ROOT::Internal::RDF::GraphDrawing::GraphNode>::operator= (this=0x80) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/include/c++/8.3.0/bits/shared_ptr.h:103
#8  0x000000000073d14a in ROOT::Internal::RDF::GraphDrawing::GraphNode::SetPrevNode (this=0x0, node=...) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDF/GraphNode.hxx:75
#9  0x0000000000735d8a in ROOT::Internal::RDF::RActionCRTP<ROOT::Internal::RDF::RAction<ROOT::Internal::RDF::AggregateHelper<main(int, char**)::<lambda(TH2D&, const ZHCuts::Helper2DObservable&)>, main(int, char**)::<lambda(TH2D, TH2D)>, void, ZHCuts::Helper2DObservable, TH2D, false>, ROOT::Detail::RDF::RNodeBase, ROOT::TypeTraits::TypeList<ZHCuts::Helper2DObservable> > >::GetGraph(void) (this=0x97da9f0) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDF/RAction.hxx:187
#10 0x00007fdc71502ee5 in ROOT::Internal::RDF::GraphDrawing::GraphCreatorHelper::RepresentGraph[abi:cxx11](ROOT::Detail::RDF::RLoopManager*) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/lib/libROOTDataFrame.so
#11 0x00007fdc715032f4 in ROOT::Internal::RDF::GraphDrawing::GraphCreatorHelper::RepresentGraph[abi:cxx11](ROOT::RDataFrame&) () from /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/lib/libROOTDataFrame.so
#12 0x000000000074df37 in ROOT::Internal::RDF::GraphDrawing::GraphCreatorHelper::operator()<ROOT::RDataFrame> (this=0x7ffd6f98594f, node=...) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDF/GraphUtils.hxx:162
#13 0x00000000007447cf in ROOT::RDF::SaveGraph<ROOT::RDataFrame> (node=..., outputFile=...) at /cvmfs/sft.cern.ch/lcg/releases/LCG_98/ROOT/v6.22.00/x86_64-centos7-gcc8-opt/include/ROOT/RDFHelpers.hxx:128
#14 0x000000000070f224 in main (argc=2, argv=0x7ffd6f9884c8) at src/first_test.cxx:671
===========================================================

Thanks,
Zdenek

ROOT Version: 6.22/00
Platform: Centos7
Compiler: g++ 8.3.0

eguiraud · November 18, 2020, 9:26am

Hi,
this looks like a problem in SaveGraph (sorry about that!), but I could not reproduce the crash with simple combinations of Defines and Aggregates. Could you please share a reproducer that we can use to debug?

Cheers,
Enrico

zhubacek · November 19, 2020, 7:03pm

Hello,
I still can’t figure out how to reproduce it with a simple code, but when writing an example, I notice some strange features from the graph -
test3.C (3.2 KB) testtree2.root (6.8 KB)
it runs on a simple tree with float and vector<float> branches
it produces
new
which is what I wanted to achieve, however when

  //latestDF = std::make_unique<RNode>(latestDF->Define("r42",[](){return 42;}));

is uncommented, the graph changes to

However, the output root stays the same/the v_float part gets filled.
I also noticed when I define multiple columns

latestDF = std::make_unique<RNode>(latestDF->Define("A",...);
latestDF = std::make_unique<RNode>(latestDF->Define("B",...);
latestDF = std::make_unique<RNode>(latestDF->Define("C",...);

this gets plotted as T->C->B->A instead of T->A->B->C?

test4.C (751 Bytes)
Cheers,
Zdenek

eguiraud · November 19, 2020, 8:41pm

Uhm, probably related to https://sft.its.cern.ch/jira/browse/ROOT-9722 .

I’ll take a look asap.

eguiraud · November 20, 2020, 4:55pm

Found the problem, it’s a bug in SaveGraph, but not ROOT-9722. Work in progress

Thanks for reporting!

eguiraud · November 23, 2020, 10:52am

Ok so, the issue is now reported as https://github.com/root-project/root/issues/6853 and I should have a fix at https://github.com/root-project/root/pull/6861 .

Do you by chance have a setup with which you can test nightly ROOT builds?

Cheers,
Enrico

zhubacek · November 23, 2020, 1:07pm

Yes, I think I could set up the nightlies -
I setup

export LCGENV_PATH=/cvmfs/sft.cern.ch/lcg/releases
export PATH=/cvmfs/sft.cern.ch/lcg/releases/lcgenv/latest:${PATH}
eval "`lcgenv x86_64-centos7-gcc8-opt all`"

for gcc and then

source /cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/thisroot.sh
which root
/cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/root
root-config --version
6.23/01

At the moment, it gives the same graph as before, should I wait?
Cheers,
Zdenek

eguiraud · November 23, 2020, 1:27pm

Yes the pull request is still not merged – after it’s merged the next nightlies will contain the change. I’ll ping you here!

eguiraud · November 30, 2020, 9:49am

Hi,
I just merged the fix in master, tomorrow’s nightlies should contain the fix.

Cheers,
Enrico

zhubacek · December 2, 2020, 10:50am

Hi,
Is there an easy way for me to see to which nightly it was propagated?
When I setup the nigtly in the way above, I seem to get root from Mon here

ls -hl /cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/root
lrwxrwxrwx. 1 cvmfs cvmfs 101 Nov  2 07:27 /cvmfs/sft.cern.ch/lcg/views/dev3python3/latest/x86_64-centos7-gcc8-opt/bin/root ->
 /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3python3/Mon/ROOT/HEAD/x86_64-centos7-gcc8-opt/bin/root

I tried to check for Tue area but it seems to be empty
Cheers,
Zdenek

eguiraud · December 2, 2020, 11:28am

Hi Zdenek,
argh sorry about that – the ROOT team does not directly maintain the LCG releases, I’ll ask the maintainers and get back to you asap.

Thank you for your patience!
Enrico

eguiraud · December 4, 2020, 9:35am

Hi,
turns out that the dev3python3 nightlies are deprecated and now simply called dev3 (i.e. Python 3 is now the default). I suggested they symlink dev3python3 to dev3 rather than leaving it behind like that, let’s see what happens

[eguiraud@lxplus747 ~]$ source /cvmfs/sft.cern.ch/lcg/views/dev3/latest/x86_64-centos7-gcc8-opt/setup.sh 
which root
[eguiraud@lxplus747 ~]$ which root
/cvmfs/sft-nightlies.cern.ch/lcg/views/dev3/Fri/x86_64-centos7-gcc8-opt/bin/root
[eguiraud@lxplus747 ~]$ root 
   ------------------------------------------------------------------
  | Welcome to ROOT 6.23/01                        https://root.cern |
  | (c) 1995-2020, The ROOT Team; conception: R. Brun, F. Rademakers |
  | Built for linuxx8664gcc on Dec 04 2020, 02:37:00                 |
  | From heads/master@v6-23-01-RF-binSampling-258-ga88a734           |
  | With g++ (GCC) 8.3.0                                             |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'       |
   ------------------------------------------------------------------

In the meanwhile I updated the ROOT website accordingly.

Let me know how that goes, and sorry this was not more straightforward.

Cheers,
Enrico