CPyCppyy Segfault on mac m1

vcroft · July 15, 2022, 2:31pm

Hi, weird error here with an unfortunately bad test case.

We are trying to update documentation and testing for RooUnfold (http://roounfold.web.cern.ch and RooUnfold / RooUnfold · GitLab) but although everything compiles as normal on all linux servers (and all of our CI and all the SWAN instances I tried with) even the simplest call to our new base class containing all the main updates we would like to share crashes without info when run on a m1 Mac.

I’ve tried several versions of ROOT now, through Conda, Brew, compiling from source (only takes a few minutes!) and even using the latest docker image. Always the same error in the same place. RooUnfold compiles and links without issue. but calling the class fails.

A sample code is attached but will need to follow the RooUnfold install instructions. This gives the following stack trace:

 *** Break *** segmentation violation
[/Users/vincentcroft/ROOT/lib/libCore.so] TUnixSystem::DispatchSignals(ESignals) (no debug info)
[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)
[/Users/vincentcroft/ROOT/lib/libHistFactory.so] ParamHistFunc::evaluate() const (no debug info)
[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::traceEval(RooArgSet const*) const (no debug info)
[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::getValV(RooArgSet const*) const (no debug info)
[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooProduct::evaluate() const (no debug info)
[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::traceEval(RooArgSet const*) const (no debug info)
[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::getValV(RooArgSet const*) const (no debug info)
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooAbsReal::getVal(RooArgSet const*) const /Users/vincentcroft/ROOT/include/RooAbsReal.h:108
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] (anonymous namespace)::findParameters(RooUnfolding::RooFitHist*, int, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> > const&) /Users/vincentcroft/RooUnfold/src/RooUnfoldFitHelpers.cxx:421
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] (anonymous namespace)::findDependantsPerBin(RooUnfolding::RooFitHist*, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> > const&, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> >&) /Users/vincentcroft/RooUnfold/src/RooUnfoldFitHelpers.cxx:438
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfolding::RooFitHist::RooFitHist(RooAbsReal*, std::__1::vector<RooAbsArg*, std::__1::allocator<RooAbsArg*> > const&, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> > const&) /Users/vincentcroft/RooUnfold/src/RooUnfoldFitHelpers.cxx:518
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::makeHistogram(RooUnfoldSpec::HistContainer const&, double) /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:619
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::makeResponse() /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:689
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::unfold(RooUnfolding::Algorithm, double) /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:754
[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::makeFunc(RooUnfolding::Algorithm, double) /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:886
[<unknown binary>] (no debug info)
[/Users/vincentcroft/ROOT/lib/libcppyy_backend3_9.so] WrapperCall(long, unsigned long, void*, void*, void*) (no debug info)
[/Users/vincentcroft/ROOT/lib/libcppyy_backend3_9.so] Cppyy::CallR(long, void*, unsigned long, void*) (no debug info)
[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::(anonymous namespace)::InstancePtrExecutor::Execute(long, void*, CPyCppyy::CallContext*) (no debug info)
[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) (no debug info)
[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::CPPMethod::ExecuteProtected(void*, long, CPyCppyy::CallContext*) (no debug info)
[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::CPPMethod::Call(CPyCppyy::CPPInstance*&, _object*, _object*, CPyCppyy::CallContext*) (no debug info)
[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::(anonymous namespace)::mp_call(CPyCppyy::CPPOverload*, _object*, _object*) (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] _PyObject_MakeTpCall (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] call_function (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] _PyEval_EvalFrameDefault (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] _PyEval_EvalCode (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] pyrun_file (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] PyRun_SimpleFileExFlags (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] Py_RunMain (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] pymain_main (no debug info)
[/Users/vincentcroft/miniforge3/bin/python3.9] main (no debug info)
[/usr/lib/dyld] start (no debug info)
Traceback (most recent call last):
  File "/Users/vincentcroft/test_pur.py", line 88, in <module>
    response = spec.makeFunc(ROOT.RooUnfolding.kInvert).unfolding().response()
cppyy.ll.SegmentationViolation: RooAbsReal* RooUnfoldSpec::makeFunc(RooUnfolding::Algorithm alg, double regparam = -1.0E+30) =>
    SegmentationViolation: segfault in C++; program state was reset

test_RooUnfoldSpec.py (1.3 KB)

a very strange bug I think. I’m happy to wrestle with it more if there’s an easier way to look at this problem. I thought docker would help diagnose but alas. I think about 95% of our users get the code from cvmfs, or otherwise on institute clusters so it’s not been an issue yet but makes any local development really tough.

etejedor · July 18, 2022, 10:14am

Hi,

The two first things I’d try are:

Run this against a ROOT installation which has debug symbols (those are also available on CVMFS) and try to understand why some line in ParamHistFunc::evaluate() segfaults.
Translate the above reproducer to C++ and check whether it segfaults too when calling RooUnfoldSpec::makeFunc to discard it’s something python-specific.

jonas · July 18, 2022, 3:56pm

I’ll also take a look. I moved the question to the “Roofit and RooStats” category so I don’t forget about it. It can keep the Python tag too, so that should be no problem!

vcroft · July 19, 2022, 8:30am

ok great I’m giving this a go. In the mean time since I can’t use cvmfs (since that all works fine) I’m recompiling locally with debug symbols. This means I found the install options that I used, that I think were also used in the Conda compilation options too.

cmake -Dimt=OFF -Dbuiltin_tbb=OFF -Druntime_cxxmodules=OFF

vcroft · July 19, 2022, 8:32am

awesome! yeah I think this isn’t a bad move since the difference is that this class builds a wrapper around RooFit objects like ParamHistFunc anyway though I haven’t seen any issues in my other RooFit test cases.

vcroft · July 19, 2022, 9:24am

Here’s the debug enhanced stack trace.

Assertion failed: (0 <= idx && idx < static_cast<Int_t>(_list.size())), function operator[], file RooArgList.h, line 124.

*** Break *** abort

[/Users/vincentcroft/ROOT/lib/libcppyy_backend3_9.so] (anonymous namespace)::do_trace(int) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:183

[/Users/vincentcroft/ROOT/lib/libcppyy_backend3_9.so] (anonymous namespace)::TExceptionHandlerImp::HandleException(int) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:195

[/Users/vincentcroft/ROOT/lib/libCore.so] TUnixSystem::DispatchSignals(ESignals) /Users/vincentcroft/root-6.26.04/core/unix/src/TUnixSystem.cxx:3614

[/Users/vincentcroft/ROOT/lib/libCore.so] SigHandler(ESignals) /Users/vincentcroft/root-6.26.04/core/unix/src/TUnixSystem.cxx:395

[/Users/vincentcroft/ROOT/lib/libCore.so] sighandler(int) /Users/vincentcroft/root-6.26.04/core/unix/src/TUnixSystem.cxx:3591

[/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info)

[/usr/lib/system/libsystem_pthread.dylib] pthread_kill (no debug info)

[/usr/lib/system/libsystem_c.dylib] abort (no debug info)

[/usr/lib/system/libsystem_c.dylib] err (no debug info)

[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooArgList::operator[](int) const /Users/vincentcroft/root-6.26.04/roofit/roofitcore/inc/RooArgList.h:124

[/Users/vincentcroft/ROOT/lib/libHistFactory.so] ParamHistFunc::getParameter(int) const /Users/vincentcroft/root-6.26.04/roofit/histfactory/src/ParamHistFunc.cxx:217

[/Users/vincentcroft/ROOT/lib/libHistFactory.so] ParamHistFunc::getParameter() const /Users/vincentcroft/root-6.26.04/roofit/histfactory/src/ParamHistFunc.cxx:225

[/Users/vincentcroft/ROOT/lib/libHistFactory.so] ParamHistFunc::evaluate() const /Users/vincentcroft/root-6.26.04/roofit/histfactory/src/ParamHistFunc.cxx:592

[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::traceEval(RooArgSet const*) const /Users/vincentcroft/root-6.26.04/roofit/roofitcore/src/RooAbsReal.cxx:362

[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::getValV(RooArgSet const*) const /Users/vincentcroft/root-6.26.04/roofit/roofitcore/src/RooAbsReal.cxx:284

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooAbsReal::getVal(RooArgSet const*) const /Users/vincentcroft/ROOT/include/RooAbsReal.h:108

[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooProduct::evaluate() const /Users/vincentcroft/root-6.26.04/roofit/roofitcore/src/RooProduct.cxx:376

[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::traceEval(RooArgSet const*) const /Users/vincentcroft/root-6.26.04/roofit/roofitcore/src/RooAbsReal.cxx:362

[/Users/vincentcroft/ROOT/lib/libRooFitCore.so] RooAbsReal::getValV(RooArgSet const*) const /Users/vincentcroft/root-6.26.04/roofit/roofitcore/src/RooAbsReal.cxx:284

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooAbsReal::getVal(RooArgSet const*) const /Users/vincentcroft/ROOT/include/RooAbsReal.h:108

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] (anonymous namespace)::findParameters(RooUnfolding::RooFitHist*, int, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> > const&) /Users/vincentcroft/RooUnfold/src/RooUnfoldFitHelpers.cxx:421

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] (anonymous namespace)::findDependantsPerBin(RooUnfolding::RooFitHist*, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> > const&, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> >&) /Users/vincentcroft/RooUnfold/src/RooUnfoldFitHelpers.cxx:438

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfolding::RooFitHist::RooFitHist(RooAbsReal*, std::__1::vector<RooAbsArg*, std::__1::allocator<RooAbsArg*> > const&, std::__1::vector<RooRealVar*, std::__1::allocator<RooRealVar*> > const&) /Users/vincentcroft/RooUnfold/src/RooUnfoldFitHelpers.cxx:518

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::makeHistogram(RooUnfoldSpec::HistContainer const&, double) /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:619

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::makeResponse() /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:689

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::unfold(RooUnfolding::Algorithm, double) /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:754

[/Users/vincentcroft/RooUnfold/build/libRooUnfold.dylib] RooUnfoldSpec::makeFunc(RooUnfolding::Algorithm, double) /Users/vincentcroft/RooUnfold/src/RooFitUnfold.cxx:886

[<unknown binary>] (no debug info)

[/Users/vincentcroft/ROOT/lib/libcppyy_backend3_9.so] WrapperCall(long, unsigned long, void*, void*, void*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:779

[/Users/vincentcroft/ROOT/lib/libcppyy_backend3_9.so] Cppyy::CallR(long, void*, unsigned long, void*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:846

[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] GILCallR(long, void*, CPyCppyy::CallContext*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/CPyCppyy/src/Executors.cxx:78

[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::(anonymous namespace)::InstancePtrExecutor::Execute(long, void*, CPyCppyy::CallContext*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/CPyCppyy/src/Executors.cxx:579

[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/CPyCppyy/src/CPPMethod.cxx:74

[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::CPPMethod::ExecuteProtected(void*, long, CPyCppyy::CallContext*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/CPyCppyy/src/CPPMethod.cxx:149

[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/CPyCppyy/src/CPPMethod.cxx:728

[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::CPPMethod::Call(CPyCppyy::CPPInstance*&, _object*, _object*, CPyCppyy::CallContext*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/CPyCppyy/src/CPPMethod.cxx:783

[/Users/vincentcroft/ROOT/lib/libcppyy3_9.so] CPyCppyy::(anonymous namespace)::mp_call(CPyCppyy::CPPOverload*, _object*, _object*) /Users/vincentcroft/root-6.26.04/bindings/pyroot/cppyy/CPyCppyy/src/CPPOverload.cxx:566

[/Users/vincentcroft/miniforge3/bin/python3.9] _PyObject_MakeTpCall (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] call_function (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] _PyEval_EvalFrameDefault (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] _PyEval_EvalCode (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] pyrun_file (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] PyRun_SimpleFileExFlags (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] Py_RunMain (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] pymain_main (no debug info)

[/Users/vincentcroft/miniforge3/bin/python3.9] main (no debug info)

[/usr/lib/dyld] start (no debug info)

etejedor · July 19, 2022, 11:00am

Thanks for the updated info @vcroft , did you have the chance to try the same thing in C++?

@jonas any idea at this point on where the issue might come from?

vcroft · July 19, 2022, 12:20pm

Hi, indeed but in c++ it gets one step further before failing the same assertion as the debug gives us:

Assertion failed: (0 <= idx && idx < static_cast<Int_t>(_list.size())), function operator[], file RooArgList.h, line 124.

So not a pyroot problem at all it seems!

Edit: just checked to make sure this is the same issue, the c++ code works fine on x86 linux.

jonas · July 19, 2022, 3:38pm

Hi!

I found the problem. Unfortunately, it was a bug that i introduced myself into ROOT 6.26.

I have opened a PR on GitHub to fix this:

github.com/root-project/root

[RF] Fixup to parameter index calculation in ParamHistFunc

root-project:master ← guitargeek:ParamHistFunc_fixup

opened 03:03PM - 19 Jul 22 UTC

guitargeek

+7 -3

Last year, with commit 3657e7c, the parameter index calculation was changed to …be on the fly instead of using a look-up map, which is much faster. However, the implemented formula was not correct for two or three dimensions, which is fixed by this commit. To make sure that the index computation is correct this time, the new code was tested in this code snippet with various inputs: ```C++ void runTest(int nx = 42, int ny = 42, int nz = 42) { const int nxy = nx * ny; const int nyz = ny * nz; for (int i = 0; i < nx; ++i) { for (int j = 0; j < ny; ++j) { for (int k = 0; k < nz; ++k) { const int index = k + j * nz + i * ny * nz; const int gammaIndex = i + j * nx + k * nx * ny; const int i2 = index / nyz; const int tmp = index - i2 * nyz; const int j2 = tmp / nz; const int k2 = tmp - j2 * nz; const int gammaIndex2 = i2 + j2 * nx + k2 * nxy; if (gammaIndex != gammaIndex) { std::cout << "The unraveled indices were not correct!" << std::endl; return; } } } } } ``` Needs to be backported to the 6.26 branch to get into the 6.26.06 patch release. This commit the following problem reported on the forum: https://root-forum.cern.ch/t/cpycppyy-segfault-on-mac-m1/50822

The fix will enter then the next 6.26.XX patch release, which is 6.26.06 that will be released soon.

Cheers,
Jonas

vcroft · July 19, 2022, 4:32pm

Ah awesome job finding it! Weird that it only showed up on this one processor… I’ll try extracting the changes lines through and rebuild until the next release is due and let you know if it works.

vcroft · July 23, 2022, 4:50pm

I’ve implemented the fix locally and it all seems to work! thanks!

system · August 6, 2022, 4:50pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.