ROOT objects can no longer be pickled for shelve

[mwilkins]

When I try to save a ROOT object using shelve, it panics and fails. This started happening after I brought my local build up-to-date with the master branch–I am not sure what changes caused the problem. Reproducer follows. Output attached.

---

import shelve
import ROOT
df = ROOT.RDataFrame(10).Define('e', 'rdfentry_')
h = df.Histo1D('e')
shelf = shelve.open('test', 'n')
shelf['h'] = h

---

ROOT Version: master
Platform: macOS
Compiler: Not Provided

---

output.txt (19.8 KB)

[eguiraud]

Hi,
at what commit was your local build before? Recently, for some definition of recently, ROOT master has switched to a completely new version of PyROOT. My first guess would be that. You can see if switching -Dpyroot_experimental=OFF when building fixes the problem, if yes, that’s it, and I guess you should open a jira ticket.

Cheers,
Enrico

[mwilkins]

Unfortunately, setting that flag appears to break pyROOT completely:

In [1]: import ROOT
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-ee8dc4376aa8> in <module>()
----> 1 import ROOT

/Applications/root_build/lib/ROOT.py in <module>()
     22 ### system and interpreter setup ------------------------------------------------
     23 import os, sys, types
---> 24 import cppyy
     25 
     26 ## there's no version_info in 1.5.2

/Applications/root_build/lib/cppyy.py in <module>()
     59       sys.setdlopenflags( 0x100 | 0x2 )    # RTLD_GLOBAL | RTLD_NOW
     60 
---> 61    import libPyROOT as _backend
     62 
     63    # reset dl flags if needed

ImportError: dlopen(/Applications/root_build/lib/libPyROOT.so, 2): Library not loaded: @rpath/Python3.framework/Versions/3.7/Python3
  Referenced from: /Applications/root_build/lib/libPyROOT.so
  Reason: image not found

Total build command:

cmake -Dall=On -Dimt=On -DPYTHON_EXECUTABLE=/Users/michael/anaconda2/bin/python2.7 -DPYTHON_INCLUDE_DIR=/Users/michael/anaconda2/include/python2.7/ -DPYTHON_LIBRARY=/Users/michael/anaconda2/lib/libpython2.7.dylib -Dpyroot_experimental=OFF  ../root/ && make

Build log attached.
build_log.txt (856.3 KB)

[eguiraud]

Well – time to hand this off to PyROOT expert @etejedor

[etejedor]

Hi,

The regular pickling works in master with the new PyROOT:

>>> import ROOT, pickle
>>> h = ROOT.TH1D()
>>> pickle.dumps ( h )
b'\x80\x03clibROOTPythonizations3_7\n_CPPInstance__expand__\nq\x00B1\x02\x00\x00@\x00\x02-\xff\xff\xff\xffTH1D\x00@\x00\x02 \x00\x03@\x00\x01\xfe\x00\x08@\x00\x00\x0e\x00\x01\x00\x01\x00\x00\x00\x00\x03\x00\x00\...'

From what I see in the error stack you sent, shelve uses cPickle:

[/Users/michael/anaconda2/lib/python2.7/lib-dynload/cPickle.so] save (no debug info)
[/Users/michael/anaconda2/lib/python2.7/lib-dynload/cPickle.so] Pickler_dump (no debug info)

If I try the code above the cPickle instead of pickle it also works.

I don’t know how exactly shelve uses cPickle to trigger that error. Do you know what are the requirements of shelve with respect to the objects to be “shelved”?

[mwilkins]

Hi, @etejedor,

So it turns out the problem is just with pickling RDF pointers, not TH1D. So, adjusting the reproducer above,

import shelve
import ROOT
df = ROOT.RDataFrame(10).Define('e', 'rdfentry_')
h = df.Histo1D('e')
h_ = h.GetPtr()
shelf = shelve.open('test', 'n')
shelf['h'] = h_
shelf.close()

works, but

shelf['h'] = h

does not. Since this is a pointer to a non-pickled ROOT object, perhaps this behavior is expected.

[etejedor]

Hi @mwilkins ,

Yes, indeed you found the reason. We can’t serialize RResultPtrs (which is what e.g. Histo1D returns) because they contain a shared pointer to the underlying object, and we can’t serialize shared pointers at the moment.

You can also use GetValue to get the underlying object and shelve that.