ROOT objects can no longer be pickled for shelve

When I try to save a ROOT object using shelve, it panics and fails. This started happening after I brought my local build up-to-date with the master branch–I am not sure what changes caused the problem. Reproducer follows. Output attached.

import shelve
import ROOT
df = ROOT.RDataFrame(10).Define('e', 'rdfentry_')
h = df.Histo1D('e')
shelf ='test', 'n')
shelf['h'] = h

ROOT Version: master
Platform: macOS
Compiler: Not Provided

output.txt (19.8 KB)

at what commit was your local build before? Recently, for some definition of recently, ROOT master has switched to a completely new version of PyROOT. My first guess would be that. You can see if switching -Dpyroot_experimental=OFF when building fixes the problem, if yes, that’s it, and I guess you should open a jira ticket.


Unfortunately, setting that flag appears to break pyROOT completely:

In [1]: import ROOT
ImportError                               Traceback (most recent call last)
<ipython-input-1-ee8dc4376aa8> in <module>()
----> 1 import ROOT

/Applications/root_build/lib/ in <module>()
     22 ### system and interpreter setup ------------------------------------------------
     23 import os, sys, types
---> 24 import cppyy
     26 ## there's no version_info in 1.5.2

/Applications/root_build/lib/ in <module>()
     59       sys.setdlopenflags( 0x100 | 0x2 )    # RTLD_GLOBAL | RTLD_NOW
---> 61    import libPyROOT as _backend
     63    # reset dl flags if needed

ImportError: dlopen(/Applications/root_build/lib/, 2): Library not loaded: @rpath/Python3.framework/Versions/3.7/Python3
  Referenced from: /Applications/root_build/lib/
  Reason: image not found

Total build command:

cmake -Dall=On -Dimt=On -DPYTHON_EXECUTABLE=/Users/michael/anaconda2/bin/python2.7 -DPYTHON_INCLUDE_DIR=/Users/michael/anaconda2/include/python2.7/ -DPYTHON_LIBRARY=/Users/michael/anaconda2/lib/libpython2.7.dylib -Dpyroot_experimental=OFF  ../root/ && make

Build log attached.
build_log.txt (856.3 KB)

Well – time to hand this off to PyROOT expert @etejedor :smiley:


The regular pickling works in master with the new PyROOT:

>>> import ROOT, pickle
>>> h = ROOT.TH1D()
>>> pickle.dumps ( h )
b'\x80\x03clibROOTPythonizations3_7\n_CPPInstance__expand__\nq\x00B1\x02\x00\x00@\x00\x02-\xff\xff\xff\xffTH1D\x00@\x00\x02 \x00\x03@\x00\x01\xfe\x00\x08@\x00\x00\x0e\x00\x01\x00\x01\x00\x00\x00\x00\x03\x00\x00\...'

From what I see in the error stack you sent, shelve uses cPickle:

[/Users/michael/anaconda2/lib/python2.7/lib-dynload/] save (no debug info)
[/Users/michael/anaconda2/lib/python2.7/lib-dynload/] Pickler_dump (no debug info)

If I try the code above the cPickle instead of pickle it also works.

I don’t know how exactly shelve uses cPickle to trigger that error. Do you know what are the requirements of shelve with respect to the objects to be “shelved”?

1 Like

Hi, @etejedor,

So it turns out the problem is just with pickling RDF pointers, not TH1D. So, adjusting the reproducer above,

import shelve
import ROOT
df = ROOT.RDataFrame(10).Define('e', 'rdfentry_')
h = df.Histo1D('e')
h_ = h.GetPtr()
shelf ='test', 'n')
shelf['h'] = h_

works, but

shelf['h'] = h

does not. Since this is a pointer to a non-pickled ROOT object, perhaps this behavior is expected.

Hi @mwilkins ,

Yes, indeed you found the reason. We can’t serialize RResultPtrs (which is what e.g. Histo1D returns) because they contain a shared pointer to the underlying object, and we can’t serialize shared pointers at the moment.

You can also use GetValue to get the underlying object and shelve that.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.