New PyROOT can't pickle derived class types

Hi all,

I just switched from v6.20 to v6.24 and ran into this problem pickling an instance of a class that derives from a ROOT class. Eg, define a derived class that should be pickle-able in

import ROOT

class Derived(ROOT.TH1F):
    def __reduce__(self):
        return (self.__class__, ())

Then try to pickle it in

from Derived import Derived
import pickle

obj = Derived()
pkl = pickle.dumps(obj)

This fails with:

Traceback (most recent call last):
  File "", line 5, in <module>
    pkl = pickle.dumps(obj)
_pickle.PicklingError: Can't pickle <class 'Derived.Derived'>: attribute lookup Derived on cppyy.gbl.__cppyy_internal failed

The problem is that it can’t pickle self.__class__. I get the same error with pickle.dumps(obj.__class__) or pickle.dumps(Derived). This worked fine with v6.20 but doesn’t work with v6.22 or v6.24.

My current workaround is to write a wrapper function that calls the constructor and return that in __reduce__, ie:

import ROOT

def _new_Derived(*args):
    return Derived(*args)

class Derived(ROOT.TH1F):
    def __reduce__(self):
        return (_new_Derived, ())

Then I can happily pickle and unpickle Derived instances.

Anyway, I just thought I’d write this down for future reference (since google didn’t help me here), or in case this is something that can be fixed in future releases.


ROOT Version: 6.24/00
Platform: CentOS7
Compiler: gcc10

Hello Michael,

Thank you for reporting! This is now:

Hi @malexand ,
Thank you for reporting your issue. Let me add a bit more context. Your workaround is indeed the right way of calling the __reduce__ syntax on any python object you would want to pickle. You need to supply a function that in turn returns an object of your class for the machinery to work.

That being said, it shouldn’t be needed in general imho. If I try to remove the __reduce__ method from the class I always get a segfault:

vpadulan@fedorathinkpad-T550 [~]: python
Python 3.8.9 (default, Apr  6 2021, 00:00:00) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ROOT
>>> import pickle
>>> class Derived(ROOT.TH1F):
...     ...
>>> obj = Derived()
>>> pickle.dumps(obj)
 *** Break *** segmentation violation
#7  0x00007f6aeb43a8ac in TClass::StreamerTObject(TClass const*, void*, TBuffer&, TClass const*) () from /home/vpadulan/Programs/rootproject/rootinstall/v6-24-00/lib/
#8  0x00007f6aeaaba5a6 in TBufferFile::WriteObjectClass(void const*, TClass const*, bool) () from /home/vpadulan/Programs/rootproject/rootinstall/v6-24-00/lib/
#9  0x00007f6aeaac1b74 in TBufferIO::WriteObjectAny(void const*, TClass const*, bool) () from /home/vpadulan/Programs/rootproject/rootinstall/v6-24-00/lib/
#10 0x00007f6add1ca55f in op_reduce(CPyCppyy::CPPInstance*, _object*) () from /home/vpadulan/Programs/rootproject/rootinstall/v6-24-00/lib/

Which highlights a different issue. By the way, what Python version are you using?

Let me clarify more. With simple python classes your initial version of __reduce__ might work since the python builtin __class__ method can be used as a callable. But in this case the class is more complex, depends on cppyy (as the error suggests) and ROOT . So the builtin __class__ is not enough to return all the minimal information needed to construct an object of that kind (as the pickle documentation states).
I still believe it should not be needed in this case though.

Hi @vpadulan,

Thanks for the explanation! I’m using python 3.8.6. I’ve also tested on my laptop: OSX 10.14.6, ROOT 6.22/06, python 3.9.1, with the same results. I also see the segfault (on both machines) when trying to pickle without defining __reduce__. In ROOT <= v6.20 I saw that if I didn’t define __reduce__ then it would just pickle the base class (TH1F in this example), presumably using the definition of __reduce__ from the base class.

Normally, when I want to pickle something in a non-standard way I would define __getstate__ and __setstate__, but I’ve found that these are ignored for classes deriving from ROOT classes, and only defining __reduce__ works. Again, maybe this is because the base class defines __reduce__ and that takes precedence over any definition of __getstate__/__setstate__.



After some investigation, this has to do with how cross inheritance is handled in cppyy (and therefore in the new PyROOT).

In cppyy, when a Python class inherits from a C++ base class, a C++ wrapper class that inherits from the C++ base class is jitted. The proxied C++ object that is created for the derived Python class is an object of the jitted wrapper class. The trouble comes when trying to serialize an object of that jitted class: it crashes.

This explains why it’s possible to pickle an object from a ROOT class, but not an object whose class inherits from a ROOT class. By redefining __reduce__ in the derived Python class, you prevent the default __reduce__ to kick in and try ROOT serialization, so it’s good as workaround for now for cross-inheritance classes.

Also, I found out that the unpickling of derived Python-C++ objects in the old PyROOT didn’t really work, since what you got back when unpickling was an object of the base (C++) class, not of the derived (Python) class.

I made this PR to provide a better error message in case of pickling cross-inheritance objects:

Lack of support of I/O for jitted classes in ROOT makes it hard to provide a reasonable generic solution (dictionaries would need to be generated before serializing and be made available for deserialization too). Therefore, in the message it is suggested to implement a custom __reduce__ method, i.e. what Michael already did. Returning a callable (plus some arguments) in __reduce__ prevents an attempt to serialize the cross-inheritance object, which is what we want (the object will be constructed and its C++ wrapper jitted by the callable during deserialization).

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.