PyROOT memory leak when using array fields

ROOT
1

Hello,
I have a python data structure, and I want to serialize objects to a ROOT file:

class SimpleEvent(object):
   event_number = 0
   hits_per_channel = np.zeros(100, dtype=np.int16)

This is the C++ counterpart (SimpleEvent.cpp):

#include "TObject.h"

#ifndef SIMPLEEVENT
#define SIMPLEEVENT
 
class SimpleEvent : public TObject {

public:
	Int_t  event_number;
	Short_t  hits_per_channel[100];
    ClassDef(SimpleEvent, 2);
};

#endif

My code for serializing goes like this (minimal working example):

ROOT.gROOT.ProcessLine('.L %s++' % "SimpleEvent.cpp")

f = ROOT.TFile("memory_leak.root", "RECREATE")
event_tree = ROOT.TTree("Events",'Tree')
root_event = ROOT.SimpleEvent()
event_tree.Branch('events', 'SimpleEvent', root_event)

index = 0
event = SimpleEvent()
while True:
   event.event_number = index #  fill to mimic real data
   setattr(root_event, "event_number", event.event_number)

   for i, x in enumerate(event.hits_per_channel):
      event.hits_per_channel[i] = i*index #  fill to mimic real data
    
   root_field = getattr(root_event, "hits_per_channel")
   root_field_type = root_field.typecode.decode("UTF-8")
   root_field_new = array.array(root_field_type, event.hits_per_channel.tolist())
   setattr(root_event, "hits_per_channel", root_field_new)	     
   index+=1
   event_tree.Fill()
   print("Pass %s" %index)

In reality of course the loop is not infinite, but you can clearly see the memory is constantly growing. Is this expected?
I just want to fill the data from the python object to the ROOT one.

Thanks upfront for any help!
Daniela

P.S. ROOT=6.04/03, python=3.4

2

No need setattr/getattr: use ‘.’ member access. Anyway, p3 not really supported and p2 no leak.

-Dom

P.S. if want know: memory_getbuf in p3 increfs underlying view when PyROOT gets info (many places in code). So, must decref, but no. Argue file bug but Axel says [url=https://root-forum.cern.ch/t/arrays-from-root-c-and-unsigned-char/20223/1 more devs[/url], so p2 still best.

3

[quote=“Dominique”]No need setattr/getattr: use ‘.’ member access. Anyway, p3 not really supported and p2 no leak.

-Dom

P.S. if want know: memory_getbuf in p3 increfs underlying view when PyROOT gets info (many places in code). So, must decref, but no. Argue file bug but Axel says [url=https://root-forum.cern.ch/t/arrays-from-root-c-and-unsigned-char/20223/1 more devs[/url], so p2 still best.[/quote]

Thanks a lot for your reply Dominique. This is a showstopper as our software runs on python3.
Can you point me to where the changes should be made? TPyBufferFactory or…?

Daniela

4

Fix-diff for your issue:[code]diff --git a/bindings/pyroot/src/Utility.cxx b/bindings/pyroot/src/Utility.cxx
index ffeae8e…34368fb 100644
— a/bindings/pyroot/src/Utility.cxx
+++ b/bindings/pyroot/src/Utility.cxx
@@ -560,6 +560,7 @@ int PyROOT::Utility::GetBuffer( PyObject* pyobject, char tc, int size, void*& bu
((bufprocs->bf_getbuffer))( pyobject, &bufinfo, PyBUF_WRITABLE );
buf = (char)bufinfo.buf;
Py_ssize_t buflen = bufinfo.len;

  •  Py_DECREF( bufinfo.obj );

#endif

   if ( buf && check == kTRUE ) {[/code]

But general wrong! Change not in all p3, multi places (also TPyBufferFactory.cxx), and INCREF is for reason: shared life of view and buffer needs track (here: Py_buffer on stack and self outlive function, alors okay). So, fix work for you, real fix need devs.

-Dom

5

Edit: find API: void PyBuffer_Release(Py_buffer view). So can do:[code]diff --git a/bindings/pyroot/src/Utility.cxx b/bindings/pyroot/src/Utility.cxx
index ffeae8e…34368fb 100644
— a/bindings/pyroot/src/Utility.cxx
+++ b/bindings/pyroot/src/Utility.cxx
@@ -560,6 +560,7 @@ int PyROOT::Utility::GetBuffer( PyObject pyobject, char tc, int size, void*& bu
((bufprocs->bf_getbuffer))( pyobject, &bufinfo, PyBUF_WRITABLE );
buf = (char)bufinfo.buf;
Py_ssize_t buflen = bufinfo.len;

  •   PyBuffer_Release( &bufinfo );

#endif

   if ( buf && check == kTRUE ) {[/code]

But API change 3.0 to 3.1 and only effect in later p3, so need version check.

-Dom

6

Perfect pointer, thank you very much Dominique!