Multiple proxies to process trees in one script


_ROOT Version: 6.06/09
_Built for linuxx8664gcc


Dear experts,

I want to read trees by processing multiple proxies in the same script one after another and independent of each other and append an example with two proxies in PyROOT.

My example does not work but gives the error

Info in TUnixSystem::ACLiC: creating shared library test2_h.so
g++: error: test1_h.so: No such file or directory
Error in : Compilation failed!
Error in TSelector::GetSelector: The file test2.h++ does not define a class named test2.

in line 27: tree2.Process(“test2.h++”). I can get it working by either not deleting the files related to the first proxy or to delete only the test1.* files, so no test1_h* files. I do not understand how these proxies should depend on each other and how I can get rid of this dependency. I need to remove this dependency because in my actual (more complicated) code I need to define objects afterwards in the automatically generated header files and get problems with redefinitions of variables.

Thanks for your help!
Thomas

Here is my example, which I was not allowed to put in my first post: test.py (1011 Bytes)

It looks like there is an implicit dependency to test1_h.so introduced by ACLiC. Maybe @pcanal could tell if it’s possible to get around this

Thanks for your fast reply. I am not completely sure, whether this is connected to ACLiC. When I remove the “++” in the Process lines, I get the following long list of problems in case I remove all files matching “test1*” before starting to work with the second proxy: error.txt (44.7 KB)

I just tried simpler Ntuples with only float branches and there it works to remove test1* before working with the second proxy. So my problem might be related to the more complicated structure of my trees? But in both examples I do basically the same and they run separately, independent of each other.

OK, thanks for the additional information, but @pcanal is our expert anyway :wink:

When building libraries with ACLiC, the system assume that the later libraries depends on the older libraries.
Adding the ++ force the rebuilding of the libraries and thus ‘forgets’ the previous dependency.

One option is to ‘prepare’ the library in a separate run of root/python:

tree1 = ROOT.TChain("tree1")
tree1.Add(path)
tree1.MakeProxy("test1.h", "macro1.C", "cut1.C")
ROOT.gROOT.ProcessLine(".L test1.h+")

and use

tree1 = ROOT.TChain("tree1")
tree1.Add(path)
tree1.Process("test1.h+")

Thanks for your answer!

Compiling in a separate run of root would be a quite inelegant change in my code. Do you think it should also work with separate (sub-) processes in python within the same script? (If so, how?)

You can execute

root.exe -b -l -q -e '.L test.h++'

or you can also request ACLiC to not link against loaded libraries either via the .rootrc or via

gEnv->SetValue("ACLiC.LinkLibs", 0);

Cheers,
Philippe.

I tried the first with “os.system” and “subprocess.call” in my script and I get the long error message as above in error.txt and I tried the second suggestion “ROOT.gEnv.SetValue(“ACLiC.LinkLibs”, 0)” directly after “import ROOT” and I get the following error:

sh: -c: line 0: une> xpected EOF while looking for matching `"’
sh: -c: line 1: syntax error: unexpected end of file
Error in : Compilation failed!
Error in TSelector::GetSelector: The file proxy_class_ef6871372082ada1fce441f5b3b9a4f0.h+ does not define a class named proxy_class_ef6871372082ada1fce441f5b3b9a4f0.

I think there is something special with gEnv in PyROOT.

CINT:

$ root -l
root [0] gEnv->GetValue("ACLiC.LinkLibs", -1)
(Int_t) -1
root [1] gEnv->SetValue("ACLiC.LinkLibs", 0)
root [2] gEnv->GetValue("ACLiC.LinkLibs", -1)
(Int_t) 0
root [3] .q

PyROOT:

$ python
Python 2.7.11 (default, Dec  1 2016, 22:08:21) 
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ROOT
>>> ROOT.gEnv.GetValue("ACLiC.LinkLibs", -1)
-1
>>> ROOT.gEnv.SetValue("ACLiC.LinkLibs", 0)
>>> ROOT.gEnv.GetValue("ACLiC.LinkLibs", -1)
1

Do you have an idea why the value 0 is not taken in PyROOT? I did both tests direcly one after another in the same setup/terminal.

The error.txt seems to give an ‘impossible’ error message:

In file included from /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0/src/DataFormats/Common/interface/RefVector.h:17:
/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0/src/DataFormats/Common/interface/RefVectorBase.h:22:5: error: constructor for 'edm::RefVectorMemberPointersHolder' must explicitly initialize the
      member 'memberPointers_' which does not have a default constructor
    RefVectorMemberPointersHolder() { }
    ^
/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_0/src/DataFormats/Common/interface/RefVectorBase.h:26:30: note: member is declared here
    std::vector<void const*> memberPointers_;
                             ^
/cvmfs/cms.cern.ch/slc6_amd64_gcc530/external/gcc/5.3.0/include/c++/5.3.0/x86_64-pc-linux-gnu/../../../../include/c++/5.3.0/bits/stl_vector.h:214:11: note: 
      'std::vector<const void *, std::allocator<const void *> >' declared here
    class vector : protected _Vector_base<_Tp, _Alloc>
          ^

where it seems to be complaining that ‘vector’ does not have a default constructor … which is very odd …

Can you upload test1.h and test2.h and all 4 macro files (and any corresponding .h file)?

My simple testing script runs with your suggestions. I only have problems with trees, that contain more complex branches than just floats, for example LorentzVectors. I would have to upload at least 7 files (maybe times 2 for the working and non-working versions) and I can only upload 2 files per post. So I suggest to discuss this via mail in case you want to see it.

And I agree with you that the error message is nonsense. It only appears if I call the second proxy as step 2. If I remove step 1 and only execute step 2, I do not get this message with the same test2.h file. So the error is not related to the MakeProxy call or the code this call creates.

I was able to reproduce the issue. I opened the following ticket for it:
https://sft.its.cern.ch/jira/browse/ROOT-10155

Thanks for the ticket!

Here are all my (temporarily created) files packed together in one txt file: files.txt (442.0 KB). And here is the code I use to create and use these proxies: https://github.com/artus-analysis/Artus/blob/master/HarryPlotter/python/utility/roottools.py#L376-L491. I get the long error message in case I uncomment the line os.remove(tmp_file), although I now do ROOT.gROOT.ProcessLine("gEnv->SetValue(\"ACLiC.LinkLibs\", 0)") at the beginning of the file.

os.remove(tmp_file)

That is likely a problem. Those files are loaded in memory at the same time as the library. They are loaded by cling/clang, parsed and stored in AST form. Removing them (especially externally to the clang) means that clang has a ‘dangling’ (of sort) reference to the file and it might (and seem to) cause havoc. I recommend removing them only at the end of the process.

Cheers,
Philippe.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.