Multithreaded RDataFrame Snapshot crashes after updating from 6.22/08 to 6.24

jblank · May 20, 2021, 2:39pm

ROOT Version: 6.24
Platform: CentOS7
Compiler: gcc 4.8.5

Hello,
our servers were recently updated from ROOT 6.22/08 to 6.24. I’ve been using RDataFrame in python for some time now and activated multithreading for faster computation. However, since the ROOT update without changing the Code, I now get multiple segmentation violations when saving filtered dataframes with the Snapshot option.

import ROOT
from ROOT import RDataFrame

cols = ROOT.std.vector('string')()
cols.push_back('some_cols_to_save')
ROOT.EnableImplicitMT(cores)
df_data = RDataFrame('myChannel', '/path/to/data.root')
df_mc = RDataFrame('myChannel', '/path/to/mc.root')
df_data = df_data.Filter('data_filter(x,y)')
df_mc = df_mc.Filter('mc_filter(x,y)')
df_data.Snapshot('myChannel', '/outpath/data.root', cols)
df_mc.Snapshot('myChannel', '/outpath/mc.root', cols)

When I run code like this I get the following error:

 *** Break *** segmentation violation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
===========================================================                                                                                                                 
There was a crash.                                                                                                                                                          
This is the entire stack trace of all threads:                                                                                                                              
===========================================================                                                                                                                                                                                                                                                                                              
Thread 2 (Thread 0x7fd524849700 (LWP 576922)):                                                                                                                              
#0  0x00007fd54728ade2 in pthread_cond_timedwait                                                                                                                                                                                                                                                                                                         
GLIBC_2.3.2 () from /lib64/libpthread.so.0                                                                                                                                  
#1  0x000055bafeb81d74 in PyCOND_TIMEDWAIT (us=<optimized out>, mut=0x55bafee0d8a0<_PyRuntime+1232>, cond=0x55bafee0d870 <_PyRuntime+1184>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/condvar.h:73                                                                                                            
#2  take_gil (tstate=0x55bb04209160, ceval=0x55bafee0d618 <_PyRuntime+584>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval_gil.h:206                                                                                                                                                                          
#3  PyEval_RestoreThread () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:399                                                 
#4  0x000055bafecc49a8 in pysleep (secs=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Modules/timemodule.c:1867             
#5  time_sleep () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Modules/timemodule.c:338                                                     
#6  0x000055bafeb8c9ab in cfunction_vectorcall_O () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/methodobject.c:482                
#7  0x000055bafec3c3c3 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd524867d20, callable=0x7fd54761be00) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:115                                                                                                           
#8  call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55bb04209160) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963                                                                                                                                             
#9  _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3469                                            
#10 0x000055bafec1e433 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd524867ba0) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4298                                                                                                                                                                            #11 _PyEval_EvalCodeWithName () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4298                                            
#12 0x000055bafec1f818 in _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:436                         
#13 0x000055bafeb89b6e in PyVectorcall_Call (kwargs=0x7fd5474e8070, tuple=0x55bafec1f4a0 <_PyFunction_Vectorcall>, callable=0x7fd53ff3ab80) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:200                                                                                                              
#14 PyObject_Call () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:228                                                        
#15 0x000055bafec3984f in do_call_core (kwdict=0x7fd5475de340, callargs=0x7fd5474e8070, func=0x7fd53ff3ab80, tstate=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:5010                                                                                                                    
#16 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3559                                           
#17 0x000055bafec1f646 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd524883040) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:738
#18 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:284                                                                                                                                    
#19 _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:411                                               
#20 0x000055bafec380f3 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd5248eebb8, callable=0x7fd525376280) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:115                                                                                                           
#21 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55bb04209160) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963                                                                                                                                             
#22 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3486                                            
#23 0x000055bafec1f646 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd5248eea40) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:738
#24 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:284                                                                                                                                    
#25 _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:411                                               
#26 0x000055bafec380f3 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd53fe46fb8, callable=0x7fd525376550) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:115                                                                                                           
#27 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55bb04209160) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963                                                                                                                                             
#28 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3486                                            
#29 0x000055bafec1f646 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd53fe46e40) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:738
#30 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:284                                                                                                                                    
#31 _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:411                                               
#32 0x000055bafec1fed1 in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:127                                                                                             
#33 method_vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/classobject.c:89                                              
#34 0x000055bafeb89b6e in PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, callable=0x7fd53fe34680) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:200                                                                                                                                     
#35 PyObject_Call () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:228                                                        
#36 0x000055bafecbd639 in t_bootstrap () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Modules/_threadmodule.c:1002                          
#37 0x000055bafec569c4 in pythread_wrapper (arg=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/thread_pthread.h:232   
#38 0x00007fd547286ea5 in start_thread () from /lib64/libpthread.so.0                                                                                                        
#39 0x00007fd54669e9fd in clone () from /lib64/libc.so.6                                                                                                                                                                                                                                                                                                  

Thread 1 (Thread 0x7fd5476a6740 (LWP 576892)):                                                                                                                               
#0  0x00007fd546665549 in waitpid () from /lib64/libc.so.6                                                                                                                   
#1  0x00007fd5465e2fb2 in do_system () from /lib64/libc.so.6                                                                                                                 
#2  0x00007fd5465e3361 in system () from /lib64/libc.so.6                                                                                                                    
#3  0x00007fd53fc7e707 in TUnixSystem::StackTrace() () from /opt/miniconda/envs/root_forge/bin/../lib/libCore.so.6.24                                                        
#4  0x00007fd53ff8360f in (anonymous namespace)::TExceptionHandlerImp::HandleException(int) () from /opt/miniconda/envs/root_forge/bin/../lib/libcppyy_backend3_8.so.6.24    
#5  0x00007fd53fc7b881 in TUnixSystem::DispatchSignals(ESignals) () from /opt/miniconda/envs/root_forge/bin/../lib/libCore.so.6.24                                           
#6  <signal handler called>                                                                                                                                                  
===========================================================                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
The lines below might hint at the cause of the crash.                                                                                                                        
You may get help by asking at the ROOT forum https://root.cern.ch/forum                                                                                                      
Only if you are really convinced it is a bug in ROOT then please submit a                                                                                                    
report at https://root.cern.ch/bugs Please post the ENTIRE stack trace                                                                                                       
from above as an attachment in addition to anything else                                                                                                                     
that might help us fixing this issue.                                                                                                                                        
===========================================================                                                                                                                                                                                                                                                                                                                                                                                                                                                                            

Fatal in <TBufferMerger>:  TBufferMergerFiles must be destroyed before the server                                                                                           aborting                                                                                                                                                                                                                                                                                                                                                  Thread 2 (Thread 0x7fd524849700 (LWP 576922)):                                                                                                                               
#0  0x00007fd54728ade2 in pthread_cond_timedwait                                                                                                                                                                                                                                                                                                          GLIBC_2.3.2 () from /lib64/libpthread.so.0                                                                                                                                   
#1  0x000055bafeb81d74 in PyCOND_TIMEDWAIT (us=<optimized out>, mut=0x55bafee0d8a0 <_PyRuntime+1232>, cond=0x55bafee0d870 <_PyRuntime+1184>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/condvar.h:73                                                                                                            
#2  take_gil (tstate=0x55bb04209160, ceval=0x55bafee0d618 <_PyRuntime+584>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval_gil.h:206                                                                                                                                                                          
#3  PyEval_RestoreThread () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:399                                                 
#4  0x000055bafecc49a8 in pysleep (secs=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Modules/timemodule.c:1867             
#5  time_sleep () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Modules/timemodule.c:338                                                     
#6  0x000055bafeb8c9ab in cfunction_vectorcall_O () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/methodobject.c:482                 
#7  0x000055bafec3c3c3 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd524867d20, callable=0x7fd54761be00) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:115                                                                                                           
#8  call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55bb04209160) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963                                                                                                                                             
#9  _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3469                                            
#10 0x000055bafec1e433 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd524867ba0) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4298                                                                                                                                                                            
#11 _PyEval_EvalCodeWithName () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4298                                           
#12 0x000055bafec1f818 in _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:436                         
#13 0x000055bafeb89b6e in PyVectorcall_Call (kwargs=0x7fd5474e8070, tuple=0x55bafec1f4a0 <_PyFunction_Vectorcall>, callable=0x7fd53ff3ab80) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:200                                                                                                              
#14 PyObject_Call () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:228                                                        
#15 0x000055bafec3984f in do_call_core (kwdict=0x7fd5475de340, callargs=0x7fd5474e8070, func=0x7fd53ff3ab80, tstate=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:5010                                                                                                                    
#16 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3559                                            
#17 0x000055bafec1f646 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd524883040) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:738
#18 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:284                                                                                                                                    
#19 _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:411                                               
#20 0x000055bafec380f3 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd5248eebb8, callable=0x7fd525376280) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:115                                                                                                           
#21 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55bb04209160) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963                                                                                                                                             
#22 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3486                                            
#23 0x000055bafec1f646 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd5248eea40) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:738
#24 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:284                                                                                                                                    
#25 _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:411                                               
#26 0x000055bafec380f3 in _PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>, args=0x7fd53fe46fb8, callable=0x7fd525376550) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:115                                                                                                           
#27 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=0x55bb04209160) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:4963                                                                                                                                             
#28 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:3486                                            
#29 0x000055bafec1f646 in PyEval_EvalFrameEx (throwflag=0, f=0x7fd53fe46e40) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/ceval.c:738
#30 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:284                                                                                                                                    
#31 _PyFunction_Vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:411                                               
#32 0x000055bafec1fed1 in _PyObject_Vectorcall (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Include/cpython/abstract.h:127                                                                                             
#33 method_vectorcall () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/classobject.c:89                                              
#34 0x000055bafeb89b6e in PyVectorcall_Call (kwargs=<optimized out>, tuple=<optimized out>, callable=0x7fd53fe34680) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:200                                                                                                                                     

#35 PyObject_Call () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Objects/call.c:228                                                        
#36 0x000055bafecbd639 in t_bootstrap () at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Modules/_threadmodule.c:1002                          
#37 0x000055bafec569c4 in pythread_wrapper (arg=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python-split_1620713755437/work/Python/thread_pthread.h:232  
#38 0x00007fd547286ea5 in start_thread () from /lib64/libpthread.so.0                                                                                                        
#39 0x00007fd54669e9fd in clone () from /lib64/libc.so.6                                                                                                                                                                                                                                                                                                  

Thread 1 (Thread 0x7fd5476a6740 (LWP 576892)):                                                                                                                               
#0  0x00007fd546665549 in waitpid () from /lib64/libc.so.6                                                                                                                   
#1  0x00007fd5465e2fb2 in do_system () from /lib64/libc.so.6                                                                                                                 
#2  0x00007fd5465e3361 in system () from /lib64/libc.so.6                                                                                                                    
#3  0x00007fd53fc7e707 in TUnixSystem::StackTrace() () from /opt/miniconda/envs/root_forge/bin/../lib/libCore.so.6.24                                                        
#4  0x00007fd53fb5c383 in DefaultErrorHandler(int, bool, char const*, char const*) () from /opt/miniconda/envs/root_forge/bin/../lib/libCore.so.6.24                         
#5  0x00007fd53fc0ea28 in ErrorHandler () from /opt/miniconda/envs/root_forge/bin/../lib/libCore.so.6.24                                                                     
#6  0x00007fd53fc0f2a6 in Fatal(char const*, char const*, ...) () from /opt/miniconda/envs/root_forge/bin/../lib/libCore.so.6.24                                             
#7  0x00007fd53e85cc3f in ROOT::Experimental::TBufferMerger::~TBufferMerger() () from /opt/miniconda/envs/root_forge/lib/libRIO.so                                           
#8  0x00007fd53e85cd2a in ROOT::Experimental::TBufferMerger::~TBufferMerger() () from /opt/miniconda/envs/root_forge/lib/libRIO.so                                           
ROOT::Internal::RDF::JitBuildAction(std::vector<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, 
std::char_traits<char>, std::allocator<char> > > > const&, 
std::shared_ptr<ROOT::Detail::RDF::RNodeBase>*, std::type_info const&, std::type_info const&, void*, 
TTree*, unsigned int, ROOT::Internal::RDF::RBookedDefines const&, ROOT::RDF::RDataSource*, 
std::weak_ptr<ROOT::Internal::RDF::RJittedAction>*) () from 
/opt/miniconda/envs/root_forge/lib/libROOTDataFrame.so.6.24.00                                                                                                                    
Aborted

When I leave out the EnableImplicitMT the code finishes without any errors, but that would be a sad solution, which increases my computation time by a factor of ~5. Also when I replace the Snapshot call with e.g. Count().GetValue() it works just fine (with and without multithrading), so it seems to be Snapshot specific. In the Release Notes for 6.24 I read that there was a change in Snapshot behaviour. Might this be related?

Thanks!

eguiraud · May 20, 2021, 2:52pm

Hi @jblank ,
welcome to the ROOT forum and thank you for the report. I’m sorry you encountered this issue, it does not seem to have to do with the change in behavior w.r.t. RVec I/O and the stacktrace points at an internal error.

I see you are in a conda environment. I will try to reproduce the crash and get back to you as soon as possible.

Cheers,
Enrico

jblank · May 20, 2021, 3:01pm

Hi @eguiraud,
thanks for the quick reply. If that helps I can give the specifications of the packages in the environment

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
_py-xgboost-mutex         2.0                       cpu_0    conda-forge
abseil-cpp                20210324.1           h9c3ff4c_0    conda-forge
afterimage                1.21              ha9998ff_1003    conda-forge
aioeasywebdav             2.4.0           py38h578d9bd_1001    conda-forge
aiohttp                   3.7.4            py38h497a2fe_0    conda-forge
alsa-lib                  1.2.3                h516909a_0    conda-forge
amply                     0.1.4                      py_0    conda-forge
anyio                     3.0.1            py38h578d9bd_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py38h497a2fe_2    conda-forge
arrow-cpp                 4.0.0           py38h8e6f228_3_cpu    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
async_generator           1.10                       py_0    conda-forge
atk-1.0                   2.36.0               h3371d22_4    conda-forge
attmap                    0.13.0             pyhd8ed1ab_0    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
awkward                   0.15.0             pyhd3deb0d_0    conda-forge
aws-c-cal                 0.5.7                h3622835_1    conda-forge
aws-c-common              0.5.11               h7f98852_0    conda-forge
aws-c-event-stream        0.2.7                h5011166_5    conda-forge
aws-c-io                  0.9.12               h7aa3e40_2    conda-forge
aws-checksums             0.1.11               hc0e0e8b_6    conda-forge
aws-sdk-cpp               1.8.186              h9ad65fb_2    conda-forge
azure-common              1.1.27             pyhd8ed1ab_0    conda-forge
azure-core                1.14.0             pyhd8ed1ab_0    conda-forge
azure-storage-blob        12.8.1             pyhd8ed1ab_0    conda-forge
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
bcrypt                    3.2.0            py38h497a2fe_1    conda-forge
binutils                  2.35.1               hdd6e379_2    conda-forge
binutils_impl_linux-64    2.35.1               h193b22a_2    conda-forge
binutils_linux-64         2.35                h67ddf6f_30    conda-forge
bleach                    3.3.0              pyh44b312d_0    conda-forge
blinker                   1.4                        py_1    conda-forge
boost                     1.74.0           py38hc10631b_3    conda-forge
boost-cpp                 1.74.0               hc6e9bd1_3    conda-forge
boost-histogram           1.0.2            py38h1fd1430_0    conda-forge
boto                      2.49.0                     py_0    conda-forge
boto3                     1.17.75            pyhd8ed1ab_0    conda-forge
botocore                  1.20.75            pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h9c3ff4c_4    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.1               h7f98852_1    conda-forge
c-compiler                1.1.3                h7f98852_0    conda-forge
ca-certificates           2020.12.5            ha878542_0    conda-forge
cachetools                4.2.2              pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
certifi                   2020.12.5        py38h578d9bd_1    conda-forge
cffi                      1.14.5           py38ha65f79e_0    conda-forge
cfitsio                   3.470                hb418390_7    conda-forge
chardet                   4.0.0            py38h578d9bd_1    conda-forge
coincbc                   2.10.5               hcee13e7_1    conda-forge
compilers                 1.1.3                ha770c72_0    conda-forge
configargparse            1.4.1              pyhd8ed1ab_0    conda-forge
cryptography              3.4.7            py38ha5dfef3_0    conda-forge
cxx-compiler              1.1.3                h4bd325d_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
datrie                    0.8.2            py38h497a2fe_2    conda-forge
davix                     0.7.6                hb44b51d_0    conda-forge
dbus                      1.13.6               h48d8840_2    conda-forge
decorator                 5.0.9              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
docutils                  0.17.1           py38h578d9bd_0    conda-forge
drmaa                     0.7.9                   py_1000    conda-forge
dropbox                   11.9.0             pyhd8ed1ab_0    conda-forge
entrypoints               0.3             pyhd8ed1ab_1003    conda-forge
expat                     2.3.0                h9c3ff4c_0    conda-forge
fftw                      3.3.9           nompi_h74d3f13_101    conda-forge
filechunkio               1.8                        py_2    conda-forge
filelock                  3.0.12             pyh9f0ad1d_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fortran-compiler          1.1.3                h1990efc_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
ftgl                      2.4.0                hbcb1f35_0    conda-forge
ftputil                   5.0.1              pyhd8ed1ab_0    conda-forge
future                    0.18.2           py38h578d9bd_3    conda-forge
gcc_impl_linux-64         9.3.0               h70c0ae5_19    conda-forge
gcc_linux-64              9.3.0               hf25ea35_30    conda-forge
gct                       6.2.1550507116       h8cfc5f6_2    conda-forge
gdk-pixbuf                2.42.6               h04a7f16_0    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
gfortran_impl_linux-64    9.3.0               hc4a2995_19    conda-forge
gfortran_linux-64         9.3.0               hdc58fab_30    conda-forge
ghostscript               9.54.0               h9c3ff4c_1    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
gitdb                     4.0.7              pyhd8ed1ab_0    conda-forge
gitpython                 3.1.17             pyhd8ed1ab_0    conda-forge
gl2ps                     1.4.2                h0708190_0    conda-forge
glew                      2.1.0                h9c3ff4c_2    conda-forge
glib                      2.68.2               h9c3ff4c_0    conda-forge
glib-tools                2.68.2               h9c3ff4c_0    conda-forge
glog                      0.5.0                h48cff8f_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gmpy2                     2.1.0b1          py38hd744826_1    conda-forge
google-api-core           1.26.3             pyhd8ed1ab_0    conda-forge
google-api-python-client  2.4.0              pyhd8ed1ab_0    conda-forge
google-auth               1.30.0             pyh44b312d_0    conda-forge
google-auth-httplib2      0.1.0              pyhd8ed1ab_0    conda-forge
google-cloud-core         1.5.0              pyhd3deb0d_0    conda-forge
google-cloud-storage      1.38.0             pyh44b312d_0    conda-forge
google-crc32c             1.1.2            py38h8838a9a_0    conda-forge
google-resumable-media    1.2.0              pyhd3deb0d_0    conda-forge
googleapis-common-protos  1.53.0           py38h578d9bd_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
graphviz                  2.47.1               h85b4f2f_1    conda-forge
grpc-cpp                  1.37.1               h2519f57_2    conda-forge
grpcio                    1.37.1           py38hdd6454d_0    conda-forge
gsl                       2.6                  he838d99_2    conda-forge
gsoap_abi                 2.8.110              ha770c72_0    conda-forge
gst-plugins-base          1.18.4               hf529b03_2    conda-forge
gstreamer                 1.18.4               h76c114f_2    conda-forge
gtk2                      2.24.33              h539f30e_1    conda-forge
gts                       0.7.6                h64030ff_2    conda-forge
gxx_impl_linux-64         9.3.0               hd87eabc_19    conda-forge
gxx_linux-64              9.3.0               h3fbe746_30    conda-forge
harfbuzz                  2.8.1                h83ec7ef_0    conda-forge
htcondor                  9.0.0            py38h578d9bd_1    conda-forge
htcondor-classads         9.0.0                hd3c618e_1    conda-forge
htcondor-utils            9.0.0                hc20c069_1    conda-forge
httplib2                  0.19.1             pyhd8ed1ab_0    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
imagemagick               7.0.11_13       pl5320hb118871_0    conda-forge
iminuit                   2.6.1            py38h709712a_0    conda-forge
importlib-metadata        4.0.1            py38h578d9bd_0    conda-forge
influxdb                  5.3.1              pyhd3deb0d_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
ipykernel                 5.5.5            py38hd0cf306_0    conda-forge
ipyparallel               6.3.0            py38h578d9bd_2    conda-forge
ipython                   7.23.1           py38hd0cf306_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
isodate                   0.6.0                      py_1    conda-forge
jbig                      2.1               h7f98852_2002    conda-forge
jedi                      0.18.0           py38h578d9bd_2    conda-forge
jinja2                    3.0.1              pyhd8ed1ab_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.0.1              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
json5                     0.9.5              pyh9f0ad1d_0    conda-forge
jsonschema                3.2.0              pyhd8ed1ab_3    conda-forge
jupyter_client            6.1.12             pyhd8ed1ab_0    conda-forge
jupyter_core              4.7.1            py38h578d9bd_0    conda-forge
jupyter_server            1.7.0            py38h578d9bd_1    conda-forge
jupyterlab                3.0.16             pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.5.2              pyhd8ed1ab_0    conda-forge
kernel-headers_linux-64   2.6.32              h77966d4_13    conda-forge
kiwisolver                1.3.1            py38h1fd1430_1    conda-forge
krb5                      1.19.1               hcc1bbae_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libblas                   3.9.0                9_openblas    conda-forge
libcblas                  3.9.0                9_openblas    conda-forge
libclang                  11.1.0          default_ha53f305_1    conda-forge
libcondor_utils           9.0.0                h6d110fb_1    conda-forge
libcrc32c                 1.1.1                h9c3ff4c_2    conda-forge
libcurl                   7.76.1               h2574ce0_2    conda-forge
libcxx                    11.1.0               h4bd325d_0    conda-forge
libcxxabi                 11.1.0               ha770c72_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-devel_linux-64     9.3.0               h7864c58_19    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libgd                     2.3.2                h78a0170_0    conda-forge
libgfortran-ng            9.3.0               hff62375_19    conda-forge
libgfortran5              9.3.0               hff62375_19    conda-forge
libglib                   2.68.2               h3e27bee_0    conda-forge
libglu                    9.0.0             he1b5a44_1001    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0                9_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libllvm11                 11.1.0               hf817b99_2    conda-forge
libllvm9                  9.0.1                hf817b99_2    conda-forge
libnghttp2                1.43.0               h812cca2_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     13.3                 hd57d9b9_0    conda-forge
libprotobuf               3.16.0               h780b84a_0    conda-forge
librsvg                   2.50.5               hc3c00ef_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libssh2                   1.9.0                ha56f1ee_6    conda-forge
libstdcxx-devel_linux-64  9.3.0               hb016644_19    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libthrift                 0.14.1               he6d91bd_1    conda-forge
libtiff                   4.2.0                hbd63e13_2    conda-forge
libtool                   2.4.6             h58526e2_1007    conda-forge
libutf8proc               2.6.1                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp                   1.2.0                h3452ae3_0    conda-forge
libwebp-base              1.2.0                h7f98852_2    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxgboost                1.4.0                h9c3ff4c_0    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.12               h72842e0_0    conda-forge
lightgbm                  3.2.1            py38h709712a_0    conda-forge
llvmlite                  0.36.0           py38h4630a5e_0    conda-forge
logmuse                   0.2.6              pyh8c360ce_0    conda-forge
lz4                       3.1.3            py38hebdc3cf_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
markupsafe                2.0.0            py38h497a2fe_0    conda-forge
matplotlib                3.4.2            py38h578d9bd_0    conda-forge
matplotlib-base           3.4.2            py38hcc49a3a_0    conda-forge
matplotlib-inline         0.1.2              pyhd8ed1ab_2    conda-forge
metakernel                0.27.5             pyhd3deb0d_0    conda-forge
mistune                   0.8.4           py38h497a2fe_1003    conda-forge
more-itertools            8.7.0              pyhd8ed1ab_1    conda-forge
mpc                       1.1.0             h04dde30_1009    conda-forge
mpfr                      4.0.2                he80fd80_1    conda-forge
mpmath                    1.2.1              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.2            py38h1fd1430_1    conda-forge
msrest                    0.6.21             pyh44b312d_0    conda-forge
multidict                 5.1.0            py38h497a2fe_1    conda-forge
munge                     0.5.13               h1a52d62_0    conda-forge
mysql-common              8.0.23               ha770c72_2    conda-forge
mysql-libs                8.0.23               h935591d_2    conda-forge
nbclassic                 0.2.8              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.3              pyhd8ed1ab_0    conda-forge
nbconvert                 6.0.7            py38h578d9bd_3    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nest-asyncio              1.5.1              pyhd8ed1ab_0    conda-forge
networkx                  2.5                        py_0    conda-forge
nlohmann_json             3.9.1                h9c3ff4c_1    conda-forge
notebook                  6.4.0              pyha770c72_0    conda-forge
nspr                      4.30                 h9c3ff4c_0    conda-forge
nss                       3.65                 hb5efdd6_0    conda-forge
numba                     0.53.1           py38h0e12cce_0    conda-forge
numpy                     1.20.2           py38h9894fe3_0    conda-forge
oauth2client              4.1.3                      py_0    conda-forge
oauthlib                  3.0.1                      py_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
orc                       1.6.7                h89a63ab_2    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pandas                    1.2.4            py38h1abd341_0    conda-forge
pandoc                    2.13                 h7f98852_0    conda-forge
pandocfilters             1.4.2                      py_1    conda-forge
pango                     1.48.4               hb8ff022_1    conda-forge
paramiko                  2.7.2              pyh9f0ad1d_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
parso                     0.8.2              pyhd8ed1ab_0    conda-forge
patsy                     0.5.1                      py_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
peppy                     0.31.1             pyhd8ed1ab_0    conda-forge
perl                      5.32.0               h36c2ea0_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.2.0            py38ha0e1e83_1    conda-forge
pip                       21.1.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pkg-config                0.29.2            h36c2ea0_1008    conda-forge
pluggy                    0.13.1           py38h578d9bd_4    conda-forge
ply                       3.11                       py_1    conda-forge
portalocker               1.7.0            py38h578d9bd_1    conda-forge
prettytable               2.1.0              pyhd8ed1ab_0    conda-forge
prometheus_client         0.10.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.18             pyha770c72_0    conda-forge
protobuf                  3.16.0           py38h709712a_0    conda-forge
psutil                    5.8.0            py38h497a2fe_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulp                      2.4              py38h578d9bd_0    conda-forge
py                        1.10.0             pyhd3deb0d_0    conda-forge
py-xgboost                1.4.0            py38h578d9bd_0    conda-forge
pyaml                     20.4.0             pyh9f0ad1d_0    conda-forge
pyarrow                   4.0.0           py38hc9229eb_3_cpu    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pygments                  2.9.0              pyhd8ed1ab_0    conda-forge
pygraphviz                1.7              py38h0d738da_0    conda-forge
pyjwt                     2.1.0              pyhd8ed1ab_0    conda-forge
pynacl                    1.4.0            py38h497a2fe_2    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyqt                      5.12.3           py38h578d9bd_7    conda-forge
pyqt-impl                 5.12.3           py38h7400c14_7    conda-forge
pyqt5-sip                 4.19.18          py38h709712a_7    conda-forge
pyqtchart                 5.12             py38h7400c14_7    conda-forge
pyqtwebengine             5.12.1           py38h7400c14_7    conda-forge
pyrsistent                0.17.3           py38h497a2fe_2    conda-forge
pysftp                    0.2.9                      py_1    conda-forge
pysocks                   1.7.1            py38h578d9bd_3    conda-forge
pytest                    6.2.4            py38h578d9bd_0    conda-forge
pythia8                   8.305            py38h709712a_0    conda-forge
python                    3.8.10          h49503c6_1_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-htcondor           9.0.0            py38h09861fb_1    conda-forge
python-irodsclient        0.9.0              pyhd8ed1ab_0    conda-forge
python-xxhash             2.0.2            py38h497a2fe_0    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
pyyaml                    5.4.1            py38h497a2fe_0    conda-forge
pyzmq                     22.0.3           py38h2035c66_1    conda-forge
qt                        5.12.9               hda022c4_4    conda-forge
ratelimiter               1.2.0                   py_1002    conda-forge
re2                       2021.04.01           h9c3ff4c_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
resample                  1.0.1                    pypi_0    pypi
root                      6.24.0           py38h1abd341_0    conda-forge
root_base                 6.24.0           py38h07e97b3_0    conda-forge
root_numpy                4.8.0           py38h45d0d40_13    conda-forge
root_pandas               0.7.0                      py_0    conda-forge
rsa                       4.7.2              pyh44b312d_0    conda-forge
s2n                       1.0.8                h9b69904_0    conda-forge
s3transfer                0.4.2              pyhd8ed1ab_0    conda-forge
scikit-learn              0.24.2           py38hdc147b9_0    conda-forge
scikit-optimize           0.8.1              pyh9f0ad1d_0    conda-forge
scipy                     1.6.3            py38h7b17777_0    conda-forge
seaborn                   0.11.1               hd8ed1ab_1    conda-forge
seaborn-base              0.11.1             pyhd8ed1ab_1    conda-forge
send2trash                1.5.0                      py_0    conda-forge
setuptools                49.6.0           py38h578d9bd_3    conda-forge
simplejson                3.17.2           py38h497a2fe_2    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
slacker                   0.14.0                     py_0    conda-forge
smart_open                5.0.0              pyhd8ed1ab_0    conda-forge
smmap                     3.0.5              pyh44b312d_0    conda-forge
snakemake                 6.3.0                hdfd78af_0    bioconda
snakemake-minimal         6.3.0              pyhdfd78af_0    bioconda
snappy                    1.1.8                he1b5a44_3    conda-forge
sniffio                   1.2.0            py38h578d9bd_1    conda-forge
sqlite                    3.35.5               h74cdb3f_0    conda-forge
statsmodels               0.12.2           py38h5c078b8_0    conda-forge
stone                     3.2.1              pyhd8ed1ab_0    conda-forge
stopit                    1.1.2                      py_0    conda-forge
sympy                     1.8              py38h578d9bd_0    conda-forge
sysroot_linux-64          2.12                h77966d4_13    conda-forge
tbb                       2020.2               h4bd325d_4    conda-forge
terminado                 0.9.5            py38h578d9bd_0    conda-forge
testpath                  0.5.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toposort                  1.6                pyhd8ed1ab_0    conda-forge
tornado                   6.1              py38h497a2fe_1    conda-forge
tqdm                      4.60.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.0.5                      py_0    conda-forge
typing-extensions         3.7.4.3                       0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
ubiquerg                  0.6.1              pyh9f0ad1d_0    conda-forge
uncertainties             3.1.5              pyhd8ed1ab_0    conda-forge
uproot                    3.13.0           py38h578d9bd_0    conda-forge
uproot-base               3.13.0           py38h578d9bd_0    conda-forge
uproot-methods            0.9.2              pyhd8ed1ab_0    conda-forge
uritemplate               3.0.1                      py_0    conda-forge
urllib3                   1.26.4             pyhd8ed1ab_0    conda-forge
vdt                       0.4.3                h9c3ff4c_0    conda-forge
vector-classes            1.4.1                h4bd325d_1    conda-forge
veracitools               0.1.3                      py_0    conda-forge
voms                      2.1.0rc0             h441846d_2    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          0.57.0           py38h578d9bd_4    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
wrapt                     1.12.1           py38h497a2fe_3    conda-forge
xgboost                   1.4.0            py38h578d9bd_0    conda-forge
xmlrunner                 1.7.7                      py_0    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.0.10               h7f98852_0    conda-forge
xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
xorg-libx11               1.6.12               h516909a_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxcursor           1.2.0                h516909a_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxfixes            5.0.3             h516909a_1004    conda-forge
xorg-libxft               2.3.3                h71203ad_0    conda-forge
xorg-libxpm               3.5.13               h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-libxt                1.1.5             h516909a_1003    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xrootd                    5.1.1            py38h62fff15_1    conda-forge
xxhash                    0.8.0                h7f98852_3    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.5.1            py38h1e0a361_0    conda-forge
zeromq                    4.3.4                h9c3ff4c_0    conda-forge
zipp                      3.4.1              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h516909a_1010    conda-forge
zstandard                 0.15.2           py38h497a2fe_0    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge

Cheers
Jonah

eguiraud · May 20, 2021, 3:04pm

I’m running this in a conda environment with ROOT 6.24 and I see no issues:

import ROOT
from ROOT import RDataFrame

ROOT.RDataFrame(100).Define('x', '100').Define('y', 'x').Snapshot('t', 'f.root')

cols = ROOT.std.vector('string')()
cols.push_back('x')
cols.push_back('y')
ROOT.EnableImplicitMT()
df_data = RDataFrame('t', 'f.root')
df_mc = RDataFrame('t', 'f.root')
df_data.Snapshot('myChannel', 'data.root', cols)
df_mc.Snapshot('myChannel', 'mc.root', cols)

I have been running that in a loop for a few minutes now. Does it crash for you? If not, can you share the exact code (possibly reduced to only the lines that are necessary to reproduce the crash) and the input files (also privately, e.g. via cernbox) with me?

Cheers,
Enrico

jblank · May 20, 2021, 3:28pm

Your example runs without problems for me. A minimum example to reproduce the crash is the following:

import ROOT
from ROOT import RDataFrame
f_d0pipi_as_d0d0 = 'MC_MagBoth_B0D0pipi_as_D0D0_15.root'
t = 'D02Kpi/DecayTree'

cols = ROOT.std.vector('string')()
cols.push_back('B0_FitDaughtersPVConst_M_0')
cols.push_back('Entry')
ROOT.EnableImplicitMT(20)
df_bkg = RDataFrame(t, f_d0pipi_as_d0d0)
df_bkg = df_bkg.Define("B0_FitDaughtersPVConst_M_0", "B0_FitDaughtersPVConst_M[0]").Filter("B0_FitDaughtersPVConst_M_0 > 5000")
df_bkg.Snapshot(t, 'test/bkg.root', cols)

I’ll get back to you for the input in a minute

Cheers
Jonah

eguiraud · May 20, 2021, 5:02pm

I can reproduce it, now debugging

eguiraud · May 21, 2021, 10:54am

Hi @jblank ,
in case it helps, a workaround for the moment is to not save the output tree in a subdirectory, e.g. saving it as “DecayTree” rather than “D02Kpi/DecayTree” in Snapshot.

I’m working on a fix but that will only be available in the next 6.24 patch release, 6.24/02.

Cheers,
Enrico

jblank · May 21, 2021, 12:38pm

Thanks @eguiraud,
that’ll work for now, since at the moment the subfolders were only for consistent naming reasons.

eguiraud · May 22, 2021, 10:05pm

For the record this is now [DF] Crash in multi-thread Snapshot in sub-directory · Issue #8226 · root-project/root · GitHub , work in progress

eguiraud · May 26, 2021, 12:49pm

Hi,
this is now fixed in master (future 6.26 release) and in the 6.24 patches branch (future 6.24.02 release).

Cheers,
Enrico

system · June 9, 2021, 12:49pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.