Filling independent trees in threads

ROOT
1

Hi,

I’ve been trying to get trees and files filling and writing in independent threads and have been running into problems. I’ve distilled an example into the shortest possible code that I can make. The code creates n trees and n files (serially) and then launches n threads. In each threads the trees are independently filled and written. If I set n=1 (ie only 1 thread), then the code runs fine. For n>1 I get seg faults. The data written to the tree is just an integer filled with the value 0 each time. If I surround the TTree::Fill command with TThread::Lock()/UnLock() statements then it does not segfault. I tried running the code within cint (from a compiled library) and from a stand alone executable with the same result.

I have attached three files - the .h/.cpp and a LinkDef file.

Below I’ve pasted the crash output I get if I run from within cint. Note that the exact output can be different on different occasions - so I have pasted four independent crash reports. I have also pasted below two gdb outputs (again it can be different)

I am runnning root 5.25/02 with gcc 4.2.1 on OS X 10.6.2

thanks for the help!
Peter

////////////////////////////////////
// Root Output 1 //
////////////////////////////////////

root [0] stressThreads(2, 1000)
running
Warning in : The rule for class: “TTree”: version, “[-16]” and data members: “fDefaultEntryOffsetLen” has been skipped because it conflicts with one of the other rules.

*** Break *** segmentation violation
done

*** Break *** segmentation violation
Root > Root >

////////////////////////////////////
// Root Output 2 //
////////////////////////////////////

root [0] stressThreads(2, 1000)
running

*** Break *** segmentation violation
Error: ‘vectorstd::string’ Incomplete template resolution in shared library /Users/peter/code/root_install//lib/root/libTree.so:-1:
Add following line in header for making dictionary
#pragma link C++ class vectorstd::string;
Error: ‘vector’ Incomplete template resolution in shared library /Users/peter/code/root_install//lib/root/libTree.so:-1:
Add following line in header for making dictionary
#pragma link C++ class vector;
Error: ‘vector’ Incomplete template resolution in shared library /Users/peter/code/root_install//lib/root/libTree.so:-1:
Add following line in header for making dictionary
#pragma link C++ class vector;
Error: ‘vector’ Incomplete template resolution in shared library /Users/peter/code/root_install//lib/root/libTree.so:-1:
Add following line in header for making dictionary
#pragma link C++ class vector;
Error: ‘vector<Int_t>’ Incomplete template resolution in shared library /Users/peter/code/root_install//lib/root/libTree.so:-1:
Add following line in header for making dictionary
#pragma link C++ class vector<Int_t>;
Error: ‘vector’ Incomplete template resolution in shared library /Users/peter/code/root_install//lib/root/libTree.so:-1:
Add following line in header for making dictionary
#pragma link C++ class vector;
Error: ‘vector’ Incomplete template resolution in shared library /Users/peter/code/root_install//lib/root/libTree.so:-1:
Add following line in header for making dictionary
#pragma link C++ class vector;
done
(class stressThreads)4340538336
*** Interpreter error recovered ***
root [1] Root >

////////////////////////////////////
// Root Output 3 //
////////////////////////////////////

root [0] stressThreads(2, 1000)
running
Error: class,struct,union or type TObject not defined /Users/peter/code/root_install//lib/root/libTree.so:-1:

*** Break *** segmentation violation

*** Break *** segmentation violation
Root > done
(class stressThreads)4340538336
*** Interpreter error recovered ***
Root >

////////////////////////////////////
// Root Output 4 //
////////////////////////////////////

root [0] stressThreads(2, 1000)
running
Error: cannot open file “Int_t” /Users/peter/code/root_install//lib/root/libTree.so:-1:
*** Interpreter error recovered ***
Error in TPluginManager::FindHandler: Cannot find plugin handler for TVirtualStreamerInfo! Does $ROOTSYS/etc/plugins/TVirtualStreamerInfo exist?

*** Break *** segmentation violation
Error in TPluginManager::FindHandler: Cannot find plugin handler for TVirtualStreamerInfo! Does $ROOTSYS/etc/plugins/TVirtualStreamerInfo exist?

*** Break *** segmentation violation
Root > done

*** Break *** segmentation violation
Root > Root >

////////////////////////////////////
// GDB Output 1 //
////////////////////////////////////

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000000
[Switching to process 8158]
0x00007fff82d55e72 in strcpy ()
(gdb) where
#0 0x00007fff82d55e72 in strcpy ()
#1 0x00000001009e173e in G__memfunc_para_setup ()
#2 0x00000001009e1e44 in G__parse_parameter_link ()
#3 0x00000001009e232f in G__memfunc_setup_imp ()
#4 0x000000010041df18 in G__setup_memfuncTVirtualStreamerInfo ()
#5 0x00000001009cf26b in G__incsetup_memfunc ()
#6 0x000000010091b07d in Cint::G__ClassInfo::HasMethod ()
#7 0x0000000100100a0f in TCint::ClassInfo_HasMethod ()
#8 0x00000001000e194c in TClass::Property ()
#9 0x00000001000e0209 in TClass::IsForeign ()
#10 0x00000001000ea720 in TClass::PostLoadCheck ()
#11 0x000000010001ec3c in TROOT::LoadClass ()
#12 0x000000010002a7c5 in TClass::GetClass ()
#13 0x000000010002887f in TBaseClass::GetClassPointer ()
#14 0x00000001000e089e in TClass::GetBaseClass ()
#15 0x00000001000e9e9b in TClass::InheritsFrom ()
#16 0x00000001000e1901 in TClass::Property ()
#17 0x00000001000e0209 in TClass::IsForeign ()
#18 0x00000001000ea720 in TClass::PostLoadCheck ()
#19 0x000000010001ec3c in TROOT::LoadClass ()
#20 0x0000000100019224 in TPluginHandler::LoadPlugin ()
#21 0x000000010002e066 in TVirtualStreamerInfo::Factory ()
#22 0x00000001000e06cb in TClass::GetStreamerInfo ()
#23 0x00000001010b46c0 in TStreamerInfo::Build ()
#24 0x0000000101079f14 in TBufferFile::WriteClassBuffer ()
#25 0x0000000101c8abae in TTree::Streamer ()
#26 0x00000001010a4263 in TKey::TKey ()
#27 0x000000010108e468 in TFile::CreateKey ()
#28 0x000000010108532a in TDirectoryFile::WriteTObject ()
#29 0x000000010008e78a in TObject::Write ()
#30 0x0000000100001551 in stressThreads::loop (p=<value temporarily unavailable, due to optimizations>) at src/stressThreads.cpp:53
#31 0x0000000102479ad3 in TThread::Function ()
#32 0x00007fff82d89f8e in _pthread_start ()
#33 0x00007fff82d89e41 in thread_start ()
(gdb)

////////////////////////////////////
// GDB Output 2 //
////////////////////////////////////
running
Warning in : The rule for class: “TTree”: version, “[-16]” and data members: “fDefaultEntryOffsetLen” has been skipped because it conflicts with one of the other rules.

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000068
[Switching to process 8447]
0x00007fff82da1cf1 in flockfile ()
(gdb) where
#0 0x00007fff82da1cf1 in flockfile ()
#1 0x00007fff82dd6c90 in fgetc ()
#2 0x00000001009c5597 in G__loadfile ()
#3 0x0000000100a0dd79 in G__process_cmd ()
#4 0x0000000100108e1b in TCint::ProcessLine ()
#5 0x0000000100107a83 in TCint::ProcessLineSynch ()
#6 0x00000001000705e2 in TApplication::ExecuteFile ()
#7 0x0000000100104f22 in TCint::ExecuteMacro ()
#8 0x000000010001d41f in TROOT::Macro ()
#9 0x0000000100019d5a in TPluginManager::LoadHandlerMacros ()
#10 0x000000010001a143 in TPluginManager::LoadHandlersFromPluginDirs ()
#11 0x000000010001aa8f in TPluginManager::FindHandler ()
#12 0x000000010002e056 in TVirtualStreamerInfo::Factory ()
#13 0x00000001000e06cb in TClass::GetStreamerInfo ()
#14 0x00000001010b46c0 in TStreamerInfo::Build ()
#15 0x0000000101079f14 in TBufferFile::WriteClassBuffer ()
#16 0x0000000101c8abae in TTree::Streamer ()
#17 0x00000001010a4263 in TKey::TKey ()
#18 0x000000010108e468 in TFile::CreateKey ()
#19 0x000000010108532a in TDirectoryFile::WriteTObject ()
#20 0x000000010008e78a in TObject::Write ()
#21 0x0000000100001551 in stressThreads::loop (p=<value temporarily unavailable, due to optimizations>) at src/stressThreads.cpp:53
#22 0x0000000102479ad3 in TThread::Function ()
#23 0x00007fff82d89f8e in _pthread_start ()
#24 0x00007fff82d89e41 in thread_start ()
(gdb)
LinkDef.h (65 Bytes)
stressThreads.h (497 Bytes)
stressThreads.cpp (1.56 KB)

2

FYI - I just tried with 5.25/04 with the same result

3

Hi,

The setting of the meta data information (TClass, TStreamerInfo) is not yet properly protected for thread use, so you will need to set it up completely before starting the first thread. In you case, you can simply run the following: // Make sure the meta is correctly set before starting the threads. TFile *tempFile = TFile::Open("/var/tmp/mytemp.root","RECREATE"); TTree *tempTree = new TTree("temp","temp"); int val = 0; tempTree->Branch("data",&val); tempTree->Fill(); tempFile->Write(); delete tempFile; gSystem->Unlink("/var/tmp/mytemp.root");[code] Also the setting of the branch in your code is most likely supposed to be:[code] trees[i]->Branch("data", &(thedata[i]), "data/I");Otherwise all the tree points to the same int.

The final writing of the TTree currently is:trees[threadIndex]->Write();which explicitly request for the TTree to be stored in the current directory this is most likely not what you want, so instead you should use: trees[threadIndex]->GetDirectory()->Write();. However writing the meta data seems to currently also be brittle and I recommend writing the files only outside of the threads.

I also added a setting of gDirectory and calls to files[i]->cd() to work around other small issues.

See the complete (working for me) example in the attachment.

Cheers,
Philippe.
stressThreads.cpp (2.16 KB)

4

Hi Philippe,

this does indeed work as advertised - thanks. I won’t claim to understand why or how :slight_smile:

I guess you guys are already thinking about building in mult-ithreading support into some of the core operations like file I/O. A tricky but IMO very necessary job as CPUs move to multi core (I’m already on 16 - most people have at least 2)

thanks as always for the help

5

Note that multi-threading does not scale to the case of many cores (say above 8 )

Rene