Randomly a segmentation fault for uninitialized TTree

ROOT
21

I didn’t change the Makefile.
Now there is a problem with compilation:

Undefined symbols for architecture x86_64:
“TClass::GetClass(std::type_info const&, bool, bool)”, referenced from:
int TTree::SetBranchAddress<std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > >(char const*, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >, TBranch) in mini.o
int TTree::SetBranchAddress(char const*, int*, TBranch**) in mini.o
int TTree::SetBranchAddress(char const*, double*, TBranch**) in mini.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [makeMini] Error 1

22

Again checking the obvious, did you rm -rf Dependencies Objects before recompiling?

23

No, sorry: you are right, of course… I will take a walk to relax… Stupid errors: I need a break.

It compiles, obviously.
I enclose the output by ValgrindValgrind.txt (207.6 KB) (which is definitely shorter than before)

I see things like
objc_msgLookup_fpret (in /usr/lib/libobjc.A.dylib)

In installing 6.22 I had this message:
Possible conflicting files are:
/usr/local/lib/libgmp.dylib → /usr/local/lib/libgmp.3.5.2.dylib

24

Do you still get a crash?
Do you still get any valgrind output if you remove the --leak-check=yes, which we don’t care about (and it should be all false positives)?

25

Sorry to jump in again (and for the randomness of my comment) .
I tried to compile your code on ROOT
From heads/v6-22-00-patches@v6-22-06-5-g5e69c10422
With the following CMakeLists.txt
CMakeLists.txt (677 Bytes)

I can run your code (./makeProva ) and doing it 20 times i never get a crash.
Just in case it’s something about the way you compile, you might need to switch to use a CMakeLists.txt and make your prova a library to link in your executable.

26

Sorry, I don’t like you waste time. I have randomly crash (I mean not always) when running a big program with a lot of histograms ploted and Minuit work: when the crash occurs, it is only after everything has been done and I am inspecting my ~30 plots (before quitting the running TApplication).
With the small version of that program (prova or mini) I never have a crash, but valgrind says things like
24 bytes in 1 blocks are possibly lost in loss record 2,165 of 8,473
and in the end

==73789== LEAK SUMMARY:
==73789== definitely lost: 13,008 bytes in 80 blocks
==73789== indirectly lost: 32 bytes in 1 blocks
==73789== possibly lost: 7,656 bytes in 161 blocks

I will inquire. I will also try on some lxplus. In case, I will let you know.

27

Try to compile all your code using (the compiler will do more checks then): -O3 -g -Wall -Wextra
Then run with “valgrind”.

BTW. Make sure that you use the same compiler which was used to build ROOT (i.e. check “root-config --cc --cxx --f77 --ld”).

28

ok, I will do!

29

Again, bytes lost don’t cause crashes :sweat: They possibly cause higher memory usage, but that’s a different issue.

Does ROOT print a stacktrace at the point of crash?
Can you provide a reproducer for the crash?

30

OK, without the leack-check I get
==81987==
==81987== HEAP SUMMARY:
==81987== in use at exit: 46,910,455 bytes in 67,571 blocks
==81987== total heap usage: 341,957 allocs, 274,386 frees, 185,029,169 bytes allocated
==81987==
==81987== LEAK SUMMARY:
==81987== definitely lost: 13,008 bytes in 80 blocks
==81987== indirectly lost: 32 bytes in 1 blocks
==81987== possibly lost: 7,656 bytes in 161 blocks
==81987== still reachable: 44,989,590 bytes in 59,467 blocks
==81987== of which reachable via heuristic:
==81987== newarray : 23,896 bytes in 41 blocks
==81987== suppressed: 1,900,169 bytes in 7,862 blocks
==81987== Rerun with --leak-check=full to see details of leaked memory
==81987==
==81987== For lists of detected and suppressed errors, rerun with: -s
==81987== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 3118 from 14)

31

That looks fine, 13kB of memory lost is most probably due to the ROOT interpreter just-in-time compiling something, and it should be harmless/expected.

32

OK, I am getting back onto the big program :mask:

33

Try with the AddressSanitizer, e.g.:

`root-config --cxx --cflags` -O0 -g -fno-omit-frame-pointer -fsanitize=address -Wall -Wextra -o a.out *.cpp -lasan `root-config --libs`
ASAN_OPTIONS="verbosity=1:abort_on_error=0" ./a.out
34

For today with the big program: version 6.22 of root, not using AddressSanitized (that I have to understand yet), compilation fine with
CXXFLAGS = -g -O3 -Wall -Wextra -fPIC -ftemplate-depth-30

Everything fine in running for three times; at the fourth time segmentation fault after all the histograms were plotted in their canvases.
I attach the messages given at the event:
segfault.txt (4.1 KB)

Line 135 of makeAnalysis.cpp is the very very and line of the main() (namely, containing the closing “}”

35

The very last lines in the main() are
app.Run(kTRUE); // kTRUE let the program continue after quitting ROOT
delete esempi;
delete analisi;
return 0 ;

Deleting object “analisi” means deleting all the histograms that were constructed and with which plots are done: I have never understood the timing of the quitting of the root session. It could randomly happen that root terminates the quitting process later then the “analisi” object is deleted… thus the segmentation fault! May it be?

36

Try (you should then see where it dies):

app.Run(kTRUE); // kTRUE = the main application eventloop returns here
gSystem->ProcessEvents(); gSystem->ProcessEvents(); // MacOS fix
std::cout << "first sleep" << std::endl; sleep(1);
// gSystem->ProcessEvents(); gSystem->ProcessEvents(); // MacOS fix
delete esempi;
gSystem->ProcessEvents(); gSystem->ProcessEvents(); // MacOS fix
std::cout << "second sleep" << std::endl; sleep(1);
// gSystem->ProcessEvents(); gSystem->ProcessEvents(); // MacOS fix
delete analisi;
gSystem->ProcessEvents(); gSystem->ProcessEvents(); // MacOS fix
std::cout << "third sleep" << std::endl; sleep(1);
// gSystem->ProcessEvents(); gSystem->ProcessEvents(); // MacOS fix
return 0;
37

In the meanwhile I run the big program with valgrind.

lazi@light allFiles % valgrind --tool=memcheck --suppressions=root-config --etcdir/valgrind-root.supp ./makeAnalysis 2&> Valgrind.output

Analysis set from June 01, 2020
Data as at February 02, 2021

zsh: segmentation fault valgrind --tool=memcheck ./makeAnalysis 2&> Valgrind.output
lazi@light allFiles %Valgrind.txt (8.5 KB)

38 
816] Booking histograms done!
makeAnalysis(14368,0x10f215dc0) malloc: Region cookie corrupted for region 0x7f816b800000 (value is 4695a000)[0x7f816b8081fc]
makeAnalysis(14368,0x10f215dc0) malloc: *** set a breakpoint in malloc_error_break to debug
zsh: abort      ./makeAnalysis

This is something I don’t know how to do!

39

I see the crash is in TGCocoa::DestroyWindow . Hard to tells what’s wrong just with this traceback. We would need some reproducer to go further.

40

I do not know what you mean for “reproducer”. Anyway I noticed something connected to creating the TApplication object. So I made a minimal program that looks like a “reproducer”.
Running with valgrind I have

lazi@light PROVA % valgrind -s --tool=memcheck  --suppressions=`root-config --etcdir`/valgrind-root.supp ./makeMini 2&> Valgrind.txt
TApplication setting
zsh: segmentation fault  valgrind -s --tool=memcheck  ./makeMini 2&> Valgrind.txt

I enclose a tar of the full staff, including the valgrind report.
Notice: I am using root 6.22/06 The ROOT Team; conception: R. Brun, F. Rademakers
Built for macosx64 on Nov 27 2020, 15:14:08 |
From tags/v6-22-06@v6-22-06
last_incredible.tgz (47.2 KB)