AddressSanitizer memory error detector

Pepe_Le_Pew · April 27, 2016, 10:56pm

The default valgrind’s “memcheck” memory error detector will not find various {stack,global}-buffer overflow bugs. Valgrind does provide a dedicated complementary “exp-sgcheck” experimental stack and global array overrun detector but, for the time being, it does not work, as soon as ROOT related libraries are used and it seems that it is possible to use it.

On the other hand, gcc 4.8 and newer compilers provide their own AddressSanitizer (a.k.a. ASan) fast memory error detector (note: you may need to install the “libasan” library, e.g. “sudo apt-get install libgcc-4.8-dev” on Ubuntu 14.04 LTS, “yum install libasan” on CentOS 7.4, “sudo apt-get install libgcc-5-dev” on Ubuntu 16.04 LTS).

This is Ubuntu 14.04 LTS / x86_64 / gcc 4.8.4 / libasan.0 here …

First, create a simple “rootlogon.C” file:

{
  TString o(gSystem->GetMakeSharedLib());
  o = o.ReplaceAll("$Opt", "-O0 -g -fno-omit-frame-pointer -fsanitize=address");
  o = o.ReplaceAll("$LinkedLibs", "-fsanitize=address -lasan $LinkedLibs");
  gSystem->SetMakeSharedLib(o.Data());
}

Warning: the above “rootlogon.C” file will prevent valgrind usage (it is possibly related to a problem reported elsewhere).

Then, create a simple “trial.cxx” file:

#include <cstdio>
void trial(void) {
  int* pointer[8];
  for (int i = 1; i < 9; i++) {
    printf("i = %d\n", i);
    pointer[i] = 0;
    if (pointer[i]) printf("value = %d\n", (*(pointer[i])));
  }
}

Finally, execute:

ASAN_OPTIONS="abort_on_error=1" gdb -ex run --args `root-config --bindir`/root.exe -l -q trial.cxx++

and enjoy:

[...]$ ASAN_OPTIONS="abort_on_error=1" gdb -ex run --args `root-config --bindir`/root.exe -l -q trial.cxx++
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /opt/ROOT/v6-06-02/bin/root.exe...done.
Starting program: /opt/ROOT/v6-06-02/bin/root.exe -l -q trial.cxx++
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
root [0] 
Processing trial.cxx++...
Info in <TUnixSystem::ACLiC>: creating shared library /..././trial_cxx.so
i = 1
i = 2
i = 3
i = 4
i = 5
i = 6
i = 7
i = 8
=================================================================
==15431== ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffffffa4f0 at pc 0x7fffeba31aac bp 0x7fffffffa470 sp 0x7fffffffa468
WRITE of size 8 at 0x7fffffffa4f0 thread T0
    #0 0x7fffeba31aab (/.../trial_cxx.so+0x2aab)
    #1 0x7ffff7e41041 (+0x1041)
    #2 0x7ffff3a38fa2 (/opt/ROOT/v6-06-02/lib/libCling.so.6.06.02+0x35cfa2)
    #3 0x7ffff3a3e606 (/opt/ROOT/v6-06-02/lib/libCling.so.6.06.02+0x362606)
    #4 0x7ffff3a3e742 (/opt/ROOT/v6-06-02/lib/libCling.so.6.06.02+0x362742)
    #5 0x7ffff3ad68b2 (/opt/ROOT/v6-06-02/lib/libCling.so.6.06.02+0x3fa8b2)
    #6 0x7ffff39bf71c (/opt/ROOT/v6-06-02/lib/libCling.so.6.06.02+0x2e371c)
    #7 0x7ffff39ad48a (/opt/ROOT/v6-06-02/lib/libCling.so.6.06.02+0x2d148a)
    #8 0x7ffff7a2d1eb (/opt/ROOT/v6-06-02/lib/libCore.so.6.06.02+0x2251eb)
    #9 0x7ffff7a2c71c (/opt/ROOT/v6-06-02/lib/libCore.so.6.06.02+0x22471c)
    #10 0x7ffff75ff1e4 (/opt/ROOT/v6-06-02/lib/libRint.so.6.06.02+0x1b1e4)
    #11 0x7ffff76006c7 (/opt/ROOT/v6-06-02/lib/libRint.so.6.06.02+0x1c6c7)
    #12 0x40104b (/opt/ROOT/v6-06-02/bin/root.exe+0x40104b)
    #13 0x7ffff6d26ec4 (/lib/x86_64-linux-gnu/libc-2.19.so+0x21ec4)
    #14 0x4010b9 (/opt/ROOT/v6-06-02/bin/root.exe+0x4010b9)
Address 0x7fffffffa4f0 is located at offset 96 in frame <trial> of T0's stack:
  This frame has 1 object(s):
    [32, 96) 'pointer'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
Shadow bytes around the buggy address:
  0x10007fff7440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff7480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x10007fff7490: 00 00 f1 f1 f1 f1 00 00 00 00 00 00 00 00[f3]f3
  0x10007fff74a0: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff74b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff74c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff74d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x10007fff74e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==15431== ABORTING

Program received signal SIGABRT, Aborted.
0x00007ffff6d3bcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) where
#0  0x00007ffff6d3bcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff6d3f0d8 in __GI_abort () at abort.c:89
#2  0x00007fffe8abb829 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.0
#3  0x00007fffe8ab23ec in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.0
#4  0x00007fffe8ab9012 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.0
#5  0x00007fffe8ab8121 in __asan_report_error () from /usr/lib/x86_64-linux-gnu/libasan.so.0
#6  0x00007fffe8ab2827 in __asan_report_store8 () from /usr/lib/x86_64-linux-gnu/libasan.so.0
#7  0x00007fffeba31aac in trial () at /..././trial.cxx:6
#8  0x00007ffff7e41042 in __cling_Un1Qu31(void*) ()
#9  0x00007ffff3a38fa3 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /opt/ROOT/v6-06-02/lib/libCling.so
#10 0x00007ffff3a3e607 in cling::Interpreter::EvaluateInternal(std::string const&, cling::CompilationOptions, cling::Value*, cling::Transaction**) () from /opt/ROOT/v6-06-02/lib/libCling.so
#11 0x00007ffff3a3e743 in cling::Interpreter::process(std::string const&, cling::Value*, cling::Transaction**) () from /opt/ROOT/v6-06-02/lib/libCling.so
#12 0x00007ffff3ad68b3 in cling::MetaProcessor::process(char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /opt/ROOT/v6-06-02/lib/libCling.so
#13 0x00007ffff39bf71d in TCling::ProcessLine (this=0x67ae70, line=<optimized out>, error=0x7fffffffb94c) at /opt/ROOT/build/root-6.06.02/core/meta/src/TCling.cxx:1929
#14 0x00007ffff39ad48b in TCling::ProcessLineSynch (this=0x67ae70, line=0x1922c70 ".X  /..././trial.cxx++", error=0x7fffffffb94c) at /opt/ROOT/build/root-6.06.02/core/meta/src/TCling.cxx:2787
#15 0x00007ffff7a2d1ec in TApplication::ExecuteFile (file=<optimized out>, error=0x7fffffffb94c, keep=<optimized out>) at /opt/ROOT/build/root-6.06.02/core/base/src/TApplication.cxx:1129
#16 0x00007ffff7a2c71d in TApplication::ProcessLine (this=0x669520, line=<optimized out>, sync=<optimized out>, err=0x7fffffffb94c) at /opt/ROOT/build/root-6.06.02/core/base/src/TApplication.cxx:978
#17 0x00007ffff75ff1e5 in TRint::ProcessLineNr (this=this@entry=0x669520, filestem=filestem@entry=0x7ffff76024ae "ROOT_cli_", line=line@entry=0x7fffffffb950 ".x trial.cxx++", error=error@entry=0x7fffffffb94c)
    at /opt/ROOT/build/root-6.06.02/core/rint/src/TRint.cxx:745
#18 0x00007ffff76006c8 in TRint::Run (this=0x669520, retrn=<optimized out>) at /opt/ROOT/build/root-6.06.02/core/rint/src/TRint.cxx:420
#19 0x000000000040104c in main (argc=1, argv=0x7fffffffdad8) at /opt/ROOT/build/root-6.06.02/main/src/rmain.cxx:30
(gdb) frame 7
#7  0x00007fffeba31aac in trial () at /..././trial.cxx:6
6           pointer[i] = 0;
(gdb) list
1       #include <cstdio>
2       void trial(void) {
3         int* pointer[8];
4         for (int i = 1; i < 9; i++) {
5           printf("i = %d\n", i);
6           pointer[i] = 0;
7           if (pointer[i]) printf("value = %d\n", (*(pointer[i])));
8         }
9       }
(gdb) quit
A debugging session is active.

        Inferior 1 [process 15431] will be killed.

Quit anyway? (y or n) y
[...]$

I “tested” this approach on Ubuntu 14.04 LTS / x86_64 / gcc 4.8.4 / libasan.0 and CentOS 7.4 / x86_64 / gcc 4.8.5 / libasan.0 with ROOT v5-34-00-patches and 6.12/04 versions.

The above procedure seems to be working fine, as long as the “libasan.0” library (e.g. with gcc 4.8.x) is used. When one tries a newer compiler / “libasan” library version, it may fail with an error:

ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

One can still, however, enjoy the “libasan” library features. Execute:

ASAN_CXX="`root-config --cxx`" ASAN_LIB="`${ASAN_CXX} -print-file-name=libasan.so`" ASAN_OPTIONS="start_deactivated=true:abort_on_error=1" LD_PRELOAD="${ASAN_LIB}" gdb -ex run --args `root-config --bindir`/root.exe -l -q trial.cxx++

I “tested” this approach on Ubuntu 16.04 LTS / x86_64 / gcc 5.4.0 / libasan.2 with ROOT v5-34-00-patches and 6.12/04 versions.

Last, but not least … it seems that several “sanitizers” are closely related to LLVM, so maybe one could somehow switch them “on” in ROOT 6 and get a “native / built-in bug detector” (“on demand”, not “automatically”, as they probably slow down the code execution speed)?

Axel · May 10, 2016, 9:43am

Hi,

Thanks for your suggestion!

After the pointer check that we now have in the master, introducing other sanitizers would indeed be the obvious thing to do! We can even sort of distinguish between user code and “trusted code” - we could run the sanitizers by default on user code.

What’s missing is someone doing it. We have only a small team that’s still taking care of the core of ROOT. So realistically I do not see this happen any time soon. If you know of someone who could contribute please let me know.

Cheers, Axel.

steved · August 22, 2016, 2:34pm

Hi,

I’ve been using the AddressSanitizer for checking code for a while now. It works perfectly smoothly with my compiled executables.

However, it seems to break my ability to run an interpreter session in which my classes are loaded with gSystem->Load(). To clarify, I’m compiling all my libraries with a normal compiler, outside of ROOT, and then loading them with gSystem->Load() inside the interpreter. I get identical behaviour with clang and g++.

My set of errors (one per lib) are:

dlopen error: libMyLib.so: undefined symbol: __asan_option_detect_stack_use_after_return Load Error: Failed to load Dynamic link library libMyLib.so

Not that surprising - I haven’t loaded the right library. However, I haven’t found a way to do that properly - I’ve tried linking at the time I create the .so files (or just doing '-fno-omit-frame-pointer -fsanitize=address during the .so generation step) and using gSystem->Load on libasan before loading my own libraries. I’ve even tried I’ve tried LD_PRELOAD. None of them seem to work.

I found this thread googling for a solution - do you have any suggestions on how to fix it? At the moment I’m recompiling with and without ASan to either use or to use interpreted scripts, which is a massive hassle.

Cheers,