Custom build, crash when running the ATLAS example


ROOT Version: v6.14.00 && v6.18.00
Platform: CC7 / lxplus
Compiler: GCC 6.2


Hi all,

we have a custom installation of root which was build using this recipe: https://github.com/ShipSoft/shipdist/blob/master/root.sh#L87

When we try to run the atlas example, we get a crash.

Step to reproduce:

$ ssh -X lxplus 
$ source /cvmfs/ship.cern.ch/SHiP-2020/latest/setUp.sh
$ root /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.18.00/src/tutorials/eve/geom_atlas.C

Let me re-state that this does not happen with the default version of root provide in lxplus, only with our build.

Are there any suggestion on how to track the problem? How is the root build in lxplus build?

Hello Simone,

I have no idea what the problem is, but at least I created a stack trace. See below.

Would you be able to tell us how the root build was configured?

#0  0x00007ffff67e2fa0 in __memcpy_ssse3 () from /lib64/libc.so.6
#1  0x00007fffdb7fbbdb in llvm::WritableMemoryBuffer::getNewUninitMemBuffer(unsigned long, llvm::Twine const&) () from /usr/lib64/libLLVM-6.0-rhel.so
#2  0x00007fffdb7fbcaf in getMemBufferCopyImpl(llvm::StringRef, llvm::Twine const&) () from /usr/lib64/libLLVM-6.0-rhel.so
#3  0x00007fffdb7fbd30 in llvm::MemoryBuffer::getMemBufferCopy(llvm::StringRef, llvm::Twine const&) () from /usr/lib64/libLLVM-6.0-rhel.so

#4  0x00007fffdc8632f5 in (anonymous namespace)::createELFDebugObject(llvm::object::ObjectFile const&, (anonymous namespace)::LoadedELFObjectInfo const&) () from /usr/lib64/libLLVM-6.0-rhel.so
#5  0x00007fffdc863db2 in (anonymous namespace)::LoadedELFObjectInfo::getObjectForDebug(llvm::object::ObjectFile const&) const ()
   from /usr/lib64/libLLVM-6.0-rhel.so
#6  0x00007fffdc812495 in (anonymous namespace)::GDBJITRegistrationListener::NotifyObjectEmitted(llvm::object::ObjectFile const&, llvm::RuntimeDyld::LoadedObjectInfo const&) () from /usr/lib64/libLLVM-6.0-rhel.so
#7  0x00007fffdc82ef43 in llvm::MCJIT::NotifyObjectEmitted(llvm::object::ObjectFile const&, llvm::RuntimeDyld::LoadedObjectInfo const&) ()
   from /usr/lib64/libLLVM-6.0-rhel.so
#8  0x00007fffdc831e07 in llvm::MCJIT::generateCodeForModule(llvm::Module*) () from /usr/lib64/libLLVM-6.0-rhel.so
#9  0x00007fffdc82e560 in llvm::MCJIT::finalizeObject() () from /usr/lib64/libLLVM-6.0-rhel.so
#10 0x00007fffdc80ff42 in LLVMGetPointerToGlobal () from /usr/lib64/libLLVM-6.0-rhel.so
#11 0x00007fffdeb0c3f6 in llvmpipe_update_fs () from /usr/lib64/dri/swrast_dri.so
#12 0x00007fffdeb042d0 in llvmpipe_update_derived () from /usr/lib64/dri/swrast_dri.so
#13 0x00007fffdeaef7a6 in llvmpipe_draw_vbo () from /usr/lib64/dri/swrast_dri.so
#14 0x00007fffde5976df in st_draw_vbo () from /usr/lib64/dri/swrast_dri.so
#15 0x00007fffde55c22b in vbo_exec_vtx_flush () from /usr/lib64/dri/swrast_dri.so
#16 0x00007fffde558987 in vbo_exec_FlushVertices () from /usr/lib64/dri/swrast_dri.so
#17 0x00007fffde3fd61d in _mesa_EndList () from /usr/lib64/dri/swrast_dri.so
#18 0x00007fffe9000760 in TGLLogicalShape::Draw (this=0x7bc4b40, rnrCtx=...)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLLogicalShape.cxx:417
#19 0x00007fffe9031524 in TGLPhysicalShape::Draw (this=0x7c06960, rnrCtx=...)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLPhysicalShape.cxx:409
#20 0x00007fffe9049e23 in TGLScene::RenderElements (this=<optimized out>, rnrCtx=..., elVec=std::vector of length 2596, capacity 4096 = {...}, 
    check_timeout=<optimized out>, clipPlanes=<optimized out>)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLScene.cxx:919
#21 0x00007fffe904a52d in TGLScene::RenderAllPasses (this=0x2190fd0, rnrCtx=..., elVec=std::vector of length 2596, capacity 4096 = {...}, 
    check_timeout=true) at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLScene.cxx:818
#22 0x00007fffe907055e in TGLViewerBase::SubRenderScenes (this=this@entry=0x21a02b0, render_foo=&virtual table offset 152)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLViewerBase.cxx:411
#23 0x00007fffe907088f in TGLViewerBase::RenderOpaque (rnr_selected=true, rnr_non_selected=true, this=0x21a02b0)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLViewerBase.cxx:492
#24 TGLViewerBase::Render (this=this@entry=0x21a02b0)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLViewerBase.cxx:425
#25 0x00007fffe90755d2 in TGLViewer::Render (this=0x21a02a0)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLViewer.cxx:521
#26 TGLViewer::DoDrawMono (this=this@entry=0x21a02a0, swap_buffers=swap_buffers@entry=true)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLViewer.cxx:638
#27 0x00007fffe9075ddf in TGLViewer::DoDraw (this=this@entry=0x21a02a0, swap_buffers=swap_buffers@entry=true)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLViewer.cxx:600
#28 0x00007fffe90762b1 in TGLViewer::RequestDraw (this=this@entry=0x21a02a0, LODInput=LODInput@entry=100)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/gl/src/TGLViewer.cxx:464
---Type <return> to continue, or q <return> to quit---
#29 0x00007fffea18d222 in TEveViewerList::RepaintAllViewers (this=0x489cb00, resetCameras=<optimized out>, dropLogicals=<optimized out>)
    at /root/ship-2020/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/graf3d/eve/src/TEveViewer.cxx:466
#30 0x00007ffff7feb481 in ?? ()
#31 0x0000000000000000 in ?? ()

Ideas, @Axel, @amadio?

This is what I get:

$ ssh -Y lxplus
Warning: Permanently added the ECDSA host key for IP address '2001:1458:d00:16::3e' to the list of known hosts.
axel@lxplus.cern.ch's password: 
* ********************************************************************
* Welcome to lxplus772.cern.ch, CentOS, 7.6.1810
* Archive of news is available in /etc/motd-archive
* Reminder: you have agreed to the CERN
*   computing rules, in particular OC5. CERN implements
*   the measures necessary to ensure compliance.
*   https://cern.ch/ComputingRules
* Puppet environment: production, Roger state: production
* Foreman hostgroup: lxplus/nodes/login
* Availability zone: cern-geneva-c
* LXPLUS Public Login Service - https://cern.ch/lxplusdoc
* lxplus alias switch to CC7 on Apr 2nd 2019 . http://cern.ch/go/K7lq
* ********************************************************************
-bash-4.2$ source /cvmfs/ship.cern.ch/SHiP-2020/latest/setUp.sh
-bash-4.2$ root.exe /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.18.00/src/tutorials/eve/geom_atlas.C
   ------------------------------------------------------------
  | Welcome to ROOT 6.18/00                  https://root.cern |
  |                               (c) 1995-2019, The ROOT Team |
  | Built for linuxx8664gcc on Aug 12 2019, 16:56:00           |
  | From tag v6-18-00, 25 June 2019                            |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
   ------------------------------------------------------------

root [0] 
Processing /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.18.00/src/tutorials/eve/geom_atlas.C...
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Error in <TGLWidget::CreateWindow>: No good OpenGL visual found!
Error in <TEveViewer::SpawnGLViewer>: Insufficient support from the graphics hardware. Aborting.
-bash-4.2$ 

With ssh -X I get:

root [0] 
Processing /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.18.00/src/tutorials/eve/geom_atlas.C...
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Error in <TGLWidget::CreateWindow>: No good OpenGL visual found!
Error in <TEveViewer::SpawnGLViewer>: Insufficient support from the graphics hardware. Aborting.

I.e. looks like I cannot reproduce that crash over ssh. I’m doing that from an Ubuntu 18 machine (with GL). What are we doing differently?

Cheers, Axel.

@StephanH, yes the stacktrace looks exactly like the one that I got.

Sorry I could have posted it as well. Let me see if I find the precise build logs.

@Axel, ubuntu 16.04 here, but I have the same report from different users. Can you run the graphics with the standard demo? So without sourcing our environment.

Could the differences be caused by a misconfigured lxplus node? I suggest to rebuild with debug symbols to get proper line numbers. And maybe ldd one of the central libraries?

I get the same stack trace as @StephanH as well. I don’t know what could be the cause, but it could be a problem of system libraries from /usr/lib64 being incompatible with SHiP’s software in CVMFS. Is the target CentOS 7?

I am re-building the stack keeping the logs. I guess tomorrow morning I will be able to provide all the build information. But in general it build from this configuration: https://github.com/ShipSoft/shipdist/blob/master/root.sh#L87 with pretty much everything enable, but the alice stuff.

When I run with my own nightly build of root (LLVM external and in debug mode), I get this:

lxplus ~ $ /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/startprefix
Entering Gentoo Prefix /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64
lxplus ~ $ root /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.18.00/src/tutorials/eve/geom_atlas.C
   ------------------------------------------------------------
  | Welcome to ROOT 6.19/01                  https://root.cern |
  |                               (c) 1995-2019, The ROOT Team |
  | Built for linuxx8664gcc on Aug 12 2019, 23:54:00           |
  | From git-r3/HEAD@v6-19-01-751-gdbab2ea9e6                  |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
   ------------------------------------------------------------

root [0] 
Processing /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.18.00/src/tutorials/eve/geom_atlas.C...
gif2ASImage():2300:</cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/root/9999/icons/eve_text.gif> (null)
Warning in <TASImage::GetMask>: No image
root.exe: /cvmfs/portage/sys-devel/llvm-5.0.2/work/llvm-5.0.2.src/lib/Support/CommandLine.cpp:293: void {anonymous}::CommandLineParser::registerCategory(llvm::cl::OptionCategory*): Assertion `count_if(RegisteredOptionCategories, [cat](const OptionCategory *Category) { return cat->getName() == Category->getName(); }) == 0 && "Duplicate option categories"' failed.
lxplus ~ $ root-6.16 $(root-config --tutdir)/eve/geom_atlas.C
   ------------------------------------------------------------
  | Welcome to ROOT 6.16/00                  https://root.cern |
  |                               (c) 1995-2018, The ROOT Team |
  | Built for linuxx8664gcc on Jan 23 2019, 09:06:13           |
  | From tags/v6-16-00@v6-16-00                                |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
   ------------------------------------------------------------

root [0] 
Processing /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib/root/9999/tutorials/eve/geom_atlas.C...
gif2ASImage():2300:</cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/root/6.16/icons/eve_text.gif> (null)
Warning in <TASImage::GetMask>: No image
root.exe: /cvmfs/portage/sys-devel/llvm-5.0.2/work/llvm-5.0.2.src/lib/Support/CommandLine.cpp:293: void {anonymous}::CommandLineParser::registerCategory(llvm::cl::OptionCategory*): Assertion `count_if(RegisteredOptionCategories, [cat](const OptionCategory *Category) { return cat->getName() == Category->getName(); }) == 0 && "Duplicate option categories"' failed.
lxplus ~ $ root-6.14 $(root-config --tutdir)/eve/geom_atlas.C
   ------------------------------------------------------------
  | Welcome to ROOT 6.14/08                http://root.cern.ch |
  |                               (c) 1995-2018, The ROOT Team |
  | Built for linuxx8664gcc                                    |
  | From tags/v6-14-08@v6-14-08, Nov 22 2018, 08:48:56         |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
   ------------------------------------------------------------

root [0] 
Processing /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib/root/9999/tutorials/eve/geom_atlas.C...
gif2ASImage():2300:</cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/root/6.14/icons/eve_text.gif> (null)
Warning in <TASImage::GetMask>: No image
TEveManager::GetGeometry  loading: 'http://root.cern.ch/files/atlas.root' -> 'http://root.cern.ch/files/atlas.root'.
Info in <TGeoManager::Import>: Reading geometry from file: http://root.cern.ch/files/atlas.root
Info in <TFile::OpenFromCache>: using local cache copy of http://root.cern.ch/files/atlas.root [./files/atlas.root]
Info in <TGeoManager::CloseGeometry>: Geometry loaded from file...
Info in <TGeoManager::SetTopVolume>: Top volume is ATLS. Master volume is ATLS
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CloseGeometry>: Voxelization retrieved from file
Info in <TGeoManager::CountLevels>: max level = 12, max placements = 307
Info in <TGeoManager::CloseGeometry>: 29046966 nodes/ 7143 volume UID's in atlas.C
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Info in <TGeoManager::SetVisLevel>: Automatic visible depth disabled
Info in <TGeoManager::SetVisLevel>: Automatic visible depth disabled
Info in <TGeoManager::SetVisLevel>: Automatic visible depth disabled
root [1] .q

So it fails with master, 6.18, and 6.16, but works with 6.14.

@StephanH o, I don’t believe the problem is a misconfigured lxplus node. I have the same report from multiple users.

@amadio I am building of CC7 and then moving the files in /cvmfs it could be an incompatibility of some kind, but I would be surprised. Any suggestion on how to check it?

Try ROOT 6.14, if it works, then we will have to see why the assertion is triggered (see my example above). You could also compile ROOT in debug mode and make LLVM be in debug mode as well. You should then see the same assertion above trigger. Cheers,

@amadio Wait I am not quite sure I understand why you think it fails with 6.18 and 6.16.

My build of 6.18 and 6.16 fails, the standard one provide in lxplus seems ok (at least for 6.18).

@StephanH I got the build log, nothing jumped to my eyes, but honestly I don’t have much exeperience.


++ [[ -n '' ]]
++ cmake /root/ship-log-keep/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00 -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/root/ship-log-keep/sw/INSTALLROOT/8d5917bdde0043008f6bda1e3bedf6d47cfe667f/slc7_x86-64/ROOT/v6-18-00_FAIRSHIP-1 -DXROOTD_ROOT_DIR=/root/ship-log-keep/sw/slc7_x86-64/XRootD/v4.8.3-1 -Dcxx11=ON -Dfreetype=ON -Dbuiltin_freetype=OFF -Dpcre=OFF -Dbuiltin_pcre=ON -Drpath=ON -DCMAKE_CXX_COMPILER=c++ -DCMAKE_C_COMPILER=cc -DCMAKE_LINKER=c++ -DCMAKE_EXE_LINKER_FLAGS=-L/root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/lib64 -DOPENSSL_ROOT= -DLIBXML2_ROOT= -DGSL_DIR=/root/ship-log-keep/sw/slc7_x86-64/GSL/v1.16_FAIRSHIP-1 -DPYTHIA8_DIR=/root/ship-log-keep/sw/slc7_x86-64/pythia/v8230-ship-1 -DPYTHIA6_LIBRARY=/root/ship-log-keep/sw/slc7_x86-64/pythia6/v6.4.28-ship1-1/lib/libpythia6.so -Dpythia6=ON -Dpgsql=OFF -Dminuit2=ON -Dgdml=ON -Droofit=ON -Dhttp=ON -Dsoversion=ON -Dshadowpw=OFF -Dvdt=ON -Dbuiltin_vdt=ON -Dvmc=ON '-DCMAKE_PREFIX_PATH=/root/ship-log-keep/sw/slc7_x86-64/FreeType/v2.6-1;;/root/ship-log-keep/sw/slc7_x86-64/GSL/v1.16_FAIRSHIP-1;;/root/ship-log-keep/sw/slc7_x86-64/Python/v2.7.10-1;/root/ship-log-keep/sw/slc7_x86-64/Python-modules/1.0-1'
-- The C compiler identification is GNU 6.2.0
-- The CXX compiler identification is GNU 6.2.0
-- Check for working C compiler: /root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/cc
-- Check for working C compiler: /root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/c++
-- Check for working CXX compiler: /root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "1.8.3.1") 
-- ROOT default compression algorithm: zlib
CMake Deprecation Warning at cmake/modules/RootBuildOptions.cmake:358 (message):
  >>> Option 'vmc' is deprecated and will be removed in the next release of
  ROOT.  Please contact root-dev@cern.ch should you still need it.
Call Stack (most recent call first):
  CMakeLists.txt:109 (include)


-- Found GCC. Major version 6, minor version 2
CMake Deprecation Warning at cmake/modules/CheckCompiler.cmake:97 (message):
  Options cxx11/14/17 are deprecated.  Please use CMAKE_CXX_STANDARD instead.
Call Stack (most recent call first):
  CMakeLists.txt:111 (include)


-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found a 64bit system
-- Found GNU compiler collection
-- Performing Test GLIBCXX_USE_CXX11_ABI
-- Performing Test GLIBCXX_USE_CXX11_ABI - Success
-- ROOT Platform: linux
-- ROOT Architecture: linuxx8664gcc
-- Build Type: RELEASE
-- Compiler Flags: -fPIC -g -O2 -std=c++11 -std=c++11 -pipe  -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread 
-- Looking for ZLib
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.7") 
-- Looking for Freetype
-- Found Freetype: /root/ship-log-keep/sw/slc7_x86-64/FreeType/v2.6-1/lib/libfreetype.so (found version "2.8.0") 
-- Looking for LZMA
-- Found LZMA includes at /usr/include
-- Found LZMA library at /usr/lib64/liblzma.so
-- Looking for xxHash
-- Could NOT find xxHash (missing: xxHash_LIBRARY xxHash_INCLUDE_DIR) 
-- xxHash not found. Switching on builtin_xxhash option
-- Looking for LZ4
-- Could NOT find LZ4 (missing: LZ4_LIBRARY LZ4_INCLUDE_DIR) 
-- LZ4 not found. Switching on builtin_lz4 option
-- Looking for X11
-- Looking for XOpenDisplay in /usr/lib64/libX11.so;/usr/lib64/libXext.so
-- Looking for XOpenDisplay in /usr/lib64/libX11.so;/usr/lib64/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found X11: /usr/lib64/libX11.so
-- X11_INCLUDE_DIR: /usr/include
-- X11_LIBRARIES: /usr/lib64/libSM.so;/usr/lib64/libICE.so;/usr/lib64/libX11.so;/usr/lib64/libXext.so
-- X11_Xpm_INCLUDE_PATH: /usr/include
-- X11_Xpm_LIB: /usr/lib64/libXpm.so
-- X11_Xft_INCLUDE_PATH: /usr/include
-- X11_Xft_LIB: /usr/lib64/libXft.so
-- X11_Xext_INCLUDE_PATH: 
-- X11_Xext_LIB: /usr/lib64/libXext.so
-- Could NOT find GIF (missing: GIF_LIBRARY GIF_INCLUDE_DIR) 
-- Could NOT find TIFF (missing: TIFF_LIBRARY TIFF_INCLUDE_DIR) 
-- Found PNG: /usr/lib64/libpng.so (found version "1.5.13") 
-- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR) 
-- Building AfterImage library included in ROOT itself
-- Looking for GSL
-- Found PkgConfig: /root/ship-log-keep/sw/slc7_x86-64/autotools/v1.5.0-1/bin/pkg-config (found version "0.29.2") 
-- Found GSL: /root/ship-log-keep/sw/slc7_x86-64/GSL/v1.16_FAIRSHIP-1/include (found suitable version "1.16", minimum required is "1.10") 
-- Looking for python
-- Found PythonInterp: /root/ship-log-keep/sw/slc7_x86-64/Python/v2.7.10-1/bin/python (found version "2.7.10") 
-- Found PythonLibs: /root/ship-log-keep/sw/slc7_x86-64/Python/v2.7.10-1/lib/libpython2.7.so (found version "2.7.10") 
-- Looking for OpenGL
-- Found OpenGL: /usr/lib64/libGL.so  
-- Looking for gl2ps
-- Could NOT find GL2PS (missing: GL2PS_LIBRARY GL2PS_INCLUDE_DIR) 
-- gl2ps not found. Switching on builtin_gl2ps option
-- Looking for LibXml2
-- Found LibXml2: /usr/lib64/libxml2.so (found version "2.9.1") 
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.0.2k") 
-- Looking for MySQL
-- Could NOT find MYSQL (missing: MYSQL_INCLUDE_DIR MYSQL_LIBRARIES) 
-- MySQL not found. Switching off mysql option
-- Looking for Oracle
-- Oracle not found.
-- Oracle: You can specify includes: -DORACLE_PATH_INCLUDES=/usr/include/oracle/10.2.0.3/client
--    currently found includes: 
-- Oracle: You can specify libs: -DORACLE_PATH_LIB=/usr/lib/oracle/10.2.0.3/client/lib
--    currently found libs: ORACLE_LIBRARY_OCCI-NOTFOUND;ORACLE_LIBRARY_CLNTSH-NOTFOUND;ORACLE_LIBRARY_LNNZ-NOTFOUND
-- Oracle not found. Switching off oracle option
-- Looking for SQLite
-- Could NOT find Sqlite (missing: SQLITE_INCLUDE_DIR SQLITE_LIBRARIES) 
-- SQLite not found. Switching off sqlite option
-- Looking for Pythia6
-- Looking for Pythia8
-- Found Pythia8: /root/ship-log-keep/sw/slc7_x86-64/pythia/v8230-ship-1/include  
-- Looking for FFTW3
-- FFTW3 not found. Set [environment] variable FFTW_DIR to point to your FFTW3 installation
--                  Alternatively, you can also enable the option 'builtin_fftw3' to build FFTW3 internally'
--                  For the time being switching OFF 'fftw3' option
-- Looking for CFITSIO
-- Could NOT find CFITSIO (missing: CFITSIO_LIBRARY CFITSIO_INCLUDE_DIR) 
-- CFITSIO not found. You can enable the option 'builtin_cfitsio' to build the library internally'
--                    For the time being switching off 'fitsio' option
-- Looking for XROOTD
-- Found Xrootd version num: 4.8.3 (setting -DROOTXRDVERS=400080003)
--              libXrdMain not found: xproofd will be a wrapper around xrootd
--              include_dirs: /root/ship-log-keep/sw/slc7_x86-64/XRootD/v4.8.3-1/include/xrootd;/root/ship-log-keep/sw/slc7_x86-64/XRootD/v4.8.3-1/include/xrootd/private
--              libraries: /root/ship-log-keep/sw/slc7_x86-64/XRootD/v4.8.3-1/lib/libXrdUtils.so;/root/ship-log-keep/sw/slc7_x86-64/XRootD/v4.8.3-1/lib/libXrdClient.so;/root/ship-log-keep/sw/slc7_x86-64/XRootD/v4.8.3-1/lib/libXrdCl.so
-- Could NOT find GFAL (missing: GFAL_INCLUDE_DIR SRM_IFCE_INCLUDE_DIR GFAL_LIBRARY) 
-- GFAL library not found. Set variable GFAL_DIR to point to your gfal installation
                      and the variable SRM_IFCE_DIR to the srm_ifce installation
-- For the time being switching OFF 'gfal' option
-- Checking for module 'davix>=0.6.4'
--   No package 'davix' found
-- Looking for uuid_generate_random in uuid
-- Looking for uuid_generate_random in uuid - found
-- Found uuid: /usr/include  
-- Davix not found, switching ON 'builtin_davix' option.
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for TBB
-- Could NOT find TBB (missing: TBB_ROOT_DIR TBB_INCLUDE_DIR TBB_LIBRARY) 
-- TBB not found. Switching on builtin_tbb option
-- Performing Test CXX_HAS_mno_rtm
-- Performing Test CXX_HAS_mno_rtm - Success
-- Looking for BLAS for optional parts of TMVA
-- A library with BLAS API not found. Please specify library location.
-- Using GSL CBLAS for optional parts of TMVA
-- Looking for Numpy
-- Found NUMPY: /root/ship-log-keep/sw/slc7_x86-64/Python-modules/1.0-1/lib/python2.7/site-packages/numpy/core/include (found version "1.16.4") 
-- Enabled support for:  asimage astiff builtin_afterimage builtin_clang builtin_davix builtin_ftgl builtin_gl2ps builtin_glew builtin_llvm builtin_lz4 builtin_pcre builtin_tbb builtin_vdt builtin_xxhash clad cling davix exceptions explicitlink gdml http imt mathmore minuit2 opengl pch pythia6 pythia8 python roofit rpath shared soversion ssl thread tmva tmva-cpu tmva-pymva vmc vdt x11 xml xrootd
-- The ASM compiler identification is GNU
-- Found assembler: /root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/cc
-- Looking for backtrace
-- Looking for backtrace - found
-- backtrace facility detected in default set of libraries
-- Found Backtrace: /usr/include  
-- Native target architecture is X86
-- Threads disabled.
-- Doxygen disabled.
-- Go bindings disabled.
-- LLVM host triple: x86_64-unknown-linux-gnu
-- LLVM default target triple: x86_64-unknown-linux-gnu
-- Building with -fPIC
-- Constructing LLVMBuild project information
-- Linker detection: GNU ld
-- SysLibs: rt;dl;tinfo;-lpthread;/usr/lib64/libz.so
-- Targeting X86
-- Clang version: 5.0.0
-- Cling version (from VERSION file): ROOT_0.6~dev
-- Cling will look for C++ headers in '/root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.2.0/../../../../include/c++/6.2.0:/root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.2.0/../../../../include/c++/6.2.0/x86_64-unknown-linux-gnu:/root/ship-log-keep/sw/slc7_x86-64/GCC-Toolchain/v6.2.0-alice1-1/bin/../lib/gcc/x86_64-unknown-linux-gnu/6.2.0/../../../../include/c++/6.2.0/backward' at runtime.
-- And then fallback to: 'c++'
-- Performing Test CXX_HAS_Wno_maybe_uninitialized
-- Performing Test CXX_HAS_Wno_maybe_uninitialized - Success
-- Performing Test __result
-- Performing Test __result - Success
-- Performing Test CXX_HAS_fno_rtti
-- Performing Test CXX_HAS_fno_rtti - Success
Recording the git revision now
-- Performing Test C_HAS_Wno_strict_overflow
-- Performing Test C_HAS_Wno_strict_overflow - Success
-- Performing Test C_HAS_Wno_maybe_uninitialized
-- Performing Test C_HAS_Wno_maybe_uninitialized - Success
-- Performing Test C_HAS_Wno_parentheses_equality
-- Performing Test C_HAS_Wno_parentheses_equality - Success
-- Looking for m
-- Looking for m - not found
-- Performing Test CXX_HAS_Wno_nonnull_compare
-- Performing Test CXX_HAS_Wno_nonnull_compare - Success
-- Performing Test CXX_HAS_Wno_delete_non_virtual_dtor
-- Performing Test CXX_HAS_Wno_delete_non_virtual_dtor - Success
-- Performing Test CXX_HAS_Wno_misleading_indentation
-- Performing Test CXX_HAS_Wno_misleading_indentation - Success
-- Performing Test CXX_HAS_Wno_overloaded_virtual
-- Performing Test CXX_HAS_Wno_overloaded_virtual - Success
-- Performing Test CXX_HAS_fno_strict_aliasing
-- Performing Test CXX_HAS_fno_strict_aliasing - Success
-- Performing Test CXX_HAS_Wno_parentheses_equality
-- Performing Test CXX_HAS_Wno_parentheses_equality - Success
-- Performing Test CXX_HAS_Wno_cast_function_type
-- Performing Test CXX_HAS_Wno_cast_function_type - Success
-- Performing Test CXX_HAS_Wno_deprecated_register
-- Performing Test CXX_HAS_Wno_deprecated_register - Success
-- Performing Test CXX_HAS_Wno_register
-- Performing Test CXX_HAS_Wno_register - Success
-- Performing Test found_setresuid
-- Performing Test found_setresuid - Success
-- Performing Test found_stdstringview
-- Performing Test found_stdstringview - Failed
-- Performing Test found_stdexpstringview
-- Performing Test found_stdexpstringview - Failed
-- Performing Test found_stdapply
-- Performing Test found_stdapply - Failed
-- Performing Test found_stdinvoke
-- Performing Test found_stdinvoke - Failed
-- Performing Test found_stdindexsequence
-- Performing Test found_stdindexsequence - Failed
-- Performing Test found_attribute_always_inline
-- Performing Test found_attribute_always_inline - Success
-- Performing Test has_found_attribute_noinline
-- Performing Test has_found_attribute_noinline - Success
Running /root/ship-log-keep/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/build/unix/compiledata.sh
Making /root/ship-log-keep/sw/BUILD/8d5917bdde0043008f6bda1e3bedf6d47cfe667f/ROOT/include/compiledata.h
-- Configuring done
-- Generating done
-- Build files have been written to: /root/ship-log-keep/sw/BUILD/8d5917bdde0043008f6bda1e3bedf6d47cfe667f/ROOT
++ FEATURES='builtin_pcre mathmore xml ssl opengl minuit2 http gdml pythia8
            pythia6 roofit soversion vdt cxx11  xrootd
            
            '
++ NO_FEATURES=builtin_freetype
++ bin/root-config --features
cxx11 asimage astiff builtin_afterimage builtin_clang builtin_davix builtin_ftgl builtin_gl2ps builtin_glew builtin_llvm builtin_lz4 builtin_pcre builtin_tbb builtin_vdt builtin_xxhash clad cling davix exceptions explicitlink gdml http imt mathmore minuit2 opengl pch pythia6 pythia8 python roofit rpath shared soversion ssl thread tmva tmva-cpu tmva-pymva vmc vdt x11 xml xrootd
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-builtin_pcre
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-mathmore
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-xml
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-ssl
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-opengl
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-minuit2
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-http
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-gdml
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ grep -q yes
++ bin/root-config --has-pythia8
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-pythia6
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-roofit
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-soversion
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ bin/root-config --has-vdt
++ grep -q yes
++ for FEATURE in '$FEATURES'
++ grep -q yes
++ bin/root-config --has-cxx11
++ for FEATURE in '$FEATURES'
++ grep -q yes
++ bin/root-config --has-xrootd
++ for FEATURE in '$NO_FEATURES'
++ bin/root-config --has-builtin_freetype
++ grep -q no
++ [[ -n '' ]]
++ make -j32 install
Scanning dependencies of target copymodulemap
Scanning dependencies of target AFTERIMAGE
Scanning dependencies of target move_header_roofit_histfactory
Scanning dependencies of target VDT
Scanning dependencies of target TBB
Scanning dependencies of target move_header_core_clib
Scanning dependencies of target move_header_core_clingutils
Scanning dependencies of target DAVIX
Scanning dependencies of target xxhash
Scanning dependencies of target move_header_core_multiproc
Scanning dependencies of target PCRE
Scanning dependencies of target move_header_core_meta
Scanning dependencies of target move_header_core_textinput
Scanning dependencies of target move_header_core_foundation
Scanning dependencies of target move_header_core_rint
Scanning dependencies of target move_header_core_cont
Scanning dependencies of target move_header_core_thread
Scanning dependencies of target move_header_core_zip
Scanning dependencies of target move_header_core_imt
Scanning dependencies of target move_header_core_newdelete
Scanning dependencies of target move_header_core_lz4
Scanning dependencies of target move_header_core_lzma
Scanning dependencies of target move_header_core_unix
Scanning dependencies of target move_header_math_mathmore
Scanning dependencies of target move_header_core_base
[  0%] Copying header /root/ship-log-keep/sw/SOURCES/ROOT/v6-18-00_FAIRSHIP/v6-18-00/roofit/histfactory/inc/RooStats/HistFactory/Asimov.h to /root/ship-log-keep/sw/BUILD/8d5917bdde0043008f6bda1e3bedf6d47cfe667f/ROOT/include

I was curious about the compiler flags. Nothing extraordinary there, though.
The only thing that puzzles me is that you have -g, but no line numbers are displayed when it crashes. I see two explanations:

  • You added the -g only now, and the old build didn’t have it.
  • For some reason, different libraries are loaded. In that case, it doesn’t matter what you compiled with.

Hi Stephan,

indeed I was a little surprised by your comment as well. The logs clearly show where the problem is originated, quoting yourself:

so line 417 from file graf3d/gl/src/TGLLogicalShape.cxx root/graf3d/gl/src/TGLLogicalShape.cxx at master · root-project/root · GitHub

Then it goes into system libraries, but there is quite obvious that it misses the line number information, isn’t?

(Now, I tried to make a release with more debugs symbols passing an CMAKE_BUILD_TYPE="RELWITHDEBINFO" but as we can see from the log, it didn’t work).

Oh, I see. That’s because builds with -g for cling are huge. To get debug flags also for cling, you also need to set -DLLVM_BUILD_TYPE=RelWithDebInfo when configuring.

But I also see that it calls down into /usr/lib64/libLLVM-6.0-rhel.so, i.e. a system LLVM. Cling might not be the problem after all.

But then, it will use the one we have built or it will pick up from system anyway?

Maybe I am wrong, but I was guessing that we should not be too concerned about what happens much deeper into the stack.

Functions like _mesa_EndList should be quite ok, shouldn’t they?

If you guys believe this is useful I will try to build also with LLVM in debug mode, but it is a not trivial effort.

EDIT: Just saw your edit, indeed it is calling stuff from system, so I am not quite sure cling is at fault here.

_mesa_EndList might be ok, but I’m not sure that the correct libraries are loaded. Are you compiling on the same system setup that’s also being run on lxplus?

How do I check it?

$ uname -a
Linux cvm-perf02.cern.ch 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/system-release
CentOS Linux release 7.6.1810 (Core)

Maybe try to run the example before publishing the build to cvmfs.

Nice idea!

But in the build machine I don’t have X forward. Other suggestions?

Maybe a check in the library version?

(I am really sorry but I have zero experience with graphics and cling so I am really clueless about what I am doing and what is important to do!)

I check all the libraries that are used during compilation, and they are all equal (same size, claim same version) between the one that were used to build and the one offered in lxplus.

The only difference is libcrypto, but it seems uncorrelated.

Any other things to try?