Very strange build error in ROOT 6.14/04

Attila_Krasznahorkay · September 18, 2018, 1:45pm

Dear All,

I just came across a build error that I’ve never-ever seen before.

While building 6.14/04, I received this:

...
Scanning dependencies of target G__Imt
Scanning dependencies of target G__RIO
[ 75%] Generating G__Imt.cxx, ../../lib/libImt.rootmap
[ 75%] Generating G__RIO.cxx, ../../lib/libRIO.rootmap
[ 75%] Linking CXX executable ../bin/root.exe
[ 75%] Built target root.exe
[ 75%] Linking CXX shared library ../../lib/libunordered_multimapDict.so
[ 75%] Built target G__Imt
Scanning dependencies of target Imt
[ 75%] Building CXX object core/imt/CMakeFiles/Imt.dir/src/TThreadExecutor.cxx.o
[ 75%] Building CXX object core/imt/CMakeFiles/Imt.dir/src/TImplicitMT.cxx.o
[ 75%] Building CXX object core/imt/CMakeFiles/Imt.dir/G__Imt.cxx.o
[ 75%] Building CXX object core/imt/CMakeFiles/Imt.dir/src/TTaskGroup.cxx.o
[ 75%] Building CXX object core/imt/CMakeFiles/Imt.dir/src/TPoolManager.cxx.o
[ 75%] Building CXX object core/imt/CMakeFiles/Imt.dir/src/base.cxx.o
[ 75%] Built target unordered_multimapDict
[ 75%] Built target G__RIO
Scanning dependencies of target RIOObjs
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TBufferIO.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TBufferText.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TBufferMerger.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/G__RIO.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/v7/src/TFile.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TBufferMergerFile.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TBufferFile.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TCollectionProxyFactory.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TArchiveFile.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TBufferJSON.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TContainerConverters.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TDirectoryFile.cxx.o
[ 75%] Building CXX object io/io/CMakeFiles/RIOObjs.dir/src/TEmulatedCollectionProxy.cxx.o
/build2/atlasexternals/1.0/AnalysisBaseExternals/src/ROOT-build/core/imt/G__Imt.cxx: In function 'void {anonymous}::TriggerDictionaryInitialization_libImt_Impl()':
/build2/atlasexternals/1.0/AnalysisBaseExternals/src/ROOT-build/core/imt/G__Imt.cxx:273:100: error: no matching function for call to 'TROOT::RegisterModule(const char [7], const char* [5], const char* [2], const char*&, const char*&, void (&)(), <brace-enclosed initializer list>, const char* [10], bool)'
         TriggerDictionaryInitialization_libImt_Impl, {}, classesHeaders, /*has no C++ module*/false);
                                                                                                    ^
In file included from /build2/atlasexternals/1.0/AnalysisBaseExternals/src/ROOT-build/core/imt/G__Imt.cxx:16:0:
/build2/atlasexternals/1.0/AnalysisBaseExternals/x86_64-slc6-gcc62-opt/lib/cmake/VecCore/../../../include/TROOT.h:305:22: note: candidate: static void TROOT::RegisterModule(const char*, const char**, const char**, const char*, const char*, void (*)(), const FwdDeclArgsToKeepCollection_t&, const char**)
    static void       RegisterModule(const char* modulename,
                      ^~~~~~~~~~~~~~
/build2/atlasexternals/1.0/AnalysisBaseExternals/x86_64-slc6-gcc62-opt/lib/cmake/VecCore/../../../include/TROOT.h:305:22: note:   candidate expects 8 arguments, 9 provided
make[5]: *** [core/imt/CMakeFiles/Imt.dir/G__Imt.cxx.o] Error 1
make[5]: *** Waiting for unfinished jobs....
...

I included a few more lines than absolutely necessary to show that ROOT was seemingly generating the G__Imt.cxx dictionary file at the right time. (My build setup is not necessarily the simplest, though ROOT should really see itself making a full build in this setup.) But somehow it ends up being broken.

This is the problematic call in G__Imt.cxx:

    static bool isInitialized = false;
    if (!isInitialized) {
      TROOT::RegisterModule("libImt",
        headers, includePaths, payloadCode, fwdDeclCode,
        TriggerDictionaryInitialization_libImt_Impl, {}, classesHeaders, /*has no C++ module*/false);
      isInitialized = true;
    }

But I can’t make heads or tails of it. Other dictionary source files, which succeeded with their build, have calls exactly like this. I really don’t know what failed here.

And mind you, in a “local” build I don’t see this issue. It only shows up on our Continuous Integration machine. And I’m quite lost as to what the heck went wrong… Any ideas would be much appreciated.

Cheers,
Attila

ROOT Version: 6.14/04
Platform: x86_64-slc6-gcc62-opt
Compiler: GCC 6.2

pcanal · September 18, 2018, 9:55pm

The only idea would be that somehow on the include path of the old machine is an ancient install of ROOT that is somehow only used during the compilation of the G__Imt.cxx (if this is the case, doing a verbose build might help narrow it down).

Attila_Krasznahorkay · September 19, 2018, 7:06am

Hi Philippe,

At this point it seems more and more likely that the “security reboots” on our build machines are somehow responsible for this. As we started seeing a lot of weird failures yesterday in our builds.

I’ll come back here once/if I learn more.

Cheers,
Attila

Attila_Krasznahorkay · September 19, 2018, 9:49am

I believe I understand the issue now. (Though my test job is still running.) The problem seems to be that we didn’t forcefully turn on the build of VecCore (-Dbuiltin_veccore:BOOL=TRUE). So, in an incremental build ROOT just found the VecCore version installed by the previous build of it, put there by the “old” ROOT build.

Now, this resulted in the build flags looking like this for Imt:

# CMAKE generated file: DO NOT EDIT!
# Generated by "Unix Makefiles" Generator, CMake Version 3.11

# compile CXX with /cvmfs/sft.cern.ch/lcg/releases/gcc/6.2.0-2bc78/x86_64-slc6-gcc62-opt/bin/g++
CXX_FLAGS =  -pipe -m64  -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -std=c++14 -O3 -g -DNDEBUG -fPIC  

CXX_DEFINES = 

CXX_INCLUDES = -isystem /build2/atlasexternals/1.0/AnalysisBaseExternals/src/ROOT-build/include -I/build2/atlasexternals/1.0/AnalysisBaseExternals/x86_64-slc6-gcc62-opt/lib/cmake/VecCore/../../../include

The second include path points at the “installed location” of ROOT. A directory that at this point still holds TROOT.h coming from ROOT 6.12/06. And as it turns out, -isystem is treated with a lower priority than -I, even if it comes earlier in the argument list. (Not sure if this is intended, or if it’s a GCC bug…)

I would still argue that ROOT’s build system is not without fault in this setup. It really shouldn’t treat its own headers as “system headers”. Much less if it then goes and picks up some (sort of) system headers with -I.

For now I added -Dbuiltin_veccore:BOOL=TRUE to our ROOT configuration, let’s see if that will make the incremental build succeed.

Cheers,
Attila

Attila_Krasznahorkay · September 19, 2018, 9:58am

Let me tag @amadio on this one. Guilherme, what do you think about this? Would you know why -isystem is used to pick up the ROOT headers for this library?

amadio · September 19, 2018, 10:03am

Hi Attila, -isystem has been used mostly to suppress warnings from headers not owned by us (e.g. Vc headers), but I believe that we will need to change that if it’s a problem. I am working on an update to ROOT’s builtin VecCore to use the latest version, but I’m not sure that we can prevent CMake from finding installed things if it’s in the CMAKE_PREFIX_PATH. The new version shouldn’t need to use -isystem anymore, though, so this problem might just go away after the update.

Attila_Krasznahorkay · September 21, 2018, 2:38pm

Hi Guilherme,

Sorry for the delay on this…

We also use -isystem for all of our “externals” in the ATLAS code. But in this case ${CMAKE_BINARY_DIR}/include is used with -isystem. Is this because you put some “builtin” externals under there during your build? The clear recommendation would then be to put all your builtins into their own directory, include their headers from those specific directories with -isystem, and always include ${CMAKE_BINARY_DIR}/include with -I. At the very beginning of the “include list”.

Yeah, it took me a while to sort these things out for the ATLAS projects as well…

Cheers,
Attila

amadio · September 21, 2018, 2:54pm

Ah, this rings a bell. Maybe we need to rely on other means to suppress warnings coming from TBB and other builtins (one more problem of builtins…). I will submit a PR making the includes for TBB just regular includes. Thanks!

system · October 5, 2018, 2:54pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.