CMake build ROOT 6 on KNL supercomputer

Please provide the following information:


ROOT Version (e.g. 6.12/02): 6.10.02
Platform, compiler (e.g. CentOS 7.3, gcc6.2): Cray Programming Environment, with gcc 5.3.0


I am trying to build ROOT 6 on a supercomputer for running on Intel KNL nodes. The environment is a Cray setup, but in principle, the compiler is GCC 5.3

I’m using

$ cmake --version
cmake version 3.5.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).

I am trying to run:

$ cmake -D x11=NO -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -DBUILD_SHARED_LIBS=FALSE -DBUILD_STATIC_LIBS=TRUE -Wno-dev -DCMAKE_INSTALL_PREFIX=/projects/AtlasADSP/tools/root-6.10.02/cmakeinstall …/

I point cmake at the Cray compiler wrappers, but if I do the following you see the version:

$ cc --version
gcc (GCC) 5.3.0 20151204 (Cray Inc.)
Copyright © 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

you can see the proper GCC version is there.

I’m running into this error:

– Constructing LLVMBuild project information
– Targeting X86
CMake Error at interpreter/llvm/src/cmake/modules/AddLLVM.cmake:561 (install):
install TARGETS given no ARCHIVE DESTINATION for static library target
“LTO”.
Call Stack (most recent call first):
interpreter/llvm/src/tools/lto/CMakeLists.txt:18 (add_llvm_library)

Any suggestions would be appreciated. It appears that CMake doesn’t know where to install the static libraries it has built.

Thanks,
Taylor

Can you try without

-DBUILD_SHARED_LIBS=FALSE -DBUILD_STATIC_LIBS=TRUE 

I do not think we are ready yet for static builds.

Cheers,
Philippe.

Sorry I should have included that I began without those flags and got the same error.

cmake -D x11=NO -DCMAKE_C_COMPILER=cc -DCMAKE_CXX_COMPILER=CC -DCMAKE_Fortran_COMPILER=ftn -Wno-dev -DCMAKE_INSTALL_PREFIX=/projects/AtlasADSP/tools/root-6.10.02/cmakeinstall …/

I’ve attached the full CMake output.
cmake.txt (11.1 KB)

Could you do an out-of-source build. E.g.

mkdir src obj
git clone root.git src
cd obj
cmake … …/src

You won’t be able to ROOT run with static libs, so don’t go down that route.

The problem is with Cray’s wrappers which default to static builds, not with the configuration. Looking into some old build scripts (for flang development, not so much ROOT, but for both the problem is in the LLVM part), adding:

export CRAYPE_LINK_TYPE=dynamic
export XTPE_LINK_TYPE=dynamic

should be the ticket (haven’t tested with ROOT as it was for flang; and this has been a while).

Note that the wrappers exist to provide access to standard HPC tools such as mpi and the Cray networking libraries. The wrappers simply add the appropriate include and link directories based on the loaded PrgEnv and other modules. Since ROOT doesn’t use any of those (nor does LLVM), you may as well invoke gcc directly and be done with it.

Of course, if you do not use icc for KNL-specific optimizations, you can just download and use the canned Linux binaries in the first place.

@wlav So this got me past the CMake setup and half way through the make. I see this error however:

Scanning dependencies of target G__Core
[ 51%] Generating G__Core.cxx, …/…/lib/libCore.rootmap
/bin/sh: line 1: 189232 Illegal instruction (core dumped) …/rootcling_stage1/src/rootcling_stage1 -f G__Core.cxx -s /projects/AtlasADSP/tools/root-6.10.02/cmakebuild/lib/libCore.so -excludePath /projects/AtlasADSP/tools/root-6.10.02 -excludePath /projects/AtlasADSP/tools/root-6.10.02/cmakebuild -rml libCore.so -rmf /projects/AtlasADSP/tools/root-6.10.02/cmakebuild/lib/libCore.rootmap -DSYSTEM_TYPE_unix -I/projects/AtlasADSP/tools/root-6.10.02 -I/projects/AtlasADSP/tools/root-6.10.02/interpreter/cling/include -I/projects/AtlasADSP/tools/root-6.10.02/cmakebuild/include TVirtualPadPainter.h TSystemFile.h TMacro.h TROOT.h TObject.h TBuffer3DTypes.h TApplication.h TBase64.h TUrl.h TMathBase.h TProcessUUID.h TColorGradient.h TApplicationImp.h TSystem.h TVirtualQConnection.h TQObject.h TStopwatch.h TBuffer3D.h TObjString.h TVirtualMonitoring.h TInspectorImp.h TPoint.h TVirtualAuth.h TError.h TExec.h TCanvasImp.h TMD5.h TAttMarker.h TRegexp.h TAtt3D.h TContextMenuImp.h TQCommand.h TVirtualGL.h TString.h TEnv.h TUUID.h TTimeStamp.h TBrowser.h TTime.h TQConnection.h TFolder.h TStorage.h TRedirectOutputGuard.h TSystemDirectory.h TAttText.h TVirtualTableInterface.h TVirtualX.h TDatime.h TControlBarImp.h TPRegexp.h TNamed.h TAttLine.h TThreadSlots.h TStringLong.h TRefCnt.h TParameter.h TAttPad.h TTimer.h TVirtualPerfStats.h TVirtualPS.h TTask.h TColor.h TException.h TAttBBox.h TVersionCheck.h TFileInfo.h TObjectSpy.h TVirtualPad.h TBuffer.h TAttFill.h TMessageHandler.h TPluginManager.h TRef.h TVirtualViewer3D.h TUri.h TBrowserImp.h TGuiFactory.h TVirtualPadEditor.h TStyle.h TSysEvtHandler.h TQClass.h TVirtualFFT.h TDirectory.h TMemberInspector.h TBenchmark.h TRootIOCtor.h TProcessID.h TVirtualMutex.h TContextMenu.h TRemoteObject.h TInetAddress.h TAttAxis.h TAttBBox2D.h TFileCollection.h GuiTypes.h MessageTypes.h KeySymbols.h Buttons.h strlcpy.h snprintf.h TArrayL.h TObjArray.h THashList.h TCollectionProxyInfo.h TSortedList.h TRefArray.h TArray.h TRefTable.h TExMap.h THashTable.h TArrayS.h TCollection.h TObjectTable.h TArrayD.h TClonesArray.h TIterator.h TArrayL64.h TArrayI.h TArrayC.h TOrdCollection.h TVirtualCollectionProxy.h TList.h TArrayF.h TClassTable.h TBtree.h TSeqCollection.h TBits.h TMap.h TClassEdit.h ESTLType.h RStringView.h TUnixSystem.h root_std_complex.h TClingRuntime.h TListOfEnums.h TGlobal.h TIsAProxy.h TListOfFunctionTemplates.h TInterpreter.h TFunctionTemplate.h TStreamer.h TVirtualRefProxy.h TClassStreamer.h TListOfFunctions.h TToggle.h TMethodCall.h TClassGenerator.h TClassRef.h TMethod.h TListOfDataMembers.h TProtoClass.h TToggleGroup.h TStreamerElement.h TClass.h TMemberStreamer.h TSchemaHelper.h TEnumConstant.h TSchemaRuleSet.h TVirtualStreamerInfo.h TDataMember.h TListOfEnumsWithLock.h TDictionary.h TGenericClassInfo.h TVirtualIsAProxy.h TDictAttributeMap.h TDataType.h TSchemaRule.h TFileMergeInfo.h TClassMenuItem.h TEnum.h TFunction.h TRealData.h TMethodArg.h TBaseClass.h TInterpreterValue.h Getline.h /projects/AtlasADSP/tools/root-6.10.02/core/base/inc/LinkDef.h
core/base/CMakeFiles/G__Core.dir/build.make:413: recipe for target ‘core/base/G__Core.cxx’ failed
gmake[2]: *** [core/base/G__Core.cxx] Error 132
CMakeFiles/Makefile2:14554: recipe for target ‘core/base/CMakeFiles/G__Core.dir/all’ failed
gmake[1]: *** [core/base/CMakeFiles/G__Core.dir/all] Error 2
Makefile:149: recipe for target ‘all’ failed
gmake: *** [all] Error 2

“Illegal instruction”?! What hardware is the interactive node on which you’re compiling? More specifically, is it that much older that it is missing instructions such as AVX512 that exist on KNL but not on the interactive node’s CPU? (Even so, I’d not expect gcc 5.3 to produce those.)

Thing is, the creation of G__Core.cxx is done by rootcling, which is itself a product of the build process (and it uses the LLVM libraries for parsing). So at this point you do have a configuration problem as rootcling needs to run on the interactive node, not on the big iron. Then again, LLVM needs to run on both …

Any chance you can start over with VERBOSE=1 and show the full CLI arguments of the command that build rootcling (there are a handful, as it entails a couple of intermediate steps)?

Here is the Verbose output:

[ 51%] Generating G__Core.cxx, …/…/lib/libCore.rootmap
cd /projects/AtlasADSP/tools/root-6.10.02/cmakebuild/core/base && …/rootcling_stage1/src/rootcling_stage1 -f G__Core.cxx -s /projects/AtlasADSP/tools/root-6.10.02/cmakebuild/lib/libCore.so -excludePath /projects/AtlasADSP/tools/root-6.10.02 -excludePath /projects/AtlasADSP/tools/root-6.10.02/cmakebuild -rml libCore.so -rmf /projects/AtlasADSP/tools/root-6.10.02/cmakebuild/lib/libCore.rootmap -DSYSTEM_TYPE_unix -I/projects/AtlasADSP/tools/root-6.10.02 -I/projects/AtlasADSP/tools/root-6.10.02/interpreter/cling/include -I/projects/AtlasADSP/tools/root-6.10.02/cmakebuild/include TVirtualPadPainter.h TSystemFile.h TMacro.h TROOT.h TObject.h TBuffer3DTypes.h TApplication.h TBase64.h TUrl.h TMathBase.h TProcessUUID.h TColorGradient.h TApplicationImp.h TSystem.h TVirtualQConnection.h TQObject.h TStopwatch.h TBuffer3D.h TObjString.h TVirtualMonitoring.h TInspectorImp.h TPoint.h TVirtualAuth.h TError.h TExec.h TCanvasImp.h TMD5.h TAttMarker.h TRegexp.h TAtt3D.h TContextMenuImp.h TQCommand.h TVirtualGL.h TString.h TEnv.h TUUID.h TTimeStamp.h TBrowser.h TTime.h TQConnection.h TFolder.h TStorage.h TRedirectOutputGuard.h TSystemDirectory.h TAttText.h TVirtualTableInterface.h TVirtualX.h TDatime.h TControlBarImp.h TPRegexp.h TNamed.h TAttLine.h TThreadSlots.h TStringLong.h TRefCnt.h TParameter.h TAttPad.h TTimer.h TVirtualPerfStats.h TVirtualPS.h TTask.h TColor.h TException.h TAttBBox.h TVersionCheck.h TFileInfo.h TObjectSpy.h TVirtualPad.h TBuffer.h TAttFill.h TMessageHandler.h TPluginManager.h TRef.h TVirtualViewer3D.h TUri.h TBrowserImp.h TGuiFactory.h TVirtualPadEditor.h TStyle.h TSysEvtHandler.h TQClass.h TVirtualFFT.h TDirectory.h TMemberInspector.h TBenchmark.h TRootIOCtor.h TProcessID.h TVirtualMutex.h TContextMenu.h TRemoteObject.h TInetAddress.h TAttAxis.h TAttBBox2D.h TFileCollection.h GuiTypes.h MessageTypes.h KeySymbols.h Buttons.h strlcpy.h snprintf.h TArrayL.h TObjArray.h THashList.h TCollectionProxyInfo.h TSortedList.h TRefArray.h TArray.h TRefTable.h TExMap.h THashTable.h TArrayS.h TCollection.h TObjectTable.h TArrayD.h TClonesArray.h TIterator.h TArrayL64.h TArrayI.h TArrayC.h TOrdCollection.h TVirtualCollectionProxy.h TList.h TArrayF.h TClassTable.h TBtree.h TSeqCollection.h TBits.h TMap.h TClassEdit.h ESTLType.h RStringView.h TUnixSystem.h root_std_complex.h TClingRuntime.h TListOfEnums.h TGlobal.h TIsAProxy.h TListOfFunctionTemplates.h TInterpreter.h TFunctionTemplate.h TStreamer.h TVirtualRefProxy.h TClassStreamer.h TListOfFunctions.h TToggle.h TMethodCall.h TClassGenerator.h TClassRef.h TMethod.h TListOfDataMembers.h TProtoClass.h TToggleGroup.h TStreamerElement.h TClass.h TMemberStreamer.h TSchemaHelper.h TEnumConstant.h TSchemaRuleSet.h TVirtualStreamerInfo.h TDataMember.h TListOfEnumsWithLock.h TDictionary.h TGenericClassInfo.h TVirtualIsAProxy.h TDictAttributeMap.h TDataType.h TSchemaRule.h TFileMergeInfo.h TClassMenuItem.h TEnum.h TFunction.h TRealData.h TMethodArg.h TBaseClass.h TInterpreterValue.h Getline.h /projects/AtlasADSP/tools/root-6.10.02/core/base/inc/LinkDef.h
/bin/sh: line 1: 47917 Illegal instruction (core dumped) …/rootcling_stage1/src/rootcling_stage1 -f G__Core.cxx -s /projects/AtlasADSP/tools/root-6.10.02/cmakebuild/lib/libCore.so -excludePath /projects/AtlasADSP/tools/root-6.10.02 -excludePath /projects/AtlasADSP/tools/root-6.10.02/cmakebuild -rml libCore.so -rmf /projects/AtlasADSP/tools/root-6.10.02/cmakebuild/lib/libCore.rootmap -DSYSTEM_TYPE_unix -I/projects/AtlasADSP/tools/root-6.10.02 -I/projects/AtlasADSP/tools/root-6.10.02/interpreter/cling/include -I/projects/AtlasADSP/tools/root-6.10.02/cmakebuild/include TVirtualPadPainter.h TSystemFile.h TMacro.h TROOT.h TObject.h TBuffer3DTypes.h TApplication.h TBase64.h TUrl.h TMathBase.h TProcessUUID.h TColorGradient.h TApplicationImp.h TSystem.h TVirtualQConnection.h TQObject.h TStopwatch.h TBuffer3D.h TObjString.h TVirtualMonitoring.h TInspectorImp.h TPoint.h TVirtualAuth.h TError.h TExec.h TCanvasImp.h TMD5.h TAttMarker.h TRegexp.h TAtt3D.h TContextMenuImp.h TQCommand.h TVirtualGL.h TString.h TEnv.h TUUID.h TTimeStamp.h TBrowser.h TTime.h TQConnection.h TFolder.h TStorage.h TRedirectOutputGuard.h TSystemDirectory.h TAttText.h TVirtualTableInterface.h TVirtualX.h TDatime.h TControlBarImp.h TPRegexp.h TNamed.h TAttLine.h TThreadSlots.h TStringLong.h TRefCnt.h TParameter.h TAttPad.h TTimer.h TVirtualPerfStats.h TVirtualPS.h TTask.h TColor.h TException.h TAttBBox.h TVersionCheck.h TFileInfo.h TObjectSpy.h TVirtualPad.h TBuffer.h TAttFill.h TMessageHandler.h TPluginManager.h TRef.h TVirtualViewer3D.h TUri.h TBrowserImp.h TGuiFactory.h TVirtualPadEditor.h TStyle.h TSysEvtHandler.h TQClass.h TVirtualFFT.h TDirectory.h TMemberInspector.h TBenchmark.h TRootIOCtor.h TProcessID.h TVirtualMutex.h TContextMenu.h TRemoteObject.h TInetAddress.h TAttAxis.h TAttBBox2D.h TFileCollection.h GuiTypes.h MessageTypes.h KeySymbols.h Buttons.h strlcpy.h snprintf.h TArrayL.h TObjArray.h THashList.h TCollectionProxyInfo.h TSortedList.h TRefArray.h TArray.h TRefTable.h TExMap.h THashTable.h TArrayS.h TCollection.h TObjectTable.h TArrayD.h TClonesArray.h TIterator.h TArrayL64.h TArrayI.h TArrayC.h TOrdCollection.h TVirtualCollectionProxy.h TList.h TArrayF.h TClassTable.h TBtree.h TSeqCollection.h TBits.h TMap.h TClassEdit.h ESTLType.h RStringView.h TUnixSystem.h root_std_complex.h TClingRuntime.h TListOfEnums.h TGlobal.h TIsAProxy.h TListOfFunctionTemplates.h TInterpreter.h TFunctionTemplate.h TStreamer.h TVirtualRefProxy.h TClassStreamer.h TListOfFunctions.h TToggle.h TMethodCall.h TClassGenerator.h TClassRef.h TMethod.h TListOfDataMembers.h TProtoClass.h TToggleGroup.h TStreamerElement.h TClass.h TMemberStreamer.h TSchemaHelper.h TEnumConstant.h TSchemaRuleSet.h TVirtualStreamerInfo.h TDataMember.h TListOfEnumsWithLock.h TDictionary.h TGenericClassInfo.h TVirtualIsAProxy.h TDictAttributeMap.h TDataType.h TSchemaRule.h TFileMergeInfo.h TClassMenuItem.h TEnum.h TFunction.h TRealData.h TMethodArg.h TBaseClass.h TInterpreterValue.h Getline.h /projects/AtlasADSP/tools/root-6.10.02/core/base/inc/LinkDef.h
core/base/CMakeFiles/G__Core.dir/build.make:413: recipe for target ‘core/base/G__Core.cxx’ failed
gmake[2]: *** [core/base/G__Core.cxx] Error 132
gmake[2]: Leaving directory ‘/lus/theta-fs0/projects/AtlasADSP/tools/root-6.10.02/cmakebuild’
CMakeFiles/Makefile2:14554: recipe for target ‘core/base/CMakeFiles/G__Core.dir/all’ failed
gmake[1]: *** [core/base/CMakeFiles/G__Core.dir/all] Error 2
gmake[1]: Leaving directory ‘/lus/theta-fs0/projects/AtlasADSP/tools/root-6.10.02/cmakebuild’
Makefile:149: recipe for target ‘all’ failed
gmake: *** [all] Error 2

No, the verbose output of creating rootcling (stage1), not that of the creation of G__Core.cxx. That will show what the target architecture for rootcling was.

Thanks, I’ll try rerunning the build with Verbose and post the output.

OK, here is the full VERBOSE output attached.cmake_make_verbose.txt (2.4 MB)

So any platform-specific parameters are still hidden in ‘CC’, as there’s nothing CPU-specific (beyond the innocent -m64) on the CLI.

What is the value of the CRAY_CPU_TARGET envar? If it’s ‘mic-knl’, can you load module ‘craype-haswell’ (or whatever is the cpu of your interactive nodes)?

I can, but then you lose any possible optimization for the worker nodes. I might as well just compile with the g++ installed on the login nodes in that case as you suggested initially.

What optimizations are you expecting that are worthwhile? For that matter, will that ancient gcc5.3 deliver them?

Teasing out rootcling (and rmkdepend) from the build process and building it for the interactive node with the rest for the batch nodes, is going to be a lot of work b/c it needs LLVM, as does the rest of ROOT during run-time.

You could compile the whole of ROOT twice: once on a KNL node, once on an interactive node, no cross-compilation. Then separate out the batch and interactive environments for the end-user.

But a more straightforward option would be to add something like ‘-march=core-avx2’ (assuming Haswell for the interactive nodes) to the build/link flags for rootcling and for LLVM. That way, most of ROOT is still cross-compiled and only dictionaries and wrapper generation will be build compatible for both platforms. Those parts would be potentially slower, but I doubt you’ll notice it for typical HEP workloads, since these are mostly called during initialization (i.e. one-offs) and I/O (i.e. already slowed down).

Hi @wlav,
OK, so after failing to simply use the gcc/g++/gfortran I installed the binaries. The CentOS and redhat versions don’t work. The ubuntu binaries work (Theta uses Suse). However, when I run ‘root’ I see this complaint:

ERROR in cling::CIFactory::createCI(): cannot extract standard library include paths!
Invoking:
LC_ALL=C g+±5 -O2 -DNDEBUG -xc++ -E -v /dev/null 2>&1 >/dev/null | awk ‘/^#include </,/^End of search/{if (!/^#include </ && !/^End of search/){ print }}’ | GREP_OPTIONS= grep -E “(c|g)++”
Results was:
With exit code 256

However my codes still work. Any way to get rid of this error?

Thanks,
Taylor

No idea; that’s really one for the ROOT folks. :confused:

Pinging @vvassilev back into the thread.

What’s the output of

LC_ALL=C g+±5 -O2 -DNDEBUG -xc++ -E -v /dev/null 2>&1 >/dev/null | awk ‘/^#include </,/^End of search/{if (!/^#include </ && !/^End of search/){ print }}’ | GREP_OPTIONS= grep -E “(c|g)++”

LC_ALL=C g++ -O2 -DNDEBUG -xc++ -E -v /dev/null 2>&1 >/dev/null | awk ‘/^#include </,/^End of search/{if (!/^#include </ && !/^End of search/){ print }}’ | GREP_OPTIONS= grep -E “(c|g)++”
/opt/intel/compilers_and_libraries_2018.0.128/linux/ipp/include
/opt/intel/compilers_and_libraries_2018.0.128/linux/mkl/include
/opt/intel/compilers_and_libraries_2018.0.128/linux/tbb/include
/opt/intel/compilers_and_libraries_2018.0.128/linux/daal/include
/usr/include/c++/4.8
/usr/include/c++/4.8/x86_64-suse-linux
/usr/include/c++/4.8/backward
/usr/lib64/gcc/x86_64-suse-linux/4.8/include
/usr/local/include
/usr/lib64/gcc/x86_64-suse-linux/4.8/include-fixed
/usr/lib64/gcc/x86_64-suse-linux/4.8/…/…/…/…/x86_64-suse-linux/include
/usr/include

Did that help identify the source of the error message?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.