Change ROOT headers path at runtime

ROOT
1

ROOT Version: 6.14/04
Platform: SLC6
Compiler: gcc 4.9.3

Dear ROOTers,
my analysis groups has finally decided to upgrade the common ROOT version to 6.14 (from 5.34) and I took the task of compiling and providing the new version for everyone.
(long story short: the two main computing clusters we use still run SLC6, so we can’t use the one on cvmfs)
That being said, I donwloaded the source code, compiled it, and then we synced the installation directory from the local afs area to our shared cvmfs path, where we keep all the SW relevant for the analysis.

When I try to run this version of root it can take up to 10s on lxplus to start, or even several tens of minutes from our computing center in Italy.
A quick look with strace reveals the following:

strace -f root -l
[...]
[pid 25679] stat("/include", 0x7fff3137feb0) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
[pid 25679] stat("/include", 0x7fff3137fe30) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/usr/include", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
[pid 25679] brk(0x148f000)              = 0x148f000
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/include", {st_mode=S_IFDIR|0755, st_size=1636, ...}) = 0
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/include/clang", 0x7fff313802a0) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/include/../tools/clang/include", 0x7fff313802a0) = -1 ENOENT (No such file or directory)
[pid 25679] open("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/allDict.cxx.pch", O_RDONLY|O_CLOEXEC) = 3
[pid 25679] readlink("/proc/self/fd/3", "/cvmfs/ams.cern.ch/Offline/dbar/"..., 4096) = 81
[pid 25679] fstat(3, {st_mode=S_IFREG|0644, st_size=101366032, ...}) = 0
[pid 25679] mmap(NULL, 101366032, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f43a9b9e000
[pid 25679] brk(0x14b0000)              = 0x14b0000
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include", {st_mode=S_IFDIR|0755, st_size=73728, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/Rpair.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include", {st_mode=S_IFDIR|0755, st_size=1636, ...}) = 0
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/Rpair.h", {st_mode=S_IFREG|0644, st_size=690, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/input_line_14", 0x7fff31380270) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/..", {st_mode=S_IFDIR|0755, st_size=20, ...}) = 0
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../input_line_14", 0x7fff31380270) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/input_line_14", 0x7fff313802f0) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/input_line_13", 0x7fff31380270) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../input_line_13", 0x7fff31380270) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/input_line_13", 0x7fff313802f0) = -1 ENOENT (No such file or directory)
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/ROOT", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/ROOT/StringConv.hxx", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/ROOT", {st_mode=S_IFDIR|0755, st_size=53, ...}) = 0
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/ROOT/StringConv.hxx", {st_mode=S_IFREG|0644, st_size=5658, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory", {st_mode=S_IFDIR|0755, st_size=2048, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/RooBarlowBeestonLL.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory", {st_mode=S_IFDIR|0755, st_size=24, ...}) = 0
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/RooBarlowBeestonLL.h", {st_mode=S_IFREG|0644, st_size=4671, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/PiecewiseInterpolation.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/PiecewiseInterpolation.h", {st_mode=S_IFREG|0644, st_size=4163, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/MakeModelAndMeasurementsFast.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/MakeModelAndMeasurementsFast.h", {st_mode=S_IFREG|0644, st_size=1313, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/LinInterpVar.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/LinInterpVar.h", {st_mode=S_IFREG|0644, st_size=1621, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/HistoToWorkspaceFactoryFast.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/HistoToWorkspaceFactoryFast.h", {st_mode=S_IFREG|0644, st_size=5227, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/HistoToWorkspaceFactory.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/HistoToWorkspaceFactory.h", {st_mode=S_IFREG|0644, st_size=4531, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/HistFactorySimultaneous.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/HistFactorySimultaneous.h", {st_mode=S_IFREG|0644, st_size=2844, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/HistFactoryNavigation.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/HistFactoryNavigation.h", {st_mode=S_IFREG|0644, st_size=7643, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/ParamHistFunc.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/ParamHistFunc.h", {st_mode=S_IFREG|0644, st_size=4316, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/HistFactoryModelUtils.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/HistFactoryModelUtils.h", {st_mode=S_IFREG|0644, st_size=995, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/HistFactoryException.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/HistFactoryException.h", {st_mode=S_IFREG|0644, st_size=941, ...}) = 0
[pid 25679] stat("/afs/cern.ch/work/d/dbar/devel/root-6.14.04-build/include/RooStats/HistFactory/FlexibleInterpVar.h", 0x7fff31380270) = -1 EACCES (Permission denied)
[pid 25679] stat("/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/etc/../include/RooStats/HistFactory/FlexibleInterpVar.h", {st_mode=S_IFREG|0644, st_size=2901, ...}) = 0

and so on…
The slowdown is caused by the numerous stat calls on header files on the original afs path, and these appear even if I set the variable

$ export ROOT_INCLUDE_PATH=$ROOTSYS/include
$ echo $ROOT_INCLUDE_PATH
/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04/include

Is there a quick way to load headers only from cvmfs and not from the original installation path on afs?
Or, alternatively, what would be the correct way to ship a compiled version of root on our common cvmfs area? (keeping in mind that I don’t have access/permission for the experiment cvmfs server)

Cheers,
Valerio

2

Hi Valerio,

I’m not quite sure why headers are still looked up from the original installation but perhaps @amadio has an idea? In any case, the safest way to get rid of AFS dependencies is making sure that there is no AFS on the build server and to re-publish to cvmfs.

You can also try if the precompiled ROOT version from sft.cern.ch can do the job for you:

. /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_latest x86_64-slc6-gcc8-opt
root -l

Cheers,
Jakob

3

Hi Jakob

That would be tricky, since I’m compiling it on lxplus, but in principle it is doable (I could just build it in a docker container) but then won’t it try to locate the headers in the original installation path, on the build machine?

This actually looks promising, but not being familiar with the dynamics of LCG I have a few doubts/questions.

  1. This actually selects ROOT 6.14/00, I was preferentially lokking at 6.14/04 since it reverted the default compression algorithm for I/O and this should give us retrocompability for reading/writing files w.r.t. the old analysis.
  2. I suppose LCG_latest will get updated from time to time, but we like to keep our environment as constant as possible. Is there some way for me to “force” the ROOT version to a particular one (e.g. 6.14/04) ?
4

That would be tricky, since I’m compiling it on lxplus, but in principle it is doable (I could just build it in a docker container) but then won’t it try to locate the headers in the original installation path, on the build machine?

While you might still have the unnecessary stat() calls, they’d target a local file system and thus be an order of magnitude faster than the ones pointing to AFS.

For the LCG releases in general, @ganis might be able to provide more details. You can certainly pin a certain release version; all available version are shown when you run

. /cvmfs/sft.cern.ch/lcg/views/setupViews.sh

Version LCG_94 contains the latest 6.14/04

Cheers,
Jakob

5

quick update: it looks like ROOT from the LCG_latest is compiled with -std=c++1z while our experiment software forces -std=c++11 on compilation and this breaks the build, argh!

edit: also LGC_94 fails, unfortunately I don’t have the authority to change the build of our experiment software (yet) I need to find a workaround if I want to use LCG distributions…
Otherwise I would like to pursue the “remove the wring header lookup” option as well…

6

Dear Valerio,

For what relates to the LCG releases, could you please expand on

?

This said, we are currently providing LCG_94 only for gcc62, gcc7 and gcc8; is there a specific why you need to stay on gcc49?

For what relates your own builds and AFS references, an option is to build under /tmp on lxplus: as Jakob said, remaining stat calls to what will became ‘local’ /tmp will be negligibly fast when starting root.

G Ganis

7

Dear G

long story incredibly short:

$ root-config --cflags
-pthread -std=c++1z -m64 -I/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include

while in our experiment software everything is compiled with
std=c++11

This leads to a boatload of errors, such as

In file included from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:28,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TNamed.h:26,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TKey.h:15,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TBasket.h:28,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/ROOT/TIOFeatures.hxx:14,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TTree.h:29,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TChain.h:24,
                 from ../include/amschain.h:8,
                 from ../CC/amschain.C:2:
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/ROOT/RStringView.hxx:67:17: error: expected type-specifier
        operator std::string_view() const { return std::string_view(fData,fLength); }
                 ^~~
In file included from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TNamed.h:26,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TKey.h:15,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TBasket.h:28,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/ROOT/TIOFeatures.hxx:14,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TTree.h:29,
                 from /cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TChain.h:24,
                 from ../include/amschain.h:8,
                 from ../CC/amschain.C:2:
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:110:13: error: expected type-specifier
    operator std::string_view() const { return std::string_view(Data(),fExtent); }
             ^~~
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:275:23: error: ‘string_view’ in namespace ‘std’ does not name a type
    TString(const std::string_view &sub);
                       ^~~~~~~~~~~
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:275:18: note: ‘std::string_view’ is only available from C++17 onwards
    TString(const std::string_view &sub);
                  ^~~
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:312:37: error: ‘string_view’ in namespace ‘std’ does not name a type
    TString    &operator=(const std::string_view &s);
                                     ^~~~~~~~~~~
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:312:32: note: ‘std::string_view’ is only available from C++17 onwards
    TString    &operator=(const std::string_view &s);
                                ^~~
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:439:9: error: ‘string_view’ in namespace ‘std’ does not name a type
    std::string_view View() const { return std::string_view(GetPointer(),Length()); }
         ^~~~~~~~~~~
/cvmfs/sft.cern.ch/lcg/releases/ROOT/6.14.04-0d8dc/x86_64-slc6-gcc8-opt/include/TString.h:439:4: note: ‘std::string_view’ is only available from C++17 onwards
    std::string_view View() const { return std::string_view(GetPointer(),Length()); }
    ^~~

As I said before, I don’t have the authority to make big changes to the build of our software so this looks like a dead end at the moment (maybe in the future life will be easier)

This looks way easier! I will try it immediately.

Thanks

8

As it has been explained before, I believe that LCG has support for SLC6, so in principle you are covered, but ROOT is compiled with C++17 support. If your experiment is not ready for it, you may have problems. LCG_94 has ROOT 6.14/04.

From the discussion in this topic, there is definitely something wrong with how you configured and installed ROOT. I will give you a recipe that should work for your case if you want to compile ROOT yourself and install into CVMFS. If you want to avoid problems with references to the build directory, you have to perform a proper installation of ROOT. Also, to avoid problems with references to AFS, you will need to configure with the final installation prefix. You’ll then have to perform a so called staged installation into some directory that you can use to create a tarball and unpack into the right place, or simply copy over.

$ cmake $SRCDIR/root ... -DCMAKE_INSTALL_PREFIX=/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04
$ cmake --build .
$ env DESTDIR=$PWD/install cmake --build . --target install

Once the last command finishes, you’ll have the directory $PWD/install/cvmfs/ams.cern.ch/Offline/dbar/public/installed/root-6.14.04, which you can then tar up and/or copy into the final destination. Let me know if you still have any problems after this. One important thing is to make sure that dependencies will be available (i.e., won’t be picked from the base system, which may change). Try to pick as much as possible from your own CVMFS stack. Cheers,

9

Yep, that’s exactly my problem right now… I wish I could enforce this but I gotta work with what I have

I guess I missed this procedure, at the end of the day I went with @ganis solution and that seems to work, although not optimal.

I’m going to write this down for the next time :wink:

Guess it can be marked as closed

Thanks again,
Valerio

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.