Error on TBufferFile::ReadVersion

weishi · November 13, 2022, 5:32pm

Dear experts,

I have a successfully built root v5-34-00-patches on an IBM POWER9 machine. After sourcing root, I can read a template root file and produce what I want. The small piece of code is here:
test.C (947 Bytes)

and the input template root file that contains two empty 2D histograms here:
OscillogramTemplate.root (25.5 KB)

Output of the code:

[wshi@login2.summit ~]$ echo $ROOTSYS
/autofs/nccs-svm1_home1/wshi/ProdROOT/v5-34-00-patches

[wshi@login2.summit ~]$ root -l -b -q test.C 
root [0] 
Processing test.C...
Read Oscillator Template file (InputFileName): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
Read Oscillator Template file (InputFileName.c_str()): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
nCoarseCosz: 52

However, when the above is implemented in our analysis framework (using the same ROOT version), in terms of .cpp and .h, it seems to have some trouble reading the root file and I got a lot of TStreamerInfo error. The full log is here:
analysis_log.txt (26.0 KB)

Fyi, the implementation of this in .cpp (starting at line 63 of .cpp file) and .h as well as makefile are here:
OscClass_CUDAProb3.h (6.7 KB)
OscClass_CUDAProb3.cpp (58.4 KB)
Makefile.txt (2.0 KB)

I tried gdb, I can see the first error happen when step through TObject.h
void *operator new(size_t sz) { return TStorage::ObjectAlloc(sz); } for this line

TFile* InFile = new TFile(InputFileName.c_str(), "READ");

But I’m not gaining a lot of useful information from it without the help of experts. Full gdb log is here, fyi:
gdb.txt (22.4 KB)

Any tips/hints would be appreciated.

Wei

_ROOT Version: v5-34-00-patches
_Platform:

NAME="Red Hat Enterprise Linux"
VERSION="8.2 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.2"

_Compiler:gcc 8.3.1

Wile_E_Coyote · November 13, 2022, 7:16pm

Your makefile sets “-std=c++11”.
Have you built ROOT using “./configure --enable-cxx11” (or “cmake -Dcxx11=ON”)?

Also, I think you’d better have: CXXFLAGS+=$(ROOTCFLAGS)

weishi · November 14, 2022, 5:38am

I have rebuilt now

[wshi@login4.summit MaCh3]$ root-config --features
asimage astiff builtin_afterimage builtin_lz4 cxx11 explicitlink genvector krb5 memstat minuit2 python reflex roofit shadowpw shared soversion ssl tmva vdt x11 xft xml thread

but the same error is still there.

Wile_E_Coyote · November 14, 2022, 8:18am

Are you using “nvcc” to build it?
If yes, try to work without CUDA.

Check the “nvcc --version” as maybe it internally uses a “g++” version, which is different from the system’s one.

You should also check what this returns (we are looking for the “__BYTE_ORDER__”):
: | nvcc -dM -E -x c++ - | grep -i endian

Try to compile this with “nvcc” (it also prints the actually used “g++” version):

// Usage: nvcc -std=c++11 main.cxx; ./a.out
#include <iostream>
int main() {
   std::cout << "__BYTE_ORDER__ = " << (__BYTE_ORDER__) << "\n";
   std::cout << "__ORDER_LITTLE_ENDIAN__ = " << (__ORDER_LITTLE_ENDIAN__) << "\n";
   std::cout << "__ORDER_BIG_ENDIAN__ = " << (__ORDER_BIG_ENDIAN__) << "\n";
   std::cout << "__VERSION__ = " << (__VERSION__) << "\n";
   std::cout << "__cplusplus = " << (__cplusplus) << "\n";
   return 0;
}

BTW. In your makefile, I would set “CXX = $(shell root-config --cxx)” and then use “$(CXX)” instead of “g++”. Also, in the places where it is used, I would temporarily add a line with “$(CXX) --version” (to make sure that it is exactly the same compiler that was used by ROOT).

weishi · November 14, 2022, 3:01pm

We do load cuda to compile a sepeatate CUDA package:

[wshi@login3.summit MaCh3]$ module load cuda/11.5.2
[wshi@login3.summit MaCh3]$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:46:07_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

Here is the endianess:

[wshi@login3.summit MaCh3]$ : | nvcc -dM -E -x c++ - | grep -i endian
nvcc fatal   : Unknown option '-dM'
[wshi@login3.summit ~]$ : | nvcc --compiler-options -dM -E -x c++ - | grep -i endian
#define __ORDER_LITTLE_ENDIAN__ 1234
#define _LITTLE_ENDIAN 1
#define __FLOAT_WORD_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __ORDER_PDP_ENDIAN__ 3412
#define __LITTLE_ENDIAN__ 1
#define __ORDER_BIG_ENDIAN__ 4321
#define __VEC_ELEMENT_REG_ORDER__ __ORDER_LITTLE_ENDIAN__
#define __BYTE_ORDER__ __ORDER_LITTLE_ENDIAN__

And the nvcc g++ version (the system gcc version is 8.3.1)

[wshi@login3.summit ~]$ nvcc -std=c++11 main.cxx; ./a.out
__BYTE_ORDER__ = 1234
__ORDER_LITTLE_ENDIAN__ = 1234
__ORDER_BIG_ENDIAN__ = 4321
__VERSION__ = 8.3.1 20191121 (Red Hat 8.3.1-5)
__cplusplus = 201103

Wile_E_Coyote · November 14, 2022, 3:08pm

So, it looks like “nvcc” calls the right “g++”.

Maybe attach the “inputs/skatm/OscillogramTemplate.root” file for inspection.

I guess we need @pcanal for a closer look.

weishi · November 14, 2022, 3:09pm

Here:
OscillogramTemplate.root (25.5 KB)

Wile_E_Coyote · November 14, 2022, 3:24pm

In your “test.C”, change “void test()” into “int main()” and then make two trials:

rm -f a.out
`root-config --cxx --cflags` test.C `root-config --libs`
./a.out
rm -f a.out
nvcc `root-config --cflags` test.C `root-config --libs`
./a.out

If both work, in the “nvcc” line, remove the ROOT “cflags” and try to add the same flags as used in your makefile.

Wile_E_Coyote · November 14, 2022, 3:48pm

Actually, I can’t find the real “main” routine of your “OscClass” executable.
Right in its beginning, try to add one line:
TApplication a("a", 0, 0); // just to make sure that the autoloading of ROOT libraries works

weishi · November 14, 2022, 5:02pm

So the first line works, the 2nd doesn’t:

[wshi@login1.summit ~]$ `root-config --cxx --cflags` test.C `root-config --libs`
[wshi@login1.summit ~]$ ./a.out
Read Oscillator Template file (InputFileName): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
Read Oscillator Template file (InputFileName.c_str()): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
nCoarseCosz: 52
[wshi@login1.summit ~]$ rm -f a.out

[wshi@login1.summit ~]$ nvcc `root-config --cflags` test.C `root-config --libs`
nvcc fatal   : Unknown option '-pthread'
[wshi@login1.summit ~]$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:46:07_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

Wile_E_Coyote · November 14, 2022, 5:16pm

So, from the “root-config --cflags” output, remove “-pthread”.

weishi · November 14, 2022, 5:29pm

Ok, that didn’t work:

[wshi@login1.summit ~]$ root-config --cflags
-pthread -std=c++11 -Wno-deprecated-declarations -m64 -fsigned-char -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include
[wshi@login1.summit ~]$ root-config --libs
-L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -pthread -lm -ldl -rdynamic
[wshi@login1.summit ~]$ nvcc `root-config --cflags` test.C `root-config --libs`
nvcc fatal   : Unknown option '-pthread'
[wshi@login1.summit ~]$ nvcc -std=c++11 -Wno-deprecated-declarations -m64 -fsigned-char -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include test.C -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -pthread -lm -ldl -rdynamic
nvcc fatal   : Unknown option '-fsigned-char'
[wshi@login1.summit ~]$ nvcc -std=c++11 -Wno-deprecated-declarations -m64 -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include test.C -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -pthread -lm -ldl -rdynamic
nvcc fatal   : Unknown option '-pthread'
[wshi@login1.summit ~]$ nvcc -std=c++11 -Wno-deprecated-declarations -m64 -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include test.C -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl -rdynamic
nvcc fatal   : Unknown option '-rdynamic'
[wshi@login1.summit ~]$ nvcc -std=c++11 -Wno-deprecated-declarations -m64 -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include test.C -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl          
nvcc fatal   : Don't know what to do with 'test.C'

Wile_E_Coyote · November 14, 2022, 5:34pm

Maybe it doesn’t know “.C” files. Try to rename it into “test.cxx”.

weishi · November 14, 2022, 5:44pm

That worked

[wshi@login1.summit ~]$ nvcc -std=c++11 -Wno-deprecated-declarations -m64 -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include test.cxx -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl 
[wshi@login1.summit ~]$ ./a.out 
Read Oscillator Template file (InputFileName): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
Read Oscillator Template file (InputFileName.c_str()): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
nCoarseCosz: 52

Then I tried to remove the root cflags and use the makefile cxx flag: CXXFLAGS = -I. -Wall -fPIC -O3 -g -std=c++11 -fopenmp, it complains no TH2D:

[wshi@login1.summit ~]$ nvcc -I. -Wall -fPIC -O3 -g -std=c++11 -fopenmp test.cxx -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl 
nvcc fatal   : Unknown option '-Wall'
[wshi@login1.summit ~]$ nvcc -I. -fPIC -O3 -g -std=c++11 -fopenmp test.cxx -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl 
nvcc fatal   : Unknown option '-fPIC'
[wshi@login1.summit ~]$ nvcc -I. -O3 -g -std=c++11 -fopenmp test.cxx -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl 
nvcc fatal   : Unknown option '-fopenmp'
[wshi@login1.summit ~]$ nvcc -I. -O3 -g -std=c++11  test.cxx -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl 
test.cxx:1:10: fatal error: TH2D.h: No such file or directory
 #include "TH2D.h"
          ^~~~~~~~
compilation terminated.

Wile_E_Coyote · November 14, 2022, 5:45pm

Add, e.g.: -I${ROOTSYS}/include

weishi · November 14, 2022, 5:48pm

Ok, this worked

[wshi@login1.summit ~]$ nvcc -I. -O3 -g -std=c++11 -I${ROOTSYS}/include  test.cxx -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl 
[wshi@login1.summit ~]$ ./a.out 
Read Oscillator Template file (InputFileName): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
Read Oscillator Template file (InputFileName.c_str()): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
nCoarseCosz: 52

So is this saying I should either avoid loading cuda, or change the make file to use the above rule to compile with nvcc?

Wile_E_Coyote · November 14, 2022, 6:05pm

Well, in the last trial, try to add “-x cu” (which exists in your makefile and it enforces the file type).

Somehow, you need to “reproduce” the error.
The first trial would be to modify the makefile so that it always uses “${CXX}”, also when CUDA is loaded.
If it works, you can try to play with the “nvcc” command line there.

weishi · November 14, 2022, 6:10pm

This also worked

[wshi@login1.summit ~]$ nvcc -I. -O3 -x cu -g -std=c++11 -I${ROOTSYS}/include  test.cxx -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -lm -ldl 
[wshi@login1.summit ~]$ ./a.out 
Read Oscillator Template file (InputFileName): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
Read Oscillator Template file (InputFileName.c_str()): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
nCoarseCosz: 52

weishi · November 14, 2022, 6:55pm

The compile part of this executable is here:
Osc.txt (3.6 KB)

I am using the exact same rule to compile test.cxx

nvcc -g -O3 -x cu  -lineinfo -std=c++11 -Xcompiler="-fopenmp -Wall -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -pthread -lm -ldl -rdynamic -lRooFit -lRooFitCore -lMinuit -lHistPainter -I. -Wall -fPIC -O3 -g -std=c++11 -fopenmp -DUSE_GPU" test.cxx

And it worked fine

[wshi@login1.summit ~]$ nvcc -g -O3 -x cu  -lineinfo -std=c++11 -Xcompiler="-fopenmp -Wall -I/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/include -L/autofs/nccs-svm1_home1/wshi/ROOT/v5-34-00-patches/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -pthread -lm -ldl -rdynamic -lRooFit -lRooFitCore -lMinuit -lHistPainter -I. -Wall -fPIC -O3 -g -std=c++11 -fopenmp -DUSE_GPU" test.cxx
[wshi@login1.summit ~]$ ./a.out 

RooFit v3.60 -- Developed by Wouter Verkerke and David Kirkby 
                Copyright (C) 2000-2013 NIKHEF, University of California & Stanford University
                All rights reserved, please read http://roofit.sourceforge.net/license.txt

Read Oscillator Template file (InputFileName): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
Read Oscillator Template file (InputFileName.c_str()): /autofs/nccs-svm1_home1/wshi/MaCh3/MaCh3/inputs/skatm/OscillogramTemplate.root
nCoarseCosz: 52

Wile_E_Coyote · November 14, 2022, 9:15pm

I think this is “mandatory”: