Issue building ROOT with cudnn

Popinaodude · July 21, 2020, 7:08pm

ROOT Version: 6.22/00
Platform: Ubuntu 20.04
Compiler: gcc 9.3.0

Hi all,

I’m spent all day trying to compile ROOT on my personal PC, and I’m getting the same error very close to completion. The problem has something to do with TMVA and cudnn. I’m using Ubuntu LTS 20.04 with a GeForce GTX 1660 Ti GPU. Here are my steps:

I installed the latest CUDA toolkit (11) as per these instructions: docs(DOT)nvidia(DOT)com/cuda/cuda-installation-guide-linux/index.html. Then I installed cudnn libraries as per these instructions: docs(DOT)nvidia(DOT)com/deeplearning/sdk/cudnn-install/index.html. (sorry these aren’t links, as a new user I can’t post more than two links)
I then pulled the root git repositories in a directory ~/Programs/root_src, made a build directory: ~/Programs/build and an install directory: ~/Programs/ROOT
I want to be able to use pyROOT so I made a new python environment using anaconda and called it ROOT
I set up cmake in ~/Programs/build with the command
$ cmake -DCMAKE_INSTALL_PREFIX="../ROOT" -DPYTHON_EXECUTABLE="/home/keanu/anaconda/envs/ROOT/bin/python" -DPYTHON_INCLUDE_DIR="/home/keanu/anaconda/envs/ROOT/include/python3.8/Python.h" -DPYTHON_LIBRARY="/home/keanu/anaconda/envs/ROOT/lib/python3.8/" -DCMAKE_CUDA_HOST_COMPILER="/usr/bin/gcc" -Dcuda=ON -Dcudnn=ON -Dtmva-gpu=ON ../root_src/

I’ve uploaded the output to that command here: cmake_output_1.txt (12.5 KB)
Overall it looks fine but I’m a little worried about the lines:

-- Found CUDNN: /usr/local/cuda-11.0/include  
-- Found cuDNN: v?  (include: /usr/local/cuda-11.0/include, library: /usr/local/cuda-11.0/lib64/libcudnn.so)

I then build using the command $ cmake --build . --target install -- -j10
I’ve uploaded the output to that command here: cmake_output_2.txt (835.7 KB)

I’ve pasted the actual error below for convenience as the above file is quite long:

/home/keanu/Programs/root_src/core/base/inc/TStorage.h:131:37: warning: declaration of ‘static void TStorage::UpdateIsOnHeap(const volatile UInt_t&, volatile UInt_t&)’ with attribute ‘noinline’ follows inline declaration [-Wattributes]
  131 | R__NEVER_INLINE void TStorage::UpdateIsOnHeap(volatile const UInt_t &uniqueID, volatile UInt_t &bits) {
      |                                     ^~~~~~~~
/home/keanu/Programs/root_src/core/base/inc/TStorage.h:91:20: note: previous declaration of ‘static void TStorage::UpdateIsOnHeap(const volatile UInt_t&, volatile UInt_t&)’ here
   91 |    static void UpdateIsOnHeap(volatile const UInt_t &uniqueID, volatile UInt_t &bits);
      |                    ^~~~~~~~~~~~~~
/home/keanu/Programs/root_src/core/base/inc/TStorage.h:131:37: warning: declaration of ‘static void TStorage::UpdateIsOnHeap(const volatile UInt_t&, volatile UInt_t&)’ with attribute ‘noinline’ follows inline declaration [-Wattributes]
  131 | R__NEVER_INLINE void TStorage::UpdateIsOnHeap(volatile const UInt_t &uniqueID, volatile UInt_t &bits) {
      |                                     ^~~~~~~~
/home/keanu/Programs/root_src/core/base/inc/TStorage.h:91:20: note: previous declaration of ‘static void TStorage::UpdateIsOnHeap(const volatile UInt_t&, volatile UInt_t&)’ here
   91 |    static void UpdateIsOnHeap(volatile const UInt_t &uniqueID, volatile UInt_t &bits);
      |                    ^~~~~~~~~~~~~~
/home/keanu/Programs/root_src/core/base/inc/TStorage.h:131:37: warning: declaration of ‘static void TStorage::UpdateIsOnHeap(const volatile UInt_t&, volatile UInt_t&)’ with attribute ‘noinline’ follows inline declaration [-Wattributes]
  131 | R__NEVER_INLINE void TStorage::UpdateIsOnHeap(volatile const UInt_t &uniqueID, volatile UInt_t &bits) {
      |                                     ^~~~~~~~
/home/keanu/Programs/root_src/core/base/inc/TStorage.h:91:20: note: previous declaration of ‘static void TStorage::UpdateIsOnHeap(const volatile UInt_t&, volatile UInt_t&)’ here
   91 |    static void UpdateIsOnHeap(volatile const UInt_t &uniqueID, volatile UInt_t &bits);
      |                    ^~~~~~~~~~~~~~
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/RecurrentPropagation.cu(135): error: identifier "cudnnSetRNNDescriptor" is undefined
          detected during:
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRecurrentDescriptors(TMVA::DNN::TDescriptors *&, RNNLayer *) [with AFloat=Float_t, RNNLayer=TMVA::DNN::RNN::TBasicRNNLayer<TMVA::DNN::TCudnn<Float_t>>]" 
/home/keanu/Programs/root_src/tmva/tmva/inc/TMVA/DNN/Architectures/TCudnn.h(178): here
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRNNDescriptors(TMVA::DNN::TDescriptors *&, TMVA::DNN::TCudnn<AFloat>::RNNLayer_t *) [with AFloat=Float_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(40): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/RecurrentPropagation.cu(135): error: identifier "cudnnSetRNNDescriptor" is undefined
          detected during:
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRecurrentDescriptors(TMVA::DNN::TDescriptors *&, RNNLayer *) [with AFloat=Float_t, RNNLayer=TMVA::DNN::RNN::TBasicLSTMLayer<TMVA::DNN::TCudnn<Float_t>>]" 
/home/keanu/Programs/root_src/tmva/tmva/inc/TMVA/DNN/Architectures/TCudnn.h(181): here
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeLSTMDescriptors(TMVA::DNN::TDescriptors *&, TMVA::DNN::TCudnn<AFloat>::LSTMLayer_t *) [with AFloat=Float_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(40): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/RecurrentPropagation.cu(135): error: identifier "cudnnSetRNNDescriptor" is undefined
          detected during:
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRecurrentDescriptors(TMVA::DNN::TDescriptors *&, RNNLayer *) [with AFloat=Float_t, RNNLayer=TMVA::DNN::RNN::TBasicGRULayer<TMVA::DNN::TCudnn<Float_t>>]" 
/home/keanu/Programs/root_src/tmva/tmva/inc/TMVA/DNN/Architectures/TCudnn.h(184): here
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeGRUDescriptors(TMVA::DNN::TDescriptors *&, TMVA::DNN::TCudnn<AFloat>::GRULayer_t *) [with AFloat=Float_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(40): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(382): error: identifier "cudnnConvolutionFwdPreference_t" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(382): error: identifier "CUDNN_CONVOLUTION_FWD_PREFER_FASTEST" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(383): error: identifier "CUDNN_CONVOLUTION_FWD_NO_WORKSPACE" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(443): error: identifier "cudnnConvolutionBwdDataPreference_t" is undefined

[100%] Building CXX object roofit/roofit/CMakeFiles/RooFit.dir/src/RooDecay.cxx.o
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(444): error: identifier "CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(444): error: identifier "CUDNN_CONVOLUTION_BWD_DATA_NO_WORKSPACE" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(486): error: identifier "cudnnConvolutionBwdFilterPreference_t" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(487): error: identifier "CUDNN_CONVOLUTION_BWD_FILTER_NO_WORKSPACE" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(488): error: identifier "CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST" is undefined

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(387): error: identifier "cudnnGetConvolutionForwardAlgorithm" is undefined
          detected during instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeConvWorkspace(TMVA::DNN::TWorkspace *&, TMVA::DNN::TDescriptors *&, const TMVA::DNN::CNN::TConvParams &, TMVA::DNN::TCudnn<AFloat>::ConvLayer_t *) [with AFloat=Float_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(40): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(446): error: identifier "cudnnGetConvolutionBackwardDataAlgorithm" is undefined
          detected during instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeConvWorkspace(TMVA::DNN::TWorkspace *&, TMVA::DNN::TDescriptors *&, const TMVA::DNN::CNN::TConvParams &, TMVA::DNN::TCudnn<AFloat>::ConvLayer_t *) [with AFloat=Float_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(40): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(490): error: identifier "cudnnGetConvolutionBackwardFilterAlgorithm" is undefined
          detected during instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeConvWorkspace(TMVA::DNN::TWorkspace *&, TMVA::DNN::TDescriptors *&, const TMVA::DNN::CNN::TConvParams &, TMVA::DNN::TCudnn<AFloat>::ConvLayer_t *) [with AFloat=Float_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(40): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/RecurrentPropagation.cu(135): error: identifier "cudnnSetRNNDescriptor" is undefined
          detected during:
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRecurrentDescriptors(TMVA::DNN::TDescriptors *&, RNNLayer *) [with AFloat=Double_t, RNNLayer=TMVA::DNN::RNN::TBasicRNNLayer<TMVA::DNN::TCudnn<Double_t>>]" 
/home/keanu/Programs/root_src/tmva/tmva/inc/TMVA/DNN/Architectures/TCudnn.h(178): here
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRNNDescriptors(TMVA::DNN::TDescriptors *&, TMVA::DNN::TCudnn<AFloat>::RNNLayer_t *) [with AFloat=Double_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(41): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/RecurrentPropagation.cu(135): error: identifier "cudnnSetRNNDescriptor" is undefined
          detected during:
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRecurrentDescriptors(TMVA::DNN::TDescriptors *&, RNNLayer *) [with AFloat=Double_t, RNNLayer=TMVA::DNN::RNN::TBasicLSTMLayer<TMVA::DNN::TCudnn<Double_t>>]" 
/home/keanu/Programs/root_src/tmva/tmva/inc/TMVA/DNN/Architectures/TCudnn.h(181): here
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeLSTMDescriptors(TMVA::DNN::TDescriptors *&, TMVA::DNN::TCudnn<AFloat>::LSTMLayer_t *) [with AFloat=Double_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(41): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/RecurrentPropagation.cu(135): error: identifier "cudnnSetRNNDescriptor" is undefined
          detected during:
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeRecurrentDescriptors(TMVA::DNN::TDescriptors *&, RNNLayer *) [with AFloat=Double_t, RNNLayer=TMVA::DNN::RNN::TBasicGRULayer<TMVA::DNN::TCudnn<Double_t>>]" 
/home/keanu/Programs/root_src/tmva/tmva/inc/TMVA/DNN/Architectures/TCudnn.h(184): here
            instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeGRUDescriptors(TMVA::DNN::TDescriptors *&, TMVA::DNN::TCudnn<AFloat>::GRULayer_t *) [with AFloat=Double_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(41): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(387): error: identifier "cudnnGetConvolutionForwardAlgorithm" is undefined
          detected during instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeConvWorkspace(TMVA::DNN::TWorkspace *&, TMVA::DNN::TDescriptors *&, const TMVA::DNN::CNN::TConvParams &, TMVA::DNN::TCudnn<AFloat>::ConvLayer_t *) [with AFloat=Double_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(41): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(446): error: identifier "cudnnGetConvolutionBackwardDataAlgorithm" is undefined
          detected during instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeConvWorkspace(TMVA::DNN::TWorkspace *&, TMVA::DNN::TDescriptors *&, const TMVA::DNN::CNN::TConvParams &, TMVA::DNN::TCudnn<AFloat>::ConvLayer_t *) [with AFloat=Double_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(41): here

/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn/Propagate.cu(490): error: identifier "cudnnGetConvolutionBackwardFilterAlgorithm" is undefined
          detected during instantiation of "void TMVA::DNN::TCudnn<AFloat>::InitializeConvWorkspace(TMVA::DNN::TWorkspace *&, TMVA::DNN::TDescriptors *&, const TMVA::DNN::CNN::TConvParams &, TMVA::DNN::TCudnn<AFloat>::ConvLayer_t *) [with AFloat=Double_t]" 
/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu(41): here

21 errors detected in the compilation of "/home/keanu/Programs/root_src/tmva/tmva/src/DNN/Architectures/Cudnn.cu".
make[2]: *** [tmva/tmva/CMakeFiles/TMVA.dir/build.make:2273: tmva/tmva/CMakeFiles/TMVA.dir/src/DNN/Architectures/Cudnn.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:27388: tmva/tmva/CMakeFiles/TMVA.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

I’m don’t believe I’ve messed up the cuda and cdnn installations as I’ve installed tensorflow into the python environment and it seems to work fine.
Any little help would be appreciated, thanks in advance!

eguiraud · July 22, 2020, 7:31am

Hi @Popinaodude,
and welcome to the ROOT forum!
Thank you for the thorough report, I believe this is this same bug that was reported yesterday. @moneta should be able to comment further on possible workarounds or ETA on a solution.

Cheers,
Enrico

moneta · July 22, 2020, 12:11pm

Thank you for reporting this problem.
Unfortunately ROOT/TMVA does not build with the new cuDNN version, version #8. If you cannot downgrade to version 7 of cuDNN (and probably Cuda version 10) you either wait for the fix of this PR (https://github.com/root-project/root/pull/6058) of build ROOT without cudnn support ( -Dcudnn=Off)

Cheers

Lorenzo

Popinaodude · July 22, 2020, 2:01pm

Hi, guys
Thanks for the response . It’s very reassuring that it’s not a mistake that I made! I think I’ll downgrade (I assume cuda 10.1 works?) for now but I’ll keep an eye out for a fix so that i can rebuild with the newer version of cuda
Thanks again!

moneta · July 22, 2020, 2:33pm

Yes, cuda 10.1 with cudnn version 7 (e.g. 7.6.5 that is the current production version) works fine

eguiraud · July 22, 2020, 5:04pm

Hi,
other users might incur in the same issue, but posts in the Newbie category are deleted after a while. Would you be ok with moving this post to the ROOT category?

Popinaodude · July 22, 2020, 6:46pm

done!

Popinaodude · July 23, 2020, 10:33am

Hi,

I’ve realised I’m not able to install older versions of cuda on ubuntu 20.04. As it’s not too important I’m going to build root without cudnn support for the moment but does anyone have any idea of the timeframe for the cudnn v8 patch?

Cheers

moneta · July 23, 2020, 2:06pm

Hi,

Thanks for letting us know. Since we have a PR open with the cudnn fixes for version 8, I am expecting to integrate then in the ROOT master within few days

Lorenzo

system · August 6, 2020, 2:06pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.