Compiling ROOT 6.24 for ARM (Zynq Ultrascale+)

Dear ROOTers,

I have been trying to write a recipe to compile ROOT for the Zynq Ultrascale+ processor (ARM64). The idea is to provide this recipe to people building and OS for their SoC so that root is compiled along the OS and deployed in the image. This would be EXTREMELY useful for many ATLAS projects and beyond.

I use the petalinux tools from Xilinx to generate the OS image, and I provide the recipe to cross-compile ROOT on the host PC for the target architecture. My recipe goes as follow :

SUMMARY = "Numerical data analysis framework (OO)"
DESCRIPTION = "Object oriented framework for large scale data analysis"
HOMEPAGE = "http://root.cern.ch"
LICENSE = "LGPLv2.1"
PV = "6.24.00"
LIC_FILES_CHKSUM = "file://LICENSE;md5=5ec773ab82cbea1f17ec5b98e8ce60cf"
SRC_URI = "https://root.cern.ch/download/root_v${PV}.source.tar.gz"

S = "${WORKDIR}/${BPN}-${PV}"

inherit cmake pkgconfig clang

TOOLCHAIN = "clang"

DEPENDS += "cmake llvm-native clang-native llvm-native libx11 libxpm libxft libxext libgcc glibc packagegroup-core-buildessential python python3 dpkg binutils gcc  gcc-runtime \
autoconf automake ccache chkconfig glib-networking glibmm \
packagegroup-core-buildessential pkgconfig  \
boost cmake zlib glib-2.0 "
SECURITY_STRINGFORMAT = ""

do_configure_prepend(){
	export FC=${GFORTRAN}
}

# Specify any options you want to pass to cmake using EXTRA_OECMAKE:
EXTRA_OECMAKE = "\
    -Drootfit=ON \
    -Dminuit2=ON \
    -Dssl=ON \
    -Dbuiltin_freetype=OFF \
    -Dbuiltin_llvm=ON \
    -Dbuiltin_clang=OFF \
    -Dbuiltin_cling=ON \
    -DCMAKE_CXX_STANDARD=14\
    -DCMAKE_CXX_STANDARD_REQUIRED=ON\
    -DPYTHON_EXECUTABLE=python3\
"

SRC_URI[md5sum] = "d3d63eb2b3dc41ff392de66090a6859a"
SRC_URI[sha256sum] = "9da30548a289211c3122d47dacb07e85d35e61067fac2be6c5a5ff7bda979989"

From this recipe, the OS building tool (bitbake) download the source code and all the dependencies listed, and attempt to configure and build for the target system. This seems to work OK, but I am having a lot of trouble getting the configuration to finish. I must admit this is a bit over my head!

In this case I provide clang as toolchain for ROOT, but it seems there is a parameter related to the processor that is not understood by clang :

|     Run Build Command(s):ninja cmTC_401eb && [1/2] Building C object CMakeFiles/cmTC_401eb.dir/testCCompiler.c.o
|     FAILED: CMakeFiles/cmTC_401eb.dir/testCCompiler.c.o
|     /home/mbenoit/projects/Mercury_XU1_ST1_Reference_Design/FETB2-OS/build/tmp/work/aarch64-xilinx-linux/root/6.24.00-r0/recipe-sysroot-native/usr/bin/aarch64-xilinx-linux/aarch64-xilinx-linux-clang   -march=armv8-a+crc -mtune=cortex-a72.cortex-a53   -mlittle-endian -Wno-error=unused-command-line-argument -Qunused-arguments -fstack-protector-strong  -D_FORTIFY_SOURCE=2   --sysroot=/home/mbenoit/projects/Mercury_XU1_ST1_Reference_Design/FETB2-OS/build/tmp/work/aarch64-xilinx-linux/root/6.24.00-r0/recipe-sysroot  -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/mbenoit/projects/Mercury_XU1_ST1_Reference_Design/FETB2-OS/build/tmp/work/aarch64-xilinx-linux/root/6.24.00-r0=/usr/src/debug/root/6.24.00-r0                      -fdebug-prefix-map=/home/mbenoit/projects/Mercury_XU1_ST1_Reference_Design/FETB2-OS/build/tmp/work/aarch64-xilinx-linux/root/6.24.00-r0=/usr/src/debug/root/6.24.00-r0                      -fdebug-prefix-map=/home/mbenoit/projects/Mercury_XU1_ST1_Reference_Design/FETB2-OS/build/tmp/work/aarch64-xilinx-linux/root/6.24.00-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/mbenoit/projects/Mercury_XU1_ST1_Reference_Design/FETB2-OS/build/tmp/work/aarch64-xilinx-linux/root/6.24.00-r0/recipe-sysroot-native=   -march=armv8-a+crc -mtune=cortex-a72.cortex-a53   -mlittle-endian -Wno-error=unused-command-line-argument -Qunused-arguments -fstack-protector-strong  -D_FORTIFY_SOURCE=2   --sysroot=/home/mbenoit/projects/Mercury_XU1_ST1_Reference_Design/FETB2-OS/build/tmp/work/aarch64-xilinx-linux/root/6.24.00-r0/recipe-sysroot -o CMakeFiles/cmTC_401eb.dir/testCCompiler.c.o   -c testCCompiler.c
|     clang-9: error: the clang compiler does not support '-mtune=cortex-a72.cortex-a53'
|     ninja: build stopped: subcommand failed.

I attached the full CMake output and error log. Is there a way to suppress this option ?


If, instead, I ask clang to be builtin with -Dbuiltin_clang=ON , and provide gcc as toolchain, I get a different error during LLVMBuild part of configuration, but I cannot really indentify the issue. I attached the output of CMake in this case.

ROOT6_ARM_LOGs.zip (16.5 KB)

I am not sure how to proceed next, so I would appreciate a better trained eye or advice on how to proceed. If you are aware of other user doing the same compilation for Xilinx SoC, I would certainly also be interested in hearing from them. I know this is not the typical way to compile root, so bear with me …

Thanks a lot in advance for any insight or info,

Mathieu

1 Like

@mathieubenoit thank you so much trying to build ROOT with such an interesting build combination! (cc @axel)

So about builtin_llvm and builting_clang we are not using just any version of LLVM and Clang but only LLVM/Clang 9 that should be build with set of ROOT specific patches on the top. So basically in your case all mentioned before option should be on.

About error, it looks like a bug, Cmake thinks that you are using macox ARM64 (?) and running test to check if timers signpost passes in Xcode Instruments, which is definitely not true:

CheckSymbolExists.c:2:10: fatal error: os/signpost.h: No such file or directory
    2 | #include <os/signpost.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

Adding @vvassilev in CC as well.
Vassil, could it be #include <os/signpost.h> check is handled somewhere here? root/HandleLLVMOptions.cmake at master · root-project/root · GitHub

Thanks a lot for having a look! Indeed I saw this in the log, but I was not sure what is a normal failure and what is not. I suspected there was something to do with ROOT for arm macs :slight_smile: So I will wait for you investigation,

Cheers,

Mathieu

We don’t have really a place to reproduce your build so it could be a bit hard…

Indeed.

We have a docker image that could be used for that, or I could write instructions to use the xilinx petalinux tool, which is free.

I will have a look.

Hi, so I have created a docker image that can be used to reproduce this issue and test a possible fix.

The image can be found here.

I wrote some instructions on how to use the image and switch between different ROOT commits, that I have put here.

With that it should be possible to test it if you have a place to run docker.

Cheers,

Mathieu

@mathieubenoit sorry for late reply could you please if the image is still available via the link https://gitlab-registry.cern.ch/mbenoit/meta-caribou:latest ?

Hi, this is correct, or you can see it here : Container Registry · Mathieu Benoit / meta-caribou · GitLab .

Cheers,

Mathieu

Hi Mathieu,

I’ll try to set-up an environment to debug your issue as soon as possible. But since I’m fairly new to CMake, new to Docker, and usually working on Windows, it might take a bit of time. Sorry about that

Cheers, Bertrand.

So I tried following your instructions and here is what I get:

[caribou@bbcc7x64 build]$ bitbake root
NOTE: Started PRServer with DBfile: /home/caribou/poky/build/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 40283, PID: 64
WARNING: Host distribution "centos-8" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
ERROR:  OE-core's config sanity checker detected a potential misconfiguration.
    Either fix the cause of this error or at your own risk disable the checker (see sanity.conf).
    Following is the list of potential problems / advisories:

    Failed to create a file in SSTATE_DIR: Permission denied.
You could try using /sstate-cache in SSTATE_MIRRORS rather than as an SSTATE_CACHE.


Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

I set write access to all to the download folder, but it doesn’t change anything…

Hi @bellenot

are you using @mbenoit 's Docker command?

docker run -it --net=host -v ${PWD}/downloads:/downloads -v  ${PWD}/sstate-cache:/sstate-cache gitlab-registry.cern.ch/mbenoit/meta-caribou bash

If yes, then you have to make sure the Docker user has access to the folder ${PWD}/sstate-cache on the host machine (so on your computer). Here, ${PWD} is the folder from where you start the docker container. Same holds for ${PWD}/downloads.

Simon

OK, thanks, I’ll try

OK, I tried everything, nothing works so far. I’ll try to find out what’s going on and I’ll keep you in touch

Thanks a lot Bertrand for having a look ! Let us know if we can be of any help. This framework is a bit cumbersome indeed …

Cheers,

Mathieu

(if there is a simpler recipe with just a Dockerfile that still fails in the same way it might make things easier)

OK, so the build is ongoing. But looking more carefully at your log, it looks like the main issue is the -mtune=cortex-a72.cortex-a53 flag not supported by clang. So it’s maybe due to the version of clang used by ROOT.
EDIT: Well, maybe not, I’m lost with the informations in the log files. I’ll try to figure out in the docker build itself

@mathieubenoit @simonspa OK, so it build, and fails. Now, how (and where) can I run cmake to try to change some options? where is the build directory of ROOT? Where is the source directory of ROOT?

@mathieubenoit @simonspa Any help about my last question?

Hi @bellenot , sorry, got a bit busy.

The easiest way, in the current image you have, is to mount an extra volume with a checkout of root git repo :

mkir root_src 
cd root_src && git clone git://github.com/root-project/root.git && cd ..
docker run -it --net=host -v ${PWD}/root_src:/root_src -v ${PWD}/downloads:/downloads -v  ${PWD}/sstate-cache:/sstate-cache gitlab-registry.cern.ch/mbenoit/meta-caribou bash

Then from inside the image, there is a convenient tool call devtool, that you can use to point to bitbake where to pick the source to compile :

devtool modify -n root /root_src/root
[caribou@docker-desktop build]$ devtool modify -n root /root_src/root
NOTE: Starting bitbake server...
INFO: Creating workspace layer in /home/caribou/poky/build/workspace
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
NOTE: Started PRServer with DBfile: /home/caribou/poky/build/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 58659, PID: 73
WARNING: Host distribution "centos-8" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Parsing recipes: 100% |################################################################################################################################################| Time: 0:00:54
Parsing of 2276 .bb files complete (0 cached, 2276 parsed). 3404 targets, 135 skipped, 0 masked, 0 errors.

Summary: There was 1 WARNING message shown.
INFO: Recipe root now set up to build from /root_src/root

From now on the local copy of root in root_src in the host PC will be used to compile.

So when you issue bitbake root, it will use this version, that you can modify at will.

The cmake options are defined , inside the image in file at /home/caribou/poky/meta-caribou/recipes-caribou/root/root_git.bb

You can modify the EXTRA_OECMAKE variable there. I know it is not the most convenient.

@simonspa , is there a way to override the EXTRA_OECMAKE directly from command line, that you know of ?

Would that work for you ?

Cheers,

Mathieu