Is it possible to download all builtin packages before compiling begins?

Hello ROOTers!

I’m considering pushing a build of the ROOT snap to arm64 platforms, providing a binary distribution for e.g, Raspberry Pi’s, however, I’m currently being held back by a rather peculiar requirement that I’m wondering might already have a solution.

I have access to some arm64 build servers, but these build servers operate behind a network proxy. Whilst the build time is not a limiting factor, the network proxy authorisation token is only valid for 2 hours, meaning that any network requests after 2 hours have passed will break the build even though the build itself can continue much longer.

I’m wondering if this use case might have already popped up (e.g, unreliable network conditions in general would effectively be wasting build time if e.g the remote servers are temporarily down). Preferably, I’d like ROOT to download all the builtin packages (e.g, clad) in advance, and then begin to compile them all at once. Otherwise, I’m rather inexperienced with CMake and while I’d probably try give this a go too, ROOT is a complex build, and I’d probably end up going down the route of trying cross compilations in QEMU emulation instead. Given how slow the build is even on native equipment, this would really suck.

Thanks in advance!
James

Hi @James-Carroll ,
at the level of cmake: we use cmake’s ExternalProject_Add to pull in dependencies, and that downloads them at build time. We recently bumped the minimum required cmake version to a cmake version that supports FetchContent, which would instead pull all dependencies at configure time, but built-ins have not been migrated to this system (nor we have a timeframe for the migration, I think).

A mix of these workarounds might do the trick:

  • use system installations of those dependencies instead of relying on the built-ins
  • tell cmake to build the targets that require a network connection (something like make clad, then the rest)
  • turn off options that require network at build time and are not usually needed

@bellenot @Axel or @amadio , who all know more than me about ROOT’s build system, might have better suggestions.

Cheers,
Enrico

2 Likes

My recommendation is to not use any builtins when creating a ROOT package, but use system provided dependencies instead.

1 Like

Thank you both for you recomendations.

The CMake FetchContent functionality looks like exactly what I’d be hoping for. It’s nice to here that ROOT may transition to this in the future, short term I’ll try mess around seeing if I can manipulate this myself just enough to unblock the builds.

I’ve re-evaluated what other builtins / network dependent components are being built and could be feasibly replaced with native packages, unfortunately there wasn’t much to gain there with an Ubuntu base (xrootd, vdt, openui5 are unpackaged). The majority of the build is LLVM/Clang/Cling and I’m told that replacing these parts with vanilla packages is something that causes problems, so I’d preferably avoid it.

And unfortunately running make clad first would complicate the snap build fairly significantly, due to some uniqueness in it’s build systems. I could give it a go, but I’d prefer to not try this in particular.

The simplest solution would appear then to simply be disable clad, purely since it falls over the two hour mark, I would like to avoid it because it’s a default enabled option and I’d prefer to keep it on to match normal user builds, but then it’s a build for low powered arm chips ( or M1 macs :wink: ) primarily anyway, so we’re already compromising utility for novelty .

Funnily enough, the build actually works fine on an s390x architecture, which I’ve basically next to zero knowledge about apart from it exists and the ROOT snap managed to build fine on it, even with the two hour network restriction. It’s meaningless, but I thought interesting enough to share.

Thanks for the suggestions and I’ll see what can be done :slight_smile:

A goal not bound to any build phase could be executed outside of the build before finally executing the package phase (and all its preceding build.

Thanks for the information. I will try to figure it out for more.