I came across another issue while configuring our Tier3 cluster. (For now I’m just connecting to the cluster as user “xrootd”.)
The PROOF master needs to access the worker directories of the workers. Usually this is accomplished by adding something as simple as
or something similar to the configuration file. But the issue for us is a bit more tricky. We use xrootd separately from the PROOF cluster to manage the disk space on the cluster. Actually each worker has two cache directories which are mapped to the “/atlas” directory with something like:
set osscachepath = /local/xrootd
oss.space public $(osscachepath)/* xa
oss.usage log /var/log/xrootd
I’ve found that with oss running, I can’t just simply export a “real” local directory. Adding the all.export line to the configuration file controlling our XRootD storage system doesn’t make the servers export their local work area.
So I thought, okay, let’s just serve the work areas using the xrootd instance running PROOF. So I added this piece of configuration to my configuration file that I use to configure the PROOF cluster:
all.export /local/home/proof r/w
However, even if I define my workers in the configuration file like this:
The PROOF master still tries to read the output ROOT files at the end of the job using the default 1094 port:
So my question is twofold:
Is it possible to export a “regular” directory with XRootD when another “virtual” directory is configured using OSS?
If not, then can I force PROOF to connect to the files created by the workers using a specific port?
Either approach would be fine for me.
Okay, so I managed to solve the issue by configuring the “data serving instance” of XRootD correctly. (I had some problems with the configuration of the authentication system.)
But I still maintain that this sort of flexibility (being able to freely specify where the master should look for the files created by the workers) would be a very useful feature. The more configurable the system, the easier it is to use it in complicated setups.
For the xrootd setups, I have to check. Perhaps with a symlink in the right place one can map the worker area as a third cache.
Anyhow, in PROOF the URL to access the local data is specified by the env LOCALDATASERVER which supports placeholders for the most common things to configure (see root.cern.ch/drupal/content/hand … OutputFile). The workdir specification in the worker (or slave) line is interpreted as local, so any URL specification in there is dropped.
Note that in 5.28/00d in principle you do not need to have a separate daemon for that (or to configure the data serving part of the PROOF xrootd to do that): if you specify ‘xpd.rootd on’ the system can access the data using the same port used for PROOF (it forks a ‘rootd’ on the server side; see root.cern.ch/drupal/content/conf … rootdallow).
Thanks for reminding me of the LOCALDATASERVER variable. I completely forgot about this one. But as I wrote before, now the separate xrootd instance seems to be configured correctly for serving the work directories of the proof processes. I just needed to add this in the right place of the configuration:
and had to make sure that the privileges for this directory are set properly. (This is where I made a mistake the first time, I didn’t know the correct syntax of the authorization files off the bat.)