Using remote XrootD filepaths with write support in ROOT

ROOT
1

Dear ROOT experts,

At KIT (GridKA), we have currently enabled authentificated writing to the user dCache via xrootd. This works fine so far, if using tools like xrd-copy.

However, if using the remote access implemented in ROOT via xrootd, using a valid proxy, I get errors,
that the accessed file is not writable. If using this code (only possible for ´dcms´ users):

TFile* f = TFile::Open("root://cmsxrootd-kit.gridka.de/pnfs/gridka.de/cms/disk-only/store/user/aakhmets/ggh_hpt_hists.root","write")
f->ls()
f->cd()
h = TH1D("h","h",1,0.0,1.0)
h.Write()

I get:

Error in <TNetXNGFile::WriteTObject>: Directory root://cmsxrootd-kit.gridka.de/pnfs/gridka.de/cms/disk-only/store/user/aakhmets/ggh_hpt_hists.root is not writable

Or, if trying to write to the file via hadd (essentially a TFile.Open() with "recreate" within TChain, as far as I understand):

hadd Target file: root://cmsxrootd-kit.gridka.de/pnfs/gridka.de/cms/disk-only/store/user/aakhmets/test.root
hadd compression setting for all output: 1
Error in <TNetXNGFile::WriteBuffer>: [ERROR] Server responded with an error: [3004] Tried to write on read only file.
Error in <TNetXNGFile::WriteBuffer>: [ERROR] Server responded with an error: [3004] Tried to write on read only file.

Other protocols (like dcap) are working with authorized, writable access (e.g. [*]) via hadd, but are limited to local networks, while XrootD is possible to be used from remote sites.

I’m using ROOT versions available at cvmfs,

either the LCG stack

source /cvmfs/sft.cern.ch/lcg/views/LCG_96b/x86_64-centos7-gcc9-opt/setup.sh

or the CMSSW version

source /cvmfs/cms.cern.ch/cmsset_default.sh
scram project CMSSW_11_0_0_patch1; cd CMSSW_11_0_0_patch1/src
eval `scramv1 runtime -sh`

Is this issue known? If yes, do you have any proposal to avoid it?

Please note, that we would prefer to write to the output directly via XrootD streaming and to avoid to produce local copies of outputs, which are then transferred with xrd-copy to the target. This would allow us to satisfy much more batch system requirements (essentially the scratch space on the batch node) in our analysis purposes.

Thank you very much in advance!

Cheers,

Artur Gottmann

[*] https://wiki.chipp.ch/twiki/bin/view/CmsTier3/HowToAccessSe#ROOT_examples

2

May be @pcanal can help you.

3

Hi!

Have you tried to set the variabel X509_USER_PROXY explicitely? You should be able to do this like the following:

export X509_USER_PROXY=`voms-proxy-info -path`

From my understanding, this is the technology behind the VOMS proxies and ROOT should pick this up then (?) If some expert like @pcanal could verify, I would appreciate :slight_smile:

Best
Stefan

4

Hi Olivier, Stefan and Philippe,

@swunsch, yes the voms proxy is set explicitly to a path via an export command for the X509_USER_PROXY variable and is picked up by ROOT, since I can read remote files. Only the write access is not working.

Cheers,

Artur

5

If gridka servers are configured like everything else in cms, you don’t need to specify local paths but use LFNs instead (just start with the /store part):

matevz@desire ~> xrdmapc cmsxrootd-kit.gridka.de:1094
0*r** cmssrm-kit.gridka.de:1094
  >   Srv 192.108.47.155:1094
matevz@desire ~> xrdfs  192.108.47.155:1094 ls  /pnfs/
[ERROR] Server responded with an error: [3011] Path not found:
matevz@desire ~> xrdfs  192.108.47.155:1094 ls /store/user/aakhmets/
/store/user/aakhmets/02-06-2019_analysis_mt_nominal
/store/user/aakhmets/03_07_2018_VBFM125_sync
/store/user/aakhmets/03_07_2018_VBFM125_sync_v2
...
6

Hi Matevz,

Thanks for your comment!

Yes, you’re right, to access files, the lfn paths can be used as a better and more general alternative. Concerning your example with xrdfs, for gridka, it actually works also with local pnfs paths (see my xrdfs <server> stat <path> command below) and copying to the remote site (via xrootd) works with an appropriate proxy too, if using xrdcopy:

akhmet@bms1:/portal/ekpbms1/home/akhmet/workdir/batch-system-merging $ echo "test" > test.txt
akhmet@bms1:/portal/ekpbms1/home/akhmet/workdir/batch-system-merging $ source /cvmfs/sft.cern.ch/lcg/views/LCG_96b/x86_64-centos7-gcc9-opt/setup.sh
akhmet@bms1:/portal/ekpbms1/home/akhmet/workdir/batch-system-merging $ xrdcopy -V
v4.10.0
akhmet@bms1:/portal/ekpbms1/home/akhmet/workdir/batch-system-merging $ xrdcopy -f test.txt root://cmsxrootd-kit.gridka.de/pnfs/gridka.de/cms/disk-only/store/user/aakhmets/test.txt
[5B/5B][100%][==================================================][5B/s]
akhmet@bms1:/portal/ekpbms1/home/akhmet/workdir/batch-system-merging $ xrdfs root://cmsxrootd-kit.gridka.de stat pnfs/gridka.de/cms/disk-only/store/user/aakhmets/test.txt
Path:   pnfs/gridka.de/cms/disk-only/store/user/aakhmets/test.txt
Id:     0
Size:   5
MTime:  2020-01-22 22:11:31
Flags:  48 (IsReadable|IsWritable)

So in summary, all xrd tools work properly. But that’s not the point, that I’d like to make.

The issue I encounter is the following:

It seems, that the access used in ROOT via TFile.Open() is using the xrootd access in a different way or just an older version of xrootd tools compiled into the ROOT build, so that writing to the remote file is not possible.

Reading an existing remote ROOT file is however not a problem, since this use-case is already widely used.

Cheers,

Artur

7

Hi Artur,

Interesting :slight_smile:
I’d guess cmssw-11 uses a relatively recent xroot … let me check … 4.10.0 … so it’s not super recent.

I think we need to test TNetXNGFile in a setup where we have more control over the situation, make sure things work with a vanilla xrootd server (GridKA is dCache, IIRC). I’ll start moving in this direction but it might take me a while to setup everything.

You might have more luck asking on CMS hyper-news, hn-cms-wanaccess.

Cheers,
Matevz

8

OK, this was easier than I thought :slight_smile:

With vanilla xrootd (started from command line, just ‘xrootd -d’, this serves local /tmp, -d for debug so you see what is happening), I can create a ROOT file and write into it:

 root [1] f=TFile::Open("root://localhost//tmp/foo.root", "recreate")
(TFile *) @0x7ffdb542ed58
root [2] h=new TH1F("a","b",100,0,1)
(TH1F *) @0x7ffdb542ed78
root [3] h->Write()
(int) 239
root [4] f->Close()
root [5] delete f

[root@desire tmp]# ll /tmp/foo.root
-rw------- 1 matevz zh 3793 Jan 22 15:25 /tmp/foo.root

Now, writing to UCSD storage, I can do xrdcp to /store/temp/user/matevz/ … but not to /store/user/matevz/, that’s how the site is setup. If I try opening a ROOT file in recreate mode, the thing fails as UCSD uses hadoop and hdfs only supports streaming writes, not random access:

matevz@desire ~> xrdcp pfc-monitor-1.eses root://gftp-3.t2.ucsd.edu//store/temp/user/pfcmonus.eses
[6.014kB/6.014kB][100%][==================================================][6.014kB/s]

root [9] f=TFile::Open("root://gftp-3.t2.ucsd.edu//store/temp/user/matevz/foo.root", "recreate")
Error in <TNetXNGFile::Flush>: [ERROR] Server responded with an error: [3005] Unable to synchronize /store/temp/user/matevz/foo.root; operation not supported

Error in <TNetXNGFile::WriteBuffer>: [ERROR] Server responded with an error: [3010] Unable to write /store/temp/user/matevz/foo.root; operation not permitted

(TFile *) @0x7ffdb542ed58

So, the bottom line, you have to figure out with GridKA admins what is supported on their site. People on hn-cms-wanaccess should be able to get you in touch with the right folks. The fact that you are talking to dCache via xrootd protocol might also be giving you additional trouble.

Good luck, I’m afraid you’re going to need it :wink:

Cheers,
Matevz

9

Hi Matevz,

Thanks a lot for having a closer look at this!

I understand, that the setup of the GridKA dDache might be very specific. If any information concerning the xrootd server configuration is needed, I may ask our colleagues from the computing center at GridKA and point to this discussion.

If I remember it correclty, the write access via XrootD was enabled just recently, since authentificated writing with XrootD was implemented properly only in the newer versions of the software (so at least >= 4.10.0, as shown by the commands in my previous comment). It might be, in fact, that the configuration of the server needs a bit more effort because of the additional write access, which seems to be, at least for the time-being, not a standard.

I’ll have also a look at the CMS hypernews you’ve pointed to, thanks for that.

Cheers,

Artur

10

Hi Matevz,

Ok thanks, I’ll point our GridKA admins to that thread :slight_smile:

Cheers,

Artur

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.