PROOF doesn't care about xrootd server port?

ROOT
1

Hello,

I have a problem when my proof session tries to access data on a xrootd server running on non-standard port 1097. This server is external, in the sense that it doesn’t belong to my PROOF nodes.

What I do is create a chain, adding files such as:
chain->Add(root://ccapl0016.in2p3.fr:1097//);

then the chain is processed via
TChain::Process(const char* filename, Option_t* option, Long64_t nentries, Long64_t firstentry)

I end up with such error:

Looking up for exact location of files: OK (2578 files)
Cannot get entries for file: root://ccapl0016.in2p3.fr//xrootd/alice … iESDs.root - skipping
Cannot get entries for file: root://ccapl0016.in2p3.fr//xrootd/alice … iESDs.root - skipping
etc…

It looks like PROOF is trying to validate the files, but the port number in the URL is lost, so it fails.
Can you confirm the problem ?
And can it be fixed :slight_smile: ?

Thanks a lot,

Renaud.

2

Renaud,

Which ROOT version are you using?
I am not able to reproduce the problem.

What do you get by listing the chain content after the last Add(…)?

root [] chain->Add("root://ccapl0016.in2p3.fr:1097//<filename>");
root [] chain->ls();

Gerri

3

Hi Gerri

the chain->ls() output looks OK :

root://ccapl0016.in2p3.fr:1097//xrootd/ … iESDs.root tree:esdTree entries=1234567890
root://ccapl0016.in2p3.fr:1097//xrootd/ … iESDs.root tree:esdTree entries=1234567890
root://ccapl0016.in2p3.fr:1097//xrootd/ … iESDs.root tree:esdTree entries=1234567890

About the ROOT version, my user session runs v5-28-00f (both on client and workers).
[in case it is relevant : the xproofd daemon running on the cluster comes from v5.28.00d]

Thanks in advance for your help.
Ciao
Renaud.

4

Hi Renaud,

The xproofd version should not matter, but I am still not able to reproduce your problem.
Can you post the minimal macro reproducing the problem? You can put your original URLs, the problem is before opening the files, so I should be able to have it and debug it.

Gerri

5

Hi Gerri
See attached simplified selector and macro.
I use two files, one on a server running xrootd on port 1094, the other on 1097.
I have tried to run on v5-28-00d and v5-28-00f, both give the same problem.

Below is the output I get:

root -l -b -q process.C
root [0]
Processing process.C…
Starting master: opening connection …
Starting master: OK
+++ Starting max 2 workers following the setting of PROOF_NWORKERS
Opening connections to workers: OK (2 workers)
Setting up worker servers: OK (2 workers)
PROOF set to parallel mode (2 workers)
root://ccapl0016:1097//xrootd/alice/dat … iESDs.root tree:esdTree entries=1234567890
root://ccxrasn001:1094//xrootd/alice/da … iESDs.root tree:esdTree entries=1234567890
+++ Starting max 2 workers following the setting of PROOF_NWORKERS
Looking up for exact location of files: OK (2 files)
Looking up for exact location of files: OK (2 files)
Cannot get entries for file: root://ccapl0016.in2p3.fr//xrootd/alice … iESDs.root - skipping
+++dating files: 1 out of 2 (50 %)
+++ About 50.00 % of the requested files (1 out of 2) are missing or unusable; details in the ‘MissingFiles’ list
+++
Mst-0: merging output objects … done
Mst-0: grand total: sent 10 objects, size: 1210 bytes

Thanks,
renaud
macros.tgz (955 Bytes)

6

Hi Renaud,

Sorry for the late answer.
Thanks for the macros. I have tried them on several clusters (except yours, because I believe is behind a firewall and I do not have a machine to tunnel …) and always got the same result: the files are not found but the ports are correct. This is what I get on CAF, for example

For the sake of completeness, can you try to run on CAF and see what happens for you?

More generally, I do not know much what is going on in your case. It may be some setting. Can you post the xproofd config file? Also, what is the master log reporting for your attempts?

Gerri

7

Hello Gerri,

Do you mean with “the ports are correct” the fact that the port information is nor lost in your case ?

indeed, the port is not lost. In my case, I think it’s lost afterwards, because the functions giving errors are not the same as yours, and actually happen at the worker level, not master (see my master log below) :

Besides that, my cluster is indeed behind a firewall, but you can use ccali.in2p3.fr on which you already have an account.
if you want to access the files on my xrootd servers, just tunnel through ccali and you should be able to xrdcp etc.
You can also connect to the Lyon cluster, providing your DN.

I have just tried but I have to register again, I haven’t used it for a couple of years, now it’s ‘AAF’.

I attach the xrd.cf. For the master log, please see above.

Thanks for your input.
Renaud
xrd.cf.txt (5.95 KB)

8

Hi Renaud,

Unfortunately my account at ccali.in2p3.fr looks disabled (it must be 3 y that I do not use it … :blush: ).

Anyhow, yes, your problem happens on the workers.

Can you run setting this before chain->Process :

root[] gProof->SetLogLevel(2, TProofDebug::kPacketizer | TProofDebug::kGlobal)

and post the full master and workers log ?
This should allow to see what the master sends to the workers and what the workers receive.

Gerri

9

Hi Gerri

thanks, attached are all the logs (client, master, worker)
does it help ?
worker.log.txt (16.8 KB)
master.log.txt (19.4 KB)
client.log.txt (5.36 KB)

10

Hi Renaud,

In your master log there is no mention of port 1097, so it looks really that it does not come to that level.
Really weird.
What happens if you use a TDSet or a TFileCollection instead of a TChain?
For TDSet do:

void process() {
  gROOT->LoadMacro("esdTree.C+");
  TProof::Open("vernet@ccaplmaster.in2p3.fr","workers=2");
  TDSet *dset = new TDSet("ds");
  dset->Add("root://ccapl0016:1097//xrootd/alice/data/2011/LHC11a/000146805/ESDs/pass2_without_SDD/11000146805016.1210/AliESDs.root");
  dset->Add("root://ccxrasn001:1094//xrootd/alice/data/2011/LHC11a/000146805/ESDs/pass2_with_SDD/11000146805001.10/AliESDs.root");
  dset->Print("a");
  gProof->Process(dset, "esdTree");
}

For TFileCollection:

void processFC() {
  gROOT->LoadMacro("esdTree.C+");
  TProof::Open("vernet@ccaplmaster.in2p3.fr","workers=2");
  TFileCollection *fc = new TFileCollection("fc","");
  fc->Add("root://ccapl0016:1097//xrootd/alice/data/2011/LHC11a/000146805/ESDs/pass2_without_SDD/11000146805016.1210/AliESDs.root");
  fc->Add("root://ccxrasn001:1094//xrootd/alice/data/2011/LHC11a/000146805/ESDs/pass2_with_SDD/11000146805001.10/AliESDs.root");
  fc->Print("F");
  gProof->Process(fc, "esdTree");
}

Gerri

11

Hello Gerri

here are the outputs.

with the dataset:

with the collection:

Ciao,
Renaud