Access to files on eos: TBasket::ReadBasketBuf, TNetXNGFile::Open

[Kogutska]

Dear ROOT experts,

I am running a physics analysis on GRID. The input files are stored on ATLAS eos. In the bulk production, about 1/3 of jobs are getting broken due to failure to read input .root file properly.

I would appreciate your advice on whether this issue is related to ROOT or its compatibility with Python version or any other possible connection. If this is more likely caused by the eos site failure or memory management in algorithm, please let me know.

In the logs, I can see two types of errors:

1. Seems like authentication error

AthenaSummarySvc     INFO  -> file incident: root://dcgftp.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/LOCALGROUPDISK/rucio/mc16_13TeV/56/a6/DAOD_PHYS.23600331._000048.pool.root.1 [GUID: ]
TNetXNGFile::Open         ERROR   [ERROR] Server responded with an error: [3011] No such file
EventSelector       ERROR could not open next file in input collection [root://dcgftp.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/LOCALGROUPDISK/rucio/mc16_13TeV/0c/f9/DAOD_PHYS.23600331._000049.pool.root.1]
EventSelector       FATAL xAODEventSelector: Unable to fetch Ntuple: root://dcgftp.usatlas.bnl.gov:1096//pnfs/usatlas.bnl.gov/LOCALGROUPDISK/rucio/mc16_13TeV/0c/f9/DAOD_PHYS.23600331._000049.pool.root.1

2. Seems like memory allocation problem when reading an input file.

TBasket::ReadBasketBuf... ERROR   fNbytes = 0, fKeylen = 99, fObjlen = 142, noutot = 0, nout=0, nin=0, nbuf=0
TBranchElement::GetBasket ERROR   File: root://xrootd.echo.stfc.ac.uk:1094/atlas:datadisk/rucio/mc16_13TeV/b2/31/DAOD_PHYS.23600331._000004.pool.root.1 at byte:0, branch:TruthElectronsAuxDyn.pdgId, entry:1606, badread=1, nerrors=1, basketnumber=6
xAOD::TAuxStore::getData  ERROR   /build/atnight/localbuilds/nightlies/AthAnalysis/21.2/athena/Control/xAODRootAccess/Root/TAuxStore.cxx:311 Couldn't read in variable pdgId
TauProvider         FATAL  Standard std::exception is caught
TauProvider         ERROR SG::ExcBadAuxVar: Attempt to retrieve nonexistent aux data item `::pdgId' (218).

ROOT Version: 6.20/06
Platform: x86_64-centos7-gcc8-opt
Compiler: 8.3.0
Athena version: AthAnalysis, 21.2.182
Python version: 2.7.16

Thank you very much for any your help.
Best regards,
Kristina

[couet]

May be @pcanal can help.

[pcanal]

Those error seems to be on the server side (or possibly the local xrootd client). v6.20/06 is now a bit old and the xrootd client might not be updated enough for that server.

On the server side it could be (transient) error access the server ( xrootd.echo.stfc.ac.uk) or it could be a problem with the file.

One thing you can do is trying to access the file by itself in the ROOT prompt

TFile *f = TFile::Open("root://xrootd.echo.stfc.ac.uk:1094/atlas:datadisk/rucio/mc16_13TeV/b2/31/DAOD_PHYS.23600331._000004.pool.root.1");
nameoftheTTree->GetEntry(0);  // And a few more of or all the entries 

To see if you can reproduce the problem locally.