Cache with raw TFile over network

ROOT
1

Hello,

I am wondering if cache can be enabled for raw files over xrootd. I could not figure out how to enable cache (TFileCacheRead) from the documentation.

I am opening a file with TFile(“root://castorlhcb.cern.ch//castor/cern.ch/…?filetype=raw”) and then doing a lot of small (~100 bytes) reads. This seems to be very slow. Any tips on how to enable caching or improve performance for such kind of access pattern is much welcome.

2

Dear rmatev,

Using the option “?filetype=raw” by-passes ROOT internals and delegates to the system doing the transfer, XRootD in this case. This means looking at the way the XRootD client does caching, which depends on the version of XRooTD.

For the old client (XrdClient, XRootD versions 3.x.x) this means playing with


   // Read ahead size
   Int_t rAheadsiz = gEnv->GetValue("XNet.ReadAheadSize",
                                     DFLT_READAHEADSIZE);
   EnvPutInt(NAME_READAHEADSIZE, rAheadsiz);


   // Cache size (<= 0 disables cache)
   Int_t rCachesiz = gEnv->GetValue("XNet.ReadCacheSize",
                                     DFLT_READCACHESIZE);

For the new client (XrdCl, versions 4.x) the variables to play with are (likely)


   val = gEnv->GetValue("NetXNG.WorkerThreads",        "");
   if (val.Length() > 0 && (!(cenv = gSystem->Getenv("XRD_WORKERTHREADS"))
                            || strlen(cenv) <= 0))
      env->PutInt("WorkerThreads", val.Atoi());

   val = gEnv->GetValue("NetXNG.CPChunkSize",          "");
   if (val.Length() > 0 && (!(cenv = gSystem->Getenv("XRD_CPCHUNKSIZE"))
                            || strlen(cenv) <= 0))
      env->PutInt("CPChunkSize", val.Atoi());

   val = gEnv->GetValue("NetXNG.CPParallelChunks",     "");
   if (val.Length() > 0 && (!(cenv = gSystem->Getenv("XRD_CPPARALLELCHUNKS"))
                            || strlen(cenv) <= 0))
      env->PutInt("CPParallelChunks", val.Atoi());

To find out which client you have, you can check $ROOTSYS/etc/plugins/TFile/P100_TXNetFile.C and see what the default does .

If you are going to do small sequential reads, in the end reading the all file, then you may be better off by copying the whole file, for example using the option CACHEREAD to TFile::Open:

TFile::Open("root://castorlhcb.cern.ch//castor/cern.ch/........?filetype=raw", "CACHEREAD")

This first copies the file in a local cache and then opens from there.

G Ganis

3

Dear Ganis,

Thanks very much for the comprehensive suggestions!

It seems that TNetXNGFile and XrdCl were being used (with xrootd 3.3.6). I tried the options but saw no improvement. Looking at the code, XrdCl::File does not do any caching or look ahead.

I switched to using the old client and increased the read ahead, which helped significantly:
XNet.UseOldClient: yes
XNet.ReadAheadSize: 1000000

Your third option is somehow less attractive as I would need to make changes in the framework I am using. (And even when I did, it copied the filed locally, but then no events were processed…)

I will see with the xrootd developers if/how the new client can be made more performant.

Rosen