Writing to a file fails in rare cases with: SysError in <TFile::WriteBuffer>: error writing to file

ROOT
,
1

Dear all,

I have a program which creates and fills a root file with a TTree with a workflow that looks something like this:

//MyClass.h

MyClass{
    public:
    //executed once at the beginning
    void init();
    //executed every event
    void processEvent();
    //executed once at the end
    void end();

    private:
    int _x{};
    std::unique_ptr<TFile> _file;
    std::unique_ptr<TTree> _tree;
}

//MyClass.cc
void MyClass::init(){
    _file.reset( new TFile("file.root", "RECREATE") );
    _tree.reset( new TTree("treename", "treename") );
    _tree->Branch("x", &_x);
}

void MyClass::processEvent(){
    _x = 42;
    _tree->Fill();
}

void MyClass::end(){
    _file->Write();
}

I compile the code into the executable/library with -O3 optimization option once.

Then, I submit about 2000 jobs on a cluster to run this executable.
Each job creates and fills TTree and writes it as a root file.
I find that a small amount of jobs (around 0.5%-0.8%) don’t finish properly with the error starting like this:

SysError in <TFile::WriteBuffer>: error writing to file results.root (-1) Stale file handle
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:xHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.xHit, nbytes=-1, entry=4266
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:yHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.yHit, nbytes=-1, entry=4266
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:zHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.zHit, nbytes=-1, entry=4266
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:tHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.tHit, nbytes=-1, entry=4266
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:energyHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.energyHit, nbytes=-1, entry=4266
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:xHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.xHit, nbytes=-1, entry=4349
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:yHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.yHit, nbytes=-1, entry=4349
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:zHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.zHit, nbytes=-1, entry=4349
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:tHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.tHit, nbytes=-1, entry=4349
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:energyHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.energyHit, nbytes=-1, entry=4349
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:layerHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.layerHit, nbytes=-1, entry=4385
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:xHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.xHit, nbytes=-1, entry=4433
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:yHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.yHit, nbytes=-1, entry=4433
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:zHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.zHit, nbytes=-1, entry=4433
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:tHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.tHit, nbytes=-1, entry=4433
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:energyHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.energyHit, nbytes=-1, entry=4433
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:xHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.xHit, nbytes=-1, entry=4492
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:yHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.yHit, nbytes=-1, entry=4492
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:zHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.zHit, nbytes=-1, entry=4492
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:tHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.tHit, nbytes=-1, entry=4492
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:energyHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.energyHit, nbytes=-1, entry=4492
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:layerHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.layerHit, nbytes=-1, entry=4510
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:xHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.xHit, nbytes=-1, entry=4574
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:yHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.yHit, nbytes=-1, entry=4574
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:zHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.zHit, nbytes=-1, entry=4574
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:tHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.tHit, nbytes=-1, entry=4574
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:energyHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.energyHit, nbytes=-1, entry=4574
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:xHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.xHit, nbytes=-1, entry=4660
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:yHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.yHit, nbytes=-1, entry=4660
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)


Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:zHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.zHit, nbytes=-1, entry=4660
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:tHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.tHit, nbytes=-1, entry=4660
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:energyHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.energyHit, nbytes=-1, entry=4660
Error in <TBranchElement::WriteBasketImpl>: basket's WriteBuffer failed.
Error in <TBranchElement::TBranch::Fill>: Failed to write out basket.

Error in <TBranchElement::Fill>: Failed filling branch:layerHit, nbytes=-1
Error in <TTree::Fill>: Failed filling branch:treename.layerHit, nbytes=-1, entry=4679
 This error is symptomatic of a Tree created as a memory-resident Tree
 Instead of doing:
    TTree *T = new TTree(...)
    TFile *f = new TFile(...)
 you should do:
    TFile *f = new TFile(...)
    TTree *T = new TTree(...)

Thus, I suspect my TTree to be memory-resident, despite creating the TFile before TTree.

Questions related to this:

  • Is there some flag that I can use in the code to check if my TTree is memory-resident or not?
  • Could -O3 optimization do something to change the order of TFile and TTree initialisation?
  • Could std::unique_ptr or the style I create my TFile and TTree have something to do with this?

Despite this error, my job finished successfully processing all events.
The output root file is created with a non-zero size.
But opening it in the TBrowser fails with an warning:

Warning in <TFile::Init>: file file.root probably not closed, trying to recover
Warning in <TFile::Init>: no keys recovered, file has been made a Zombie

Simply re-running such failed jobs a second time works as intended, but it is very annoying to identify which jobs have failed and wait until they re-run for a second time every time I need to process 2k jobs.

ROOT Version: 6.28/06
Platform: CentOS Linux 7 (Core)
Compiler: g++ (Spack GCC) 12.2.0

2

At first look, this sounds like a disk hardware failure (and/or running out of disk space or quota). Can you try storing the files in a different file system?

3

Unfortunately, I can’t debug it with a different file system…

There might indeed be connection failure with the nodes; now, as I look at the job logs, they have some unexpected disconnections.

However, after the disconnect, the job is automatically rescheduled. But maybe a previously non-closed root file from a failed execution avoids the creation of a new root file?

Space/quotas must be fine.

4

Based on the error message, it is plausible that another process is deleting the file and/or the directory in which the file is written.

Note that in your example the file name does not vary so if you run the same script twice on the same machine and/or in the same network file system directory on a different machine, the 2 process will step on each others toes.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.