Opening a file in update mode causes Error in <TBasket::Streamer>: The value of fNbytes is incorrect

jvavrek1 · February 1, 2021, 2:28am

I’m using Geant4’s G4AnalysisManager ROOT interface to save simulation output to a .root file. I can fill an Ntuple with data at each step with no issues, and the output file is fine. However, I would also like to store the number of primary particles in each run. To do this, I store a vector of number of primaries and write this to a TVectorD in the output .root file after the simulation is done, e.g.:

RunAction::~RunAction() {
  G4AnalysisManager *analysisManager = G4AnalysisManager::Instance();

  analysisManager->Write();
  analysisManager->CloseFile();

  delete G4AnalysisManager::Instance();

  WritePrimaries();

  G4cout << "\n--> Wrote to " << file_name.str() << G4endl;
 }

where my WritePrimaries method is:

void RunAction::WritePrimaries() {
  TFile *f = (TFile*) TFile::Open(TString(file_name.str()), "UPDATE");
  f->cd();

  size_t n = primaries.size();
  TVectorD p(n);
  for (size_t i = 0; i < n; ++i)
    p[i] = primaries[i];
  p.Write("primaries");

  f->Close();
}

This seems to corrupt the .root file. The data looks fine in a TBrowser but in terminal I get:

root [0] 
Attaching file output345.root as _file0...
(TFile *) 0x7f92dd832d70
root [1] TBrowser t
(TBrowser &) Name: Browser Title: ROOT Object Browser
root [2] Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:0, badread=1, nerrors=1, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:1, badread=1, nerrors=2, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:2, badread=1, nerrors=3, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:3, badread=1, nerrors=4, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:4, badread=1, nerrors=5, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:5, badread=1, nerrors=6, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:6, badread=1, nerrors=7, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:7, badread=1, nerrors=8, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:8, badread=1, nerrors=9, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
 file probably overwritten: stopping reporting error messages
Error in <TBranch::GetBasket>: File: output345.root at byte:0, branch:edep, entry:9, badread=1, nerrors=10, basketnumber=0
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
Error in <TBasket::Streamer>: The value of fNbytes is incorrect (-66564769) ; trying to recover by setting it to zero
...<repeated for many lines>...

and similarly if I do photonEdep->Scan() I can see that the first column is all zero instead of my test value of 1.23.

In fact, even if I don’t write anything at all, the file still gets corrupted! It seems that merely opening and closing the file in update mode is enough to corrupt it. In addition, this only happens on one branch—I was using this code without issue earlier and it has only started failing with the addition of a new branch.

Is there a safer way I can update this ROOT file?

ROOT Version: 6.18/00
Platform: Mac OSX 10.14.6
Compiler: Not Provided

pcanal · February 1, 2021, 10:06pm

It needs a file->Write (and a delete).

void RunAction::WritePrimaries() {
  TFile *f = (TFile*) TFile::Open(TString(file_name.str()), "UPDATE");
  f->cd();

  size_t n = primaries.size();
  TVectorD p(n);
  for (size_t i = 0; i < n; ++i)
    p[i] = primaries[i];
  p.Write("primaries");

  f->Write();
  delete f;
}

pcanal · February 1, 2021, 10:08pm

Depending on the type of ‘primaries’ you might also be able to skip the TVectorD:

f->Write(primaries, "primaries");

Another question is what is the scale of n ( low number or millions )?

jvavrek1 · February 4, 2021, 7:38pm

Hmm, I still get the same error here.

Currently it’s a std::vector<unsigned long int> which I was converting to a TVectorD for simplicity.

n is the number of Geant4 runs within the same simulation, so it’s a low number, typically 1 or 2 in my current application. In other cases it might top out at a few dozen.

pcanal · February 4, 2021, 8:16pm

so it’s a low number, typically 1 or 2 in my current application. In other cases it might top out at a few dozen.

So indeed saving it as a vector should be fine.

which I was converting to a TVectorD for simplicity.

Out of curiosity, how does it simplify things?

Hmm, I still get the same error here.

I can not see/guess why. Maybe running the failing writing with valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp ... would give a hint …

Are you using the actual ROOT libraries or the Geant4 re-implementation of (a subset) of Root I/O?

jvavrek1 · February 4, 2021, 9:14pm

It was a simple solution to eliminate the error raised when trying to write the std::vector<unsigned long int> directly, but certainly could be changed.

I think both here. I’m using the Geant4 re-implementation (via their AnalysisManager) to create the file and write Ntuples, but then I explicitly #include "TFile.h" for the WritePrimaries method. I’ll try to build a pure-ROOT MWE and see if that has the same problem.

pcanal · February 5, 2021, 1:46am

It was a simple solution to eliminate the error raised when trying to write the std::vector<unsigned long int> directly,

What error were you seeing?

jvavrek1 · February 5, 2021, 5:48pm

The error was:

/Users/jvavrek/ta_stj/src/RunAction.cc:154:6: error: no matching member function for call to 'Write'
  f->Write(primaries, "primaries");
  ~~~^~~~~
/Users/jvavrek/root/install/include/TFile.h:279:24: note: candidate function not viable: no known conversion from
      'std::vector<unsigned long>' to 'const char *' for 1st argument
   virtual Int_t       Write(const char *name=0, Int_t opt=0, Int_t bufsiz=0);
                       ^
/Users/jvavrek/root/install/include/TFile.h:280:24: note: candidate function not viable: no known conversion from
      'std::vector<unsigned long>' to 'const char *' for 1st argument
   virtual Int_t       Write(const char *name=0, Int_t opt=0, Int_t bufsiz=0) const;

Based on this post should it be WriteObject instead of Write?

pcanal · February 5, 2021, 6:05pm

Yes

jvavrek1 · February 6, 2021, 12:17am

Excellent, WriteObject seems to work. However the initial error still persists, and I can’t replicate it with a pure-ROOT MWE.

Do you mind taking a look at the attached example root files? I unfortunately have no experience with valgrind.

output_bad.root is the version where Geant4 writes the file, then I merely open and close it without even writing to it:

void RunAction::WritePrimaries() {
  TFile *f = (TFile*) TFile::Open(TString(file_name.str()), "UPDATE");
  f->cd();
  f->Write();
  f->Close();
  delete f;
}

Clicking on photonEdep/edep in a TBrowser triggers the error for me, but photonEdep/z does not.

Meanwhile output_good.root has not gone through this process.

output_bad.root (1.4 MB)

output_good.root (1.4 MB)

pcanal · February 6, 2021, 12:40am

Which version of Geant4 are you using? (I vaguely remember a similar sounding bug in their re-implementation).

jvavrek1 · February 6, 2021, 1:29am

Geant4 version Name: geant4-10-06 (6-December-2019)

jvavrek1 · February 6, 2021, 4:24am

The 10.7 release notes mention the following under Analysis management & plotting:

Fixed the memory leak reported by Valgrind in deleting files.

I’ll try out 10.7 as well when I get a chance.

jvavrek1 · February 6, 2021, 6:12am

The issue still exists in 10.7

pcanal · February 9, 2021, 10:03pm

Indeed. The geant4 reimplementation has a flaw that makes the file not update-able. Instead you need to first copy the file with ROOT (hadd -f output_updatable.root output_good.root) then update the copy.

Namely the problem is that TFile meta data record stored by the Geant4 is shorter than expected. Hence when ROOT write the TFile meta data it inadvertently overwrites enough of the data to make the file unusable.

See https://gitlab.cern.ch/geant4/geant4-dev/-/blob/master/source/analysis/g4tools/include/tools/rroot/directory#L85
vs
ROOT: TDirectoryFile Class Reference

Cheers,
Philippe.

jvavrek1 · February 10, 2021, 4:47am

Thanks for this investigation. This hadd trick works for me, so I’ll mark as a solution. Cheers!

system · February 24, 2021, 4:48am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.