Delete TFile doesn't work. SysError in <TFile::TFile>: file ~~.root can not be opened for reading (Too many open files)

Hello,
I built 3 programs that do the following:

  1. c++ executable that is opened by bash, reads, performs analysis, writing a TFile then exits.
  2. c++ executable that is opened by a bash script that lists all files in specified directories, performs analysis, updates all TFiles (one at a time but loops through all files within the single executable) related then exits.
  3. c++ executable that is opened by a bash script that lists all files in specified directories, reads all TFiles related (one at a time but loops through all files within the single executable) then exits.

\break

  1. Runs perfectly fine as it dynamically kills the executable after an individual TFile freeing up resources.
  2. Originally had “Too many open files” issue, but changing making sure to closing and deleting the TFiles in the loop helped. Now runs well.
  3. Declares the following:
    TFile*
    TObject*
    2TH1I*
    1TH2I*
    TCanvas*
    In the beginning, I check for null pointer
    TFile * file=NULL;
   //Check for zombie
    file = new TFile( filename.c_str() , "read");
    if ( file==NULL){//file->IsZombie() ){
        cout<<"This is a dead file"<<endl;
        file->Close();
        delete file;
        return;

then at the end of the method, I close
TCanvas, TFile and delete TCanvas, TFile and TObject I seem to get the Too many open files error.

I am having trouble understanding how to properly “close” files and how TFile counts as a “file” by bash. I am thinking of changing programs 2 and 3 to similar type as 1 to avoid this problem, but is there another solution? Is there a proper way to close TFiles?

As you’ve seen, right now I close TFile* file by first
file->Close();
then
delete file;

Thank you for your time.

// I should mention that I can go around it with setting ulimit to hard limit, but I rather not because this is more TFile issue rather than number of files that need to be open.


_ROOT Version: 6+
Platform: OpenSUSE LEAP 15.0
Compiler: g++7+


I guess @pcanal can help you.

I think you meant to write (to avoid using a nullptr):

    TFile * file = nullptr;
   //Check for zombie
    file = new TFile( filename.c_str() , "read");
    if ( file==nullptr) {
        cout<<"This is a dead file"<<endl;
        return;
    } else if (file->IsZombie()) {
        cout<<"This is a Zombie file"<<endl;
        delete file;  // This also does the Close when appropriate.
        return;
    }

Hello,
You are right, but it doesn’t help me close and delete TFile properly since after this point, the pointer is still pointing at the file.

My question is to get around the error

SysError in <TFile::TFile>: file ~~.root can not be opened for reading (Too many open files)

TFile *file = TFile::Open(filename.c_str()); // try to open it for "reading"
if ((!file) || file->IsZombie()) { delete file; return; } // just a precaution
// ...
// ... retrieve and use any objects you need from your "file"
// ...
delete file; // no longer needed ("cleanup")

As Wile is pointing out you also need to delete the files that are successfully opened.

Thank you @Wile_E_Coyote and @pcanal

I actually read one of the posts suggesting using TFile::option rather than to use the constructor, and unfortunately it didn’t change the outcome for me.

Furthermore, I found that if I use the TFile:: option, I get a segment fault if the file does not exist.

However I will try the following:

  1. use nullptr instead of value NULL
  2. delete file as soon as the elements I am looking for has been read.

A quick question though. What if iI am opening the file with “update” option to read, then write new components in it?

It would not. For local file they are equivalent.

get a segment fault if the file does not exist.

Maybe you are not checking for nullptr before using it …

What if iI am opening the file with “update” option to read, then write new components in it?

Don’t forget to call ‘file->Write()’ before deleting it then.

Using the null pointer and using the constructor
TFile file* = new TFile(file, “update”);
followed by
file->Close();
delete file;
at the end of the loop so far seems to work so far.

It seemed to have solved the issue but I am getting the same error yet again. I am confused and do not understand beause I did it very carefully. All pointers are kept in their original state until delete, and all pointers are deleted, and the only one TFile* is closed and deleted.

Question from this post:


How did TH1::AddDirectory(kFALSE) help?
Furthermore,
TFile *file = TFile::Open(filename.c_str());
yields segment fault whereas
file = new TFile( filename.c_str() , “read”);
does not, if the root file exists. The constructor gives me a chance to check for null pointer before crashing.

TFile *file = TFile::Open(filename.c_str(), "UPDATE"); // open a file for writing
if ((!file) || file->IsZombie()) { delete file; return; } // just a precaution
// ...
// ... retrieve and / or create and use any objects you need
// ...
file->Write(); // make sure everything is really written
delete file; // no longer needed ("cleanup")

Unfortunately, still not working. In fact using the function Tfile::Open crashes the executable way earlier than before when I would use the constructor.

I guess you need to attach some minimal version of your source code for inspection.

Below is the censored/minimalistic version of the source code.

#includes...
//Using the C standard namespace
using namespace std;

//Methods shall be listed below until the main.
//many smaller methods censored//
//textdrawer
void drawText(double x, double y, TString tstring, int col, double size=0.04) {
    TLatex tex;// = new TLatex();
    tex.SetNDC();
    tex.SetTextFont(42);
    tex.SetTextSize(size);
    tex.SetTextColor(col);
    tex.DrawLatex(x,y,tstring);
    //delete tex;
}//textdrawer Ends


void load_header(TFile* file, string keywords[], float headerlist[] ){

    TTree *dummytree;
    file->GetObject("file_info", dummytree);
    float dummyfloat=0;
    dummytree->GetEntries();
    for(int i=0; i  <17; i++){
        dummytree->SetBranchAddress( keywords[i].c_str() , &dummyfloat);
        dummytree->GetEntry(0);
        headerlist[i]=dummyfloat;
    }
    delete dummytree;
}


//General 1D or 2D plotter Begins
void General_Plotter(string filename, string plot){
    cout<<"Reading "<<filename.c_str()<<endl;
    //    TFile * file=nullptr;
   //Check for zombie

    ///    TH1::AddDirectory(kFALSE);
    //    TH2::AddDirectory(kFALSE);
    //    TFile* file = new TFile( filename.c_str() , "read");
    TFile* file = TFile::Open( filename.c_str() , "read");
    if ( file==nullptr){//file->IsZombie() ){
        cout<<"This is a dead file"<<endl;
        file->Close();
        delete file;
        return;
        //exit(0);
    }


    string keywords[17];//censored
    float headerlist[17];
    load_header(file, keywords, headerlist);

    //Reformatting the input filename and plot type
    eliminateExt(filename);
    string outputfilename = newPostfix(filename,".png");
    outputfilename = newPrefix(plot , outputfilename);
    //Outputfilename declared
    string directoryname;
    stringstream dummystringstream;
    dummystringstream.str( std::string() );
    string plotaccessname;
    dummystringstream<<"DIR_"<<plot.c_str();
    cout<<"Plotting "<<plot.c_str()<<filename.c_str()<<" As "<<outputfilename.c_str()<<endl;
    dummystringstream<<extension_count(filename);
    directoryname=dummystringstream.str();
    //Directoryname declared
    cout<<"Creating directory: "<<directoryname.c_str()<<endl;
    dummystringstream.str( std::string() );
    dummystringstream<<"mkdir -p "<<directoryname<<endl;
    string directorycommand=dummystringstream.str();
    //made Directory
    dummystringstream.str( std::string() );
    dummystringstream<<plot.c_str()<<filename.c_str();
    plotaccessname= dummystringstream.str();
    dummystringstream.str( std::string() );
	
	
    TObject *drawer= file->Get( plotaccessname.c_str() ) ;
    //Getting the plot
    //Checking for dead plot
    if( drawer==nullptr ){ // drawer->IsZombie() ){
        cout<<"This is a dead plot"<<endl;
        delete drawer;
        file->Close();
        delete file;
        return;
        //exit(0);
    }
    //Canvas for drawing objects
    TCanvas *can = new TCanvas( plotaccessname.c_str(),plotaccessname.c_str(), 2560,1440);
    system (directorycommand.c_str() );


    string Classname= drawer->ClassName();
    string outputfilename2 = outputfilename;
    dummystringstream.str( std::string() );
    dummystringstream<<directoryname.c_str()<<"/"<<outputfilename.c_str();
    outputfilename= dummystringstream.str();
    dummystringstream.str( std::string() );

    //If case when TH1I
    if (Classname == "TH1I" ){
        cout<<"It's TH1I"<<endl;
        TH1I* dummy1d= (TH1I*)file-> Get( plotaccessname.c_str() ) ;
        can->SetLogy();
        gStyle->SetOptStat(111111);
        //If it is TH1I print all stats and plot it.
        //If it is a zeroed TH1I change the axis by ghost dummy.
        //dummy1d->GetXaxis()->SetRange(dummy1d->FindFirstBinAbove(0) , dummy1d->FindLastBinAbove(0));
        //check(dummy1d->FindFirstBinAbove(0));
        //check(dummy1d->FindLastBinAbove(0));


        if(plot=="Positives_" || plot == "Negatives_"){
            can->SetLogy(0);
        }

        dummy1d->DrawCopy();
		}
    //If case when TH2I
    if (Classname == "TH2I" ){
        cout<<"It's TH2I"<<endl;
        TH2I* dummy2d=(TH2I*)file->Get( plotaccessname.c_str() ) ;
        gStyle->SetOptStat(0);
        //Eliminate the stats box to see more clearly
        //If it is a zeroes image set to low range to see more clearly
        gStyle->SetPalette(1);
        dummy2d->SetMaximum(2500);
        dummy2d->SetMinimum(-250);


    }//If case when Th2I ends

    float increment=0.015;
    for (int i=0; i<16; i++){
        drawText(  0 ,0.05+increment*i, newPostfix (keywords[i],"="),kGreen, increment );
        drawText( increment*4,0.05+increment*i, to_string( (int)headerlist[i] ),kGreen, increment );
    }
    time_t now = time(0);
    string timestamp = ctime(&now);
    drawText( 0 ,0.90, "Time Stamp =",kRed, increment*2 );
    drawText( increment*4*2,0.90, timestamp ,kRed, increment*2 );
    can->Print( outputfilename.c_str() );
    delete drawer;
    file->Close();
    delete file;
    can->Clear();
    can->Close();
    delete can;
}//General 1D or 2D plotter ends.



//Main Begins
int main(int argc,char *argv[]){
    //If no entry don't do anything... for now apparently.
    if (argc==1){}
    //If more than 1 entry, plot it unless "all is called"
    if (argc>1){
        //If asked for all create a temp list and plot them all
        if(argv[1] == ( char* ) "all" ){
            cout<<"asked for all"<<endl;
            system(" ls | grep '\\.root$' > megarootsfilez.SJL.tmp");
            string dummystring;
            ifstream rootfilelist;
            rootfilelist.open("megarootsfilez.SJL.tmp");
            //While reading the temp file list
            while(rootfilelist >> dummystring){
                cout<<dummystring<<endl;
				General_Plotter(dummystring, /*censored*/);

            }//While read Ends.
        }//All case Ends.
    //Else case
        else{
//censored
        }//Else case Ends
    }//If more than 1 entry case Ends.
    return 0;
}//Main Ends

You can debug your executable (and / or use valgrind):

`root-config --cxx --cflags` -O0 -g -Wall -Wextra -o YourExecutable YourMain.cxx `root-config --libs`

Yes, I have already done that and valgrind gave me 100+ handful of definitely lost referring to the ROOT packages. None of which are coming from my source code, but only from root libraries.

At least valgrind says it’s an issue with root 6.14. I doubt 6.16 will be any better.

This is very unusual. For local files the 2 call do essentially the same. When using TFile::Open what is the stack trace at the crash? When using TFile::Open was is the valgrind error reported (don’t forget to use --suppressions=$ROOTSYS/etc/valgrind-root.supp)

I will use the suppression option, I’ve actually not use that for valgrind. I will get back to you in a while with the stack trace because I am currently working on a work-around which seems to work for now.

I think I described it wrong above,

TFile *file = TFile::Open(filename.c_str(), "read"  );
//Yields segment fault IFF root file for filename.c_str() does not exist.
TFile *file = new TFile( filename.c_str() , “read”);
//Yields error throw IF root file for filename.c_str() does not exist.

If I use the former, the executable crashes with a segement fault in case the filename does not give a file address. If I use the latter

if(file==nullptr){
file->Close();
delete file;
return;
}

picks it up and ends the method to the next item.

if(file==nullptr){
    file->Close();

That should always crash … since the value of ‘file’ is a nullptr …

If using either of the method, the proper error handling can be

if (!file || file->IsZombie()) {
    delete file;
    return;
}