Skip many bad rootfiles in TChain? I tried "-1"

tigerstripedcat · December 17, 2007, 10:28pm

I join together several hundred root files and prune some of the trees. However, I’m getting crashing from bad root files.

First I chain the root files with:

TChain *ccbarChain = new TChain("GammaGamma");
ccbarChain -> Add("/data/jmorris/gammagammadata/ccbar/rootfiles/*.root",-1);

I’ve tried nentries="-1", “0”, “10”

1180   in -> Draw(">>inEvList", cut)
...
Error in <TChain::LoadTree>: Cannot find tree with name GammaGamma in file /data/jmorris/gammagammadata/uds/rootfiles/ntup_113.root
Error in <TChain::LoadTree>: Cannot find tree with name GammaGamma in file /data/jmorris/gammagammadata/uds/rootfiles/ntup_12.root
Error in <TChain::LoadTree>: Cannot find tree with name GammaGamma in file /data/jmorris/gammagammadata/uds/rootfiles/ntup_126.root
Error in <TChain::LoadTree>: Cannot find tree with name GammaGamma in file /data/jmorris/gammagammadata/uds/rootfiles/ntup_132.root
Error in <TChain::LoadTree>: Cannot find tree with name GammaGamma in file /data/jmorris/gammagammadata/uds/rootfiles/ntup_141.roo
...


in -> GetEntry(ev -> GetEntry(i));

I have 464 files. If 20 or so fail, nentries=“0” seems to work. If 30 or so, I get a seg fault. Is there a better way of handling these bad files? Am I using TChain correctly above if I use nentries=“0”?

Thanks,
James

pcanal · December 17, 2007, 11:04pm

Hi,

You should only use a positive number is all the tree in the file have the same number of entries (i.e. the number you are passing).
“0” and “-1” should behave the same.

Either way, this should not crash .
What is the stack trace at the time of the crash?
What does valgrind say when you run it with this example?

Cheers,
Philippe.

tigerstripedcat · December 18, 2007, 1:24am

Thanks for the response Philippe,

Here’s the backtrace:

*** Break *** segmentation violation
(no debugging symbols found)
Using host libthread_db library "/lib/tls/libthread_db.so.1".
Attaching to program: /proc/14538/exe, process 14538
(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread -1218588000 (LWP 14538)]
(no debugging symbols found)...done.
(no debugging symbols found)...done.
0x05b6851e in __waitpid_nocancel ()
   from /lib/tls/libc.so.6
Thread 1 (Thread -1218588000 (LWP 14538)):
#0  0x05b6851e in __waitpid_nocancel () from /lib/tls/libc.so.6
#1  0x05afd1d4 in do_system () from /lib/tls/libc.so.6
#2  0x00254d7f in system () from /lib/tls/libpthread.so.0
#3  0x005eb985 in TUnixSystem::Exec () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#4  0x005ebdb5 in TUnixSystem::StackTrace () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#5  0x005e9eab in TUnixSystem::DispatchSignals () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#6  0x005e7fa3 in SigHandler () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#7  0x005ee5e7 in sighandler () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#8  <signal handler called>
#9  0x02bc2040 in TTreeCache::SetEntryRange () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libTree.so
#10 0x01121e0c in TTreePlayer::Process () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libTreePlayer.so
#11 0x0111b8de in TTreePlayer::DrawSelect () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libTreePlayer.so
#12 0x02bb9b05 in TTree::Draw () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libTree.so
#13 0x02b9939a in TChain::Draw () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libTree.so
#14 0x02b992af in TChain::Draw () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libTree.so
#15 0x02be436a in G__G__Tree_131_0_37 () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libTree.so
#16 0x00b85dba in Cint::G__ExceptionWrapper () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#17 0x00c20fb0 in G__call_cppfunc () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#18 0x00c0fd8f in G__interpret_func () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#19 0x00bfeb7a in G__getfunction () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#20 0x00c80608 in G__getstructmem () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#21 0x00c7a202 in G__getvariable () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#22 0x00bf5ad7 in G__getitem () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#23 0x00bf475b in G__getexpr () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#24 0x00c38070 in G__exec_function () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#25 0x00c3f02c in G__exec_statement () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#26 0x00c116ba in G__interpret_func () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#27 0x00bff1c0 in G__getfunction () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#28 0x00bf5b67 in G__getitem () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#29 0x00bf475b in G__getexpr () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#30 0x00c38070 in G__exec_function () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#31 0x00c3f02c in G__exec_statement () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#32 0x00bdb952 in G__exec_tempfile_core () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#33 0x00bdbb72 in G__exec_tempfile () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#34 0x00c471cc in G__process_cmd () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCint.so
#35 0x0055ef57 in TCint::ProcessLine () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#36 0x0055f0b8 in TCint::ProcessLineSynch () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#37 0x0049a604 in TApplication::ProcessFile () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#38 0x00499cd3 in TApplication::ProcessLine () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#39 0x001ffd93 in TRint::HandleTermInput () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libRint.so
#40 0x001fe790 in TTermInputHandler::Notify () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libRint.so
#41 0x00200648 in TTermInputHandler::ReadNotify () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libRint.so
#42 0x005ea2e0 in TUnixSystem::CheckDescriptors () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#43 0x005e91cc in TUnixSystem::DispatchOneEvent () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#44 0x005198bc in TSystem::InnerLoop () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#45 0x00519861 in TSystem::Run () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#46 0x0049a826 in TApplication::Run () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libCore.so
#47 0x001ff824 in TRint::Run () from /afs/slac.stanford.edu/g/babar/package/root/5.14-00e/Linux24SL3_i386_gcc323/lib/libRint.so
#48 0x080488dd in main ()

How do you run valgrind with root?

I’ve tried valgrind root -l but it hangs at the segfault. Is there a preferred way to run this?

James

pcanal · December 18, 2007, 1:38am

[quote]I’ve tried valgrind root -l but it hangs at the segfault. Is there a preferred way to run this? [/quote]Not quite, you must use valgrind root.exe -l <myscript>because the executable named ‘root’ is small wrapper (with some sanity checks) around the executable named ‘root.exe’ and valgrind probably does not follow through the fork.

Cheers,
Philippe.

tigerstripedcat · December 18, 2007, 7:25am

I’m sorry Philippe, but I’m not very familiar with valgrind. But, naively, this seems to be a problem with my code:

==16058==  Address 0xBEFEA154 is not stack'd, malloc'd or (recently) free'd
==16058==
==16058== ERROR SUMMARY: 26 errors from 17 contexts (suppressed: 52 from 3)
==16058== malloc/free: in use at exit: 10,721,418 bytes in 59,661 blocks.
==16058== malloc/free: 1,393,551 allocs, 1,333,890 frees, 612,510,374 bytes allocated.
==16058== For counts of detected errors, rerun with: -v
==16058== searching for pointers to 59,661 not-freed blocks.
==16058== checked 7,362,656 bytes.
==16058==
==16058== LEAK SUMMARY:
==16058==    definitely lost: 68,481 bytes in 985 blocks.
==16058==      possibly lost: 406,016 bytes in 9,252 blocks.
==16058==    still reachable: 10,246,921 bytes in 49,424 blocks.
==16058==         suppressed: 0 bytes in 0 blocks.
==16058== Rerun with --leak-check=full to see details of leaked memory.

What is your opinion on this output? And I’d appreciate your expert opinion on handling memory leaks. Otherwise, I’m off to google a HOWTO guide; that is, unless you you think this a problem somewhere in the root source.

Thanks again for your help,

James

pcanal · December 19, 2007, 3:28am

[quote]==16058== Address 0xBEFEA154 is not stack’d, malloc’d or (recently) free’d [/quote]Humm strange, usually valgrind give a stack trace (at least partial) of where this is used. Anyway from your other stack trace, I have a couple of idea on how to reproduce the problem (of course, it would be even easier/faster if you provided a complete runnable example ).

Cheers,
Philippe

tigerstripedcat · December 19, 2007, 8:25pm

Thanks again for your help Philippe.

Here’s my shortest working example. I can tar a few of my ntuples and put it somewhere if you need these too. It seems like my code works up to a certain chain size. So I don’t think it’s a particular file, but after a certain number of failed files, it doesn’t work. And I get the seg fault above. But here’s the code:

{
double wUDS = 1.09391;

TCut masterCut = "";

TString myfile = "gammagamma1to5debug.root";
TFile *allgammagammaSubTrees = new TFile(myfile, "recreate");


TChain *udsChain = new TChain("GammaGamma");
udsChain -> Add("./uds/rootfiles/*.root",0);
TTree *udsSubTree = new TTree("uds", "uds gammagamma 1to5 DStar");

double weightOut;
bool  dStarMassOn = true;
double  dStarMassIn, dStarMassOut;


udsSubTree -> Branch("weight", &weightOut, "weight/D");

if(dStarMassOn){
  udsChain-> SetBranchAddress("dStarMass", &dStarMassIn);
  udsSubTree -> Branch("dStarMass", &dStarMassOut, "dStarMass/D");
}

udsChain-> Draw(">>inEvList", masterCut);
TEventList *ev = (TEventList *)gDirectory->Get("inEvList");
Int_t n = ev -> GetN();


TDirectory* savedir = gDirectory;
for (int i = 0; i < n; i++)
{
  udsChain-> GetEntry(ev -> GetEntry(i));
  weightOut = wUDS;

  if(dStarMassOn){dStarMassOut = dStarMassIn;}
  udsSubTree -> Fill();
}

udsSubTree->Write();
allgammagammaSubTrees->Close();
}

If you need me to tar together a few of my files (failing and working together, ~50meg) I can. Or if there is anything else you need, let me know.

Thanks again for your help.

james

pcanal · December 19, 2007, 8:28pm

[quote]If you need me to tar together a few of my files (failing and working together, ~50meg) I can. Or if there is anything else you need, let me know.[/quote] Please, do.

Thanks,
Philippe

tigerstripedcat · December 19, 2007, 8:39pm

Here you go:

slac.stanford.edu/~jmorris/ntups.tar.gz

51meg

tigerstripedcat · December 27, 2007, 9:11pm

Philippe,

Did you have any chance to look at this? There’s no real rush, I was just curious.

Thanks for helping me out.

James

pcanal · February 8, 2008, 10:01pm

Just to note that we have been unable to reproduce this problem so far (and that it was with an older version of ROOT (v4.xx?)).

Cheers,
Philippe.

tigerstripedcat · February 9, 2008, 4:14am

Yes, this is at SLAC and the problem seem to exist up to 5.14, but 5.18 works fine. Sorry for the noise.

james