Segfault with weighted fit

Hi all,

We’re in the final stages of an analysis which uses weighted RooDataSets and a pretty complicated PDF. We have one RooDataSet, from which I produce two weighted versions with different per-event weights set. With one dataset, the fit converges properly and gives sensible SumW2 errors. For the other, we get a segfault in the integrator code. I attach the relevant output and a backtrace from GDB. It looks like RooTreeDataStore::get is returning null. Could someone take at a look at the messages below? I’d be very grateful for any hints on how we could fix or work around this.

We’re currently using a CMS build of the roofit from 5.26.00, but we can try other versions if this would help.

Ideas or suggestions would be very welcome!

Cheers!
Will

====
RooDataSet::data[HiJetsClass,Mt_pf,nJetBinLow,nBTaggedJetBin,weight:W] = 915 entries (1122.91 weighted)
start fitting…
Weighted? 1
[#1] INFO:Minization – p.d.f. provides expected number of events, including extended term in likelihood.
[#1] INFO:Minization – RooMinuit::optimizeConst: activating const optimization
[#1] INFO:Fitting – RooAbsTestStatistic::initSimMode: creating slave calculator #0 for state {HiJets0} (240 dataset entries)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(other_Mt_3j_Cruijff_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:Fitting – RooAbsTestStatistic::initSimMode: creating slave calculator #1 for state {HiJets1} (291 dataset entries)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(other_Mt_3j_Cruijff_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:Fitting – RooAbsTestStatistic::initSimMode: creating slave calculator #2 for state {HiJets2} (267 dataset entries)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(other_Mt_3j_Cruijff_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:Fitting – RooAbsTestStatistic::initSimMode: creating slave calculator #3 for state {HiJets3} (117 dataset entries)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(other_Mt_3j_Cruijff_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:Eval – RooAbsTestStatistic::initMPMode: starting remote server process #0
RooRealMPFE::initialize(nll_myFit_data_a00a028_MPFE0) successfully forked server process 17467
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(sig_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_1_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(other_Mt_3j_Cruijff_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(ttbar_Mt_3j_DoubleCruijff_2_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)
[#1] INFO:NumericIntegration – RooRealIntegral::init(other_Mt_3j_Cruijff_Int[Mt_pf]) using numeric integrator RooIntegrator1D to calculate Int(Mt_pf)

*** Break *** segmentation violation

Program received signal SIGSEGV, Segmentation fault.
0x02eaaa23 in RooTreeDataStore::get(int) const () from /home/lukas/Software/root/lib/libRooFitCore.so
(gdb) backtrace
#0 0x02eaaa23 in RooTreeDataStore::get(int) const () from /home/lukas/Software/root/lib/libRooFitCore.so
#1 0x02eaa5a8 in RooTreeDataStore::cacheArgs(RooAbsArg const*, RooArgSet&, RooArgSet const*) () from /home/lukas/Software/root/lib/libRooFitCore.so
#2 0x02cea48f in RooAbsData::cacheArgs(RooAbsArg const*, RooArgSet&, RooArgSet const*) () from /home/lukas/Software/root/lib/libRooFitCore.so
#3 0x02cfe6f8 in RooAbsOptTestStatistic::optimizeConstantTerms(bool) () from /home/lukas/Software/root/lib/libRooFitCore.so
#4 0x02cfeae0 in RooAbsOptTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#5 0x02d360e7 in RooAbsTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#6 0x02cfe86c in RooAbsOptTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#7 0x02e746a8 in RooRealMPFE::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#8 0x02d36133 in RooAbsTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#9 0x02cfe86c in RooAbsOptTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#10 0x02e1eb39 in RooMinuit::optimizeConst(bool) () from /home/lukas/Software/root/lib/libRooFitCore.so
#11 0x02d0c070 in RooAbsPdf::fitTo(RooAbsData&, RooLinkedList const&) () from /home/lukas/Software/root/lib/libRooFitCore.so
#12 0x02d06288 in RooAbsPdf::fitTo(RooAbsData&, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg) () from /home/lukas/Software/root/lib/libRooFitCore.so
#13 0x08054732 in Fitter_NBTaggedJet_Mt_Sim15::Fit (this=0xbfffed94, theFit=…, data=) at Fitter_NBTaggedJet_Mt_Sim15.cc:375
#14 0x0804dd9d in main (argc=6, argv=0xbffff1a4) at fit_pf_nbtaggedjet_mt_sim15.cc:168
(gdb) backtrace
#0 0x02eaaa23 in RooTreeDataStore::get(int) const () from /home/lukas/Software/root/lib/libRooFitCore.so
#1 0x02eaa5a8 in RooTreeDataStore::cacheArgs(RooAbsArg const*, RooArgSet&, RooArgSet const*) () from /home/lukas/Software/root/lib/libRooFitCore.so
#2 0x02cea48f in RooAbsData::cacheArgs(RooAbsArg const*, RooArgSet&, RooArgSet const*) () from /home/lukas/Software/root/lib/libRooFitCore.so
#3 0x02cfe6f8 in RooAbsOptTestStatistic::optimizeConstantTerms(bool) () from /home/lukas/Software/root/lib/libRooFitCore.so
#4 0x02cfeae0 in RooAbsOptTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#5 0x02d360e7 in RooAbsTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#6 0x02cfe86c in RooAbsOptTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#7 0x02e746a8 in RooRealMPFE::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#8 0x02d36133 in RooAbsTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#9 0x02cfe86c in RooAbsOptTestStatistic::constOptimizeTestStatistic(RooAbsArg::ConstOpCode) () from /home/lukas/Software/root/lib/libRooFitCore.so
#10 0x02e1eb39 in RooMinuit::optimizeConst(bool) () from /home/lukas/Software/root/lib/libRooFitCore.so
#11 0x02d0c070 in RooAbsPdf::fitTo(RooAbsData&, RooLinkedList const&) () from /home/lukas/Software/root/lib/libRooFitCore.so
#12 0x02d06288 in RooAbsPdf::fitTo(RooAbsData&, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg, RooCmdArg) () from /home/lukas/Software/root/lib/libRooFitCore.so
#13 0x08054732 in Fitter_NBTaggedJet_Mt_Sim15::Fit (this=0xbfffed94, theFit=…, data=) at Fitter_NBTaggedJet_Mt_Sim15.cc:375
#14 0x0804dd9d in main (argc=6, argv=0xbffff1a4) at fit_pf_nbtaggedjet_mt_sim15.cc:168
(gdb)

Hi all,

I’ve not identified the problem, but it is not an issue in 5.28.00. I guess the problem has been fixed. I will report back if I see the problem again.

Thanks,
Will

Hi Will,

Several problems in memory management were fixed between 5.26 and 5.28, mostly thanks to a Coverity analysis. I remember fix in an issue in RooTreeDataStore specifically, so that most likely fixed your problem too.

Wouter

Hi Wouter,

Thanks for letting me know and for the continuing hard work! Our weighted fit is now working well in 5.28.

Cheers!
Will