How to do sPlot with weighted events

[rooter_03]

Hi @Axel ,

OK, I am going to go though all of this in detail and carefully.

@Axel

More ROOTers? Always! But CERN tells us it doesn’t have an infinite budget, and so we need to prioritize, from beam to buildings to personnel.

I do not know how CERN spends their budget in detail. What I know is this. A student makes around 20-30 K a year. A postdoc makes around 50K a year. If you have bad software that student and postdoc will spend half of their time doing work that is not needed. How many students and postdocs work at CERN in a given year? Let’s say 200 postdocs and 400 students, that’s at least 5 million dollars wasted because of bad software. I am pretty sure that improving the software will cost less than that in man power and I bet CERN’s calculations are not taking into account the ammount of energy and effort that we waste.

We physicists are constantly faced with money issues. In order to get grants, we need to communicate the importance of the problem and the urgency to solve it to the people with the authority to provide funding. I think the key word here are communicating, convincing, urgency.

It is urgent to communicate the urgency of improving the software in order to allow physicists to perform and waste less resources.

Of course, I am not the one having to deal with management. They probably are stubborn and no argument, regardless of how well built, might get to them.

@Axel

So what you’re saying is that you’re unhappy with how we invest in ROOT. Because your activity here tells me that you’re well aware that we invest. (And no, sadly the argument “and that’s why the world doesn’t use ROOT” isn’t working, there’s tons of super crucial software with no support, and vice versa.)

Oh yea, I am definitely not happy. About the way you invest in ROOT? No, I did not say that. I think I did not specify why I am unhappy. I am unhappy because ROOT is badly designed, badly documented and It just requires far too much work to use. Let’s put it this way, I expect ROOT to be 1% of my life. The rest of my work already is 30% of my life. I know that for the developers it is 10%, maybe even 30% of their lives. However, the idea is that you have to deal with building good software, so that we do not have to deal with it. If I am a carpenter and my hammer is well built, I will never think of the hammer, I will think of the chair, because the chair is my job, not the hammer. I should not even notice that the hammer exists, it’s always there, it always works, it is easy to use and I could not live without it.

Also there are plenty of questions that required attention and did not get it. However, I do accept that there are places where support is far better than others. For example, I have got plenty of support when asking questions about RDataFrame. On the other hand, questions dealing with RooFit and RooStats mostly go unanswered or answered late. I can ask a question now but if you are going to reply in 3 weeks, that’s not good for me. By then I will have fixed it myself doing some nasty and time consuming hack.

Regarding the ROOT usage part. Well, if there is software that is crucial and no one maintains, that’s not a valid argument. Saying “ROOT does not have good support, but then what? XXX does not have good support either” is unnacceptable, we won’t be mediocre because others are. Ok, another advice here:

To PhD students: Steer away from ROOT as much as possible, although it won’t be possible always. You want to be a faculty? Of course you do, I do too, most of us want to be professor Dr XXX. But guess what? Only the very best will make it and sadly you probably aren’t among the very best. In my experience, most people end up quitting and doing Data Science, which is not bad, you will live a decent life. However, ROOT will be useless for you there. I have never seen (and feel free to check in glassdoor) a job ad for data scientist positions that requires the use of ROOT, TMVA, etc. Eventually, you might need to learn to use Python and libraries like TensorFlow, numpy, scipy, etc. Start now. Try to make this PhD experience also a learning experience that will serve you in your future career. For example, you have to train a BDT? Do not use TMVA, use TensorFlow. Do you have to make plots quickly? Try using matplotlib, not ROOT.

Back to you, ROOT people. This is another important point. For Physicists doing data analysis, C++ is a really bad language. How much raw talent will we waste seeing students trying to use C++ to do their analysis instead of Python? You need at least 3 years to be good at C++, and most students have little to no previous knowledge of C++ when they start their PhD. Maybe you @Axel think C++ is easier than it is, because you have worked with it for years. However it is not easy, it is hard and these students are not learning to be developers; they want to get plots of distributions and results to publish and put in their theses. I think, us people with this knowledge, should discourage as much as possible students from using C++. The algorithms can be written in C++, by an expert in C++. That algorithm should be available to the user in Python. And given what I have already said about the relities of the job market and academia (no faculty jobs + the job market does not care about ROOT) we should encourage students to become more familiar with python.

@Axel

Maybe it’s a bit brutal of you to resurrect a post from 15 years ago that hasn’t seen an answer; maybe it’s unfair of us to do development even though ROOT has bugs. Whatever it is: we are trying our best to keep you, one of our physicists, productive. Apologies if this failed here. Shall we try to address the issue you have? Maybe in a new topic, where you describe what issue you’d like us to address?

I understand your point, you think I cherry-picked an old post. This is what happened:

1. Work on a bug, a problem I do not understand and I cannot solve.
2. It’s been hours, maybe 5-7 hours working on the same thing.
3. I finally found a way to implement the test but I need to be able to do X.
4. Check the documentation but cannot find a way to do it.
5. Check the forum and find 3 or 4 posts about it, all of them ended up been unanswered. One of them seems promissing but, it also went unanswered, years ago.
6. Get angry and rant about it.
7. Spend time writting functions that I should not have to write myself, because X should have been implemented years ago.

My problem “X” is exposed here:

More Friendly interface to plot sPlotted data and deal with datasets in general

Hello, When working with the RooStats::sPlot class we get a dataset with multiple columns that can be interpreted as weights. For instance, if the model is: PDF=S(m) nsig + B(m) nbkg the resulting dataset will have the columns nsig_sw and nbkg_sw added. However, in practice, these columns cannot be read as weights, because weights have to be declared as such when the dataset is built. Also, it seems that you can have only one weight. In my use case, I need to generate a toy, calculate sweigh…

It is fair to say that It is a 15 year old post and that ROOT Is not like that anymore. However, I do believe that ROOT is not like that everywhere, but it is still like that in some places, like RooFit and RooStats, e.g:

Splot: picking background

Hi I am using the splot tool to disentangle signal and background in the Jpsi invariant mass distribution. I modified the splot tutorial to suit my requirements. Please have a look at the distributions from the toyTool Top left is the jpsi + exp background distribution fitted with the pdf model Top right is the same distribution with the components after calculating the sweights. 3)Bottom left is the jpsi mass distribution from the weighted dataset for jpsi yield 4)Bottom right is the exp…

Apply sPlot weights twice

Hello, @StephanH, We would really prefer to stick with unbinned if possible. 2D fits pose their own challenges, complicating what should be a simple fit. Multiplying them is certainly wrong as s-weights can be negative. It is, in my understanding, also incorrect to multiply s-weights with already-applied weights in general, however, because they are already factored into the weights output by SPlot. I am asking the forum primarily because I am not sure whether this is still true in the cas…

And It took me just one minute to find. Where the original developers are not in charge of the code anymore and people who had no knowledge of the libraries had to join and learn about them. I understand that for those people, learning how RooFit and RooStats work must be challenging. Now my point is relevant. A library as cental to Physicists as RooFit cannot be developed by someone who will dissapear and not show up anymore to provide support. This work has to be done by high quality, well paid, full time, permanent developers, ideally two of them, so that if one leaves, the other will be around to train a replacement and provide support.

I saw your reply days ago, however I waited to have a clear mind before replying to you. Am I been unfair with my comments? Maybe, I do not claim to know everything, but I do believe that there is some truth in what I have said and I never heard anyone saying this before. So I hope you do not take my words as an insult, but as the thoughts of a heavy ROOT user that have grown througout the years.

Cheers.