Hadd performance question followup

ROOT
1

Hello,

I would like to ask about the status of this question:

This topic was closed but it seems without the resolution.

Just to explain that waiting for hadd can take hours, which has direct negative impact on efficiency of our analysis work. We will be grateful for speedy resolution of this question.

Thanks,
Rustem

2

Hello Rustem,

@pcanal will be back from holidays on Monday, I suggest we wait for him to provide you with an answer.

3

I cannot be completely sure, but it used to be the case that deleting all histograms at the end would take a large amount of time. Which version of ROOT are you using? I think this was fixed in later versions of ROOT, at least for hadd.

4

OK, thanks for the replies. Do you already which ROOT version includes this now? Otherwise we will wait till Philippe is back on Monday.

Thanks,
Rustem

5

Hello @pcanal and @etejedor,

Just checking if there are some news on this?

Thanks,
Rustem

6

Hello @pcanal and @etejedor,

This issue has been open for more than two months. It seems like a trivial fix yet it would signficantly improve efficiency of our workflow. Currently, we run hundreds of jobs to produce large histogram files and often we have to wait longer for hadd them than time it takes to run jobs.

We would grateful for your response and help on this issue. Would it be possible to ask for your reply to this very real issue experienced by at least a few users?

Thanks,
Rustem

7

Hi @rustem,

Sorry for the delay, I will personally ask @pcanal about this.

Cheers,
Enric

8

Morning @etejedor and @pcanal,

Do you have any news on this issue?

As I have already mentioned, fixing this issue can really speed up our workflow. I have just talked to a student who has said hadd has been running for hours.

Thanks,
Rustem

9

Perhaps @Axel can also comment on this?

10

I pinged @pcanal . And I know that @StephanH suggested a patch removing the histogram recursive remove - what happened to that, Stephan?

11

I don’t know what happened. I made the suggestion, but I don’t know whether it was picked up in the end.

12

@pcanal is still investigating a solution!

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

15

As a workaround attempt, can you pass the option -n 2 (actually you can experiment with the numerical value)?
Also can you send me a sample of your files?

16

Hi,

I finally managed to go through hadd is to consistently delete the objects and directories ‘right’ after use (rather than accumulating them in memory) and reduce the amount of cleanups needed. This should resolved all (known) performance issues with hadd. The fix is (will be) available in v6.20/00 and 6.18/06.

Cheers,
Philippe.