Problems with hadd in lxplus

couet · June 11, 2024, 2:31pm

If i merge only the first 10 files it seems to work:

$ hadd -O merged.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0001.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0002.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0003.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0004.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0005.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0006.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0007.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0008.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0009.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0010.3.root
hadd Target file: merged.root
hadd compression setting for all output: 101
hadd Source file 1: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0001.3.root
hadd Source file 2: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0002.3.root
hadd Source file 3: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0003.3.root
hadd Source file 4: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0004.3.root
hadd Source file 5: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0005.3.root
hadd Source file 6: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0006.3.root
hadd Source file 7: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0007.3.root
hadd Source file 8: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0008.3.root
hadd Source file 9: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0009.3.root
hadd Source file 10: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0010.3.root
hadd Target path: merged.root:/

root_userj · June 11, 2024, 3:13pm

Sorry just to say that the -f0 is also required to prevent truncations of entries in the files.

I don’t think I see an obvious reason for this to not work for the following 3 files?

couet · June 11, 2024, 3:25pm

It works also with -f0 with only 10 files:

$ hadd -f0 -O merged.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0001.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0002.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0003.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0004.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0005.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0006.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0007.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0008.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0009.3.root data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0010.3.root
hadd Target file: merged.root
hadd compression setting for all output: 0
hadd Source file 1: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0001.3.root
hadd Source file 2: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0002.3.root
hadd Source file 3: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0003.3.root
hadd Source file 4: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0004.3.root
hadd Source file 5: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0005.3.root
hadd Source file 6: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0006.3.root
hadd Source file 7: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0007.3.root
hadd Source file 8: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0008.3.root
hadd Source file 9: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0009.3.root
hadd Source file 10: data24_13p6TeV.00473317.debugrec_hlt.reproc.HIST_DEBUGSTREAMMON.g69._0010.3.root
hadd Target path: merged.root:/

root_userj · June 11, 2024, 3:39pm

Yes it also seems to work with the addition of the 13th file but not the 11th or 12th and I don’t see an obvious reason for this?

couet · June 11, 2024, 3:43pm

May be they are corrupted somehow ?

yus · June 11, 2024, 3:55pm

Hi,
check this out: Check the structure of root files before hadd

What happens if you put your 11th (or 12th) file in the beginning of the list of files to be merged?

root_userj · June 11, 2024, 4:15pm

Hi @yus ,

Thanks for the info, the failure occurs for different ordering as can be seen in one of the previous replies. It is also not consistent when failing in general or across different operating systems.

root_userj · June 12, 2024, 10:18am

Hi @yus and @couet ,

The files should not be corrupted and can be merged but it is not consistent when they fail.

The merging works when the 11th or 12th or 13th files are placed at the start, but the only major difference I can see would be one of the string lengths (or one unfilled branch but there are many others that are fine during merging).

I already have to account for this by ordering for a different branch so I cannot simultaneously order all files based on many conditions (these are just one set of examples of many that need to be merged without input every time).

Is the string length in a branch preallocated based on the first file when running hadd (even if no compression is used)?

couet · June 12, 2024, 10:21am

I am not sure may be @pcanal knows ?

root_userj · June 18, 2024, 9:47am

Hi @couet,

Thanks, if @pcanal is able to give some insight that would be great thanks!

root_userj · June 27, 2024, 11:42am

Hi @couet , @pcanal, @vpadulan ,

Just wondering if there is any advice on this issue?

Cheers