I get a bunch of numbers per event and would like to store them.
I have implemented a tree to do it. But what I noticed was, hadd takes incredibly large amount of time, of the order of 10 hours for what used to take just under a minute, when there’s no tree. the populated data is also not much (<50MB with tree) per file over 100s of files.
With tree : < 50MB per file, hadd : 10+ hours
Without tree < 1 MB per file , hadd : < 1 minute
Is there an alternative data structure that I can use or a way to reduce the time it takes to hadd the files ?