RDataFrame performance for branch reduction and event filtering (slimming and skimming)

[eguiraud]

@apetukho

/eos/
[83.4, 43.8, 54. 8] s
60.6 +- 20.5 s

root://
[65.1, 26.2, 69.5] s
53.6 +- 23.8 s

Mmmh you are dominated by noise and/or other latencies. root:// should consistently demonstrate the smallest minimum runtimes though.

@apetukho

where can I learn more about the available compression algorithms and their pros and cons?

@pcanal might be able to point you to some docs. The available algorithms are kZLIB, kLZMA, kLZ4, kZSTD. My understanding is the following: LZ4 is faster but compresses less well, LZMA is slower but compresses more, ZLIB and ZSTD are a bit of a middle ground but ZSTD is newer and typically closer to LZ4 speed while still having good enough compression.

@apetukho

I get a run time not proportional to the size of the files

Just a wild guess, but if the files are very small, the overhead of opening and closing the file might become important w.r.t. the actual event loop time. I’m happy to take a look though, “runtime does not scale linearly with number of events” is something that we should have a good explanation for Certainly the expectation is that it is!

Cheers,
Enrico