Replacing TTree I/O Traversal with Arithmetic: A Coordinate-Indexed Read Path for CMS Open Data


The SSCCS Foundation is a non-profit organization developing open-source infrastructure to lower the barriers to scientific computing. One of our foundational projects, synTagma, is a coordinate engine that maps multidimensional coordinates, such as run, luminosity block, and event number, directly to memory addresses using closed-form arithmetic. It computes addresses in O(1) with no hash tables, B-trees, or index scans.

As we sought to validate synTagma at a production level, focusing on real-world impact and public utility, we came across the well-documented I/O bottleneck in CERN ROOT’s TTree.

The CERN ROOT TTree Problem

ROOT is the de facto data analysis framework for high-energy physics. Its TTree format stores billions of collision events in a columnar layout. The bottleneck lies in how a single event is read.

The legacy path follows a complex traversal: Branch, Basket, Cache, File. Each step accumulates system calls, cache misses, and context switches. The 2025 Fermilab CCESOP analysis documents the production signature: 372,000 singular reads averaging 4.6 KB, at an effective 33 KB/s, over roughly 14 hours. The cost is per request, not per byte; about 1.6 GB moves in seconds at raw bandwidth.

Two bottleneck regimes matter here. In the remote production pattern, per-request I/O wait dominates, around 98.8 percent of wall time, while the CPU stays mostly idle. On a local full-file measurement, the baseline is decompression CPU bound (CPU 182.9 s of 192.8 s wall). Both regimes are structural properties of the traversal, and both disappear when the traversal is replaced.

Replacing Traversal with Arithmetic

We integrated the synTagma C++ core into a ROOT fork and replaced the traversal path with a single arithmetic operation:

Index = (r Ă— L + â„“) Ă— E + e, where L is the maximum number of luminosity blocks and E the maximum number of events per block. Byte Offset = Index Ă— record size.

Given (Run, Luminosity Block, Event Number), the byte offset resolves in one step. No hash, no B-trees, no cache lookups.

Crucially, the TTree API remains unchanged. Analysis code, build chains, and researcher workflows stay exactly the same. We replaced only the interior read path, leaving the researcher interface untouched.

Measured Results: Validated on CMS Open Data

We measured the full CMS Run2016G DoubleMuon NanoAOD first file (2,315,223 events, about 2 GB) on the same local medium.

Read Path Wall Time Speedup
Legacy ROOT 192.8 s 1Ă—
Coordinate (standard I/O) 2.79 s 69.0Ă—
Coordinate (memory-mapped) 1.06 s 181.7Ă—

The memory-mapped path issues zero application-level read system calls; every event is copied directly from kernel-mapped memory, and the mapped pages are demand-paged once by the kernel.

Correctness was verified at two levels: the served bytes match a sidecar checksum, and an analysis workload over the same events reproduced identical results on both paths, 20,861 selected events with exact histogram equality, at 60.9Ă—. The one-time store conversion cost 226.3 s, comparable to a single baseline read pass and amortized from the second analysis pass onward.

One boundary, stated plainly: the documented 14-hour production workload is the reference pattern, and the measured scale is the full local file. We reproduced the access signature, reads of 1.37 to 2.76 per event at 2.6 to 0.8 KB per read, rather than the remote end-to-end run. The recorded rows are the committed benchmark artifact in the fork, reproducible from the runbook.

Public Contribution and Open Access

Beyond publishing the code, we wanted a permanent academic record. We registered the technical report on Zenodo, CERN’s open research repository, which assigns a free DOI to every research output. It closes a loop: using CERN’s infrastructure to share an open solution to a CERN problem.

Why This Matters

The pattern repeats beyond ROOT. Genomics, astrophysics, and climate modeling all share pointer chasing, cache misses, and many tiny reads. What we demonstrated is a structural principle: when traversal is replaced with arithmetic, per-request overhead disappears. The address becomes the traversal.

The technical report, implementation, and benchmarks are all open and linked below.

  • Technical report (DOI): doi.org/10.5281/zenodo.21888670
  • Technical report: docs.ssccs.org/works/cern/root-ttree
  • ROOT fork: github.com/ssccsorg/root
  • synTagma C++ core: github.com/ssccsorg/syntagma

ROOT Version: 6.41/01 (development version, master)
Platform: macOS 15.7.1 (arm64), Apple M1 Max, single Arm Firestorm 3.2 GHz core
Compiler: AppleClang 17.0.0 (clang-1700.6.3.2), C++17


Hi,

Thanks for sharing this nice research!
Do you have a workflow, e.g. an open data analysis or data processing, e.g. by an experiment, or even done with ROOT’s RDataFrame that is demonstrated to be faster thanks to synTagma?

I think I do not fully get what is meant by “We measured the full CMS Run2016G DoubleMuon NanoAOD first file (2,315,223 events, about 2 GB) on the same local medium.”: apologies in advance if I am missing something important!

Cheers,
Danilo

Hi Danilo,
Thanks for your reply!

Short answers after I cross-checked with AI:

  1. Yes, there is a demonstrated analysis workflow. The Analysis workload https://docs.ssccs.org/works/cern/root-ttree/#analysis-workload section runs a real selection (MET_pt > 100 GeV and at least one muon) filling the MET_pt histogram over the full 2.3M-event CMS Open Data file. Both paths reproduce identical results (20,861 selected events, histogram mean 132.696) and the coordinate path is 60.9x faster. It is GetEntry-based. RDataFrame is not demonstrated yet: from the ROOT source, TTreeReader reads via LoadTree and branch-level reads, so it does not go through the TTree::GetEntry short circuit today. Extending the seam to that layer is open work we plan to explore; it is not yet in the report’s Status plan.

  2. “Same local medium”, for the full-file comparison, means all three rows ran on the same machine against the same local disk: the baseline reads the original NanoAOD file, the coordinate rows read the converted fixed-width store on the same disk. Keeping the medium constant isolates the read path, which is the thing replaced. Why this matters is in Cost Analysis https://docs.ssccs.org/works/cern/root-ttree/#cost-analysis: remote EOS is I/O-wait bound (98.8% of wall), local is decompression-CPU bound (94.84%), and the same-medium setup keeps the storage layer out of the comparison. Dataset details are in Benchmark Methodology https://docs.ssccs.org/works/cern/root-ttree/#benchmark-methodology.

Full numbers, harness, and runbook are open in the fork:

  • https://github.com/ssccsorg/root/blob/master/benchmarks/tagma/tagma_bench.C
  • https://github.com/ssccsorg/root/blob/master/benchmarks/tagma/README.md
  • https://github.com/ssccsorg/root/blob/bench-artifact-reference/benchmarks/tagma/result/bench-20260810-153200-474b9af1ad.json

Cheers

Hello,

This is a lot of material to digest, sorry if I ask obvious things: but what is it compared exactly here?
I understand we deal with a toy analysis filling a histogram from Open Data CMS NanoAOD.
I understand that the IO mode of the baseline is remote reads (via XRootD to CERN’s EOS) without TTreeCache of a compressed dataset.
What is the investigated IO mode? Is the data local? If yes on which medium? Is the data still read compressed or uncompressed?

Best,
Danilo

Hi,

The headline comparison is local-to-local; remote EOS is only a separate 2,000-event slice.

Two separate mechanisms: the addressing removes the index and, when mapped, the system calls; the uncompressed store removes decompression (local baseline is decompression-CPU bound, 182.9s of 192.8s wall).

Compression-preserving addressing is the planned next phase; phase 1 uses the raw store. We will state this in the article.

Thanks for your question.

(add: as it’s an early integration/close to PoC level, the roadmap is not fixed. I would very welcome wider community discussion before the next phases (e.g. variable-length records, compression semantics, broader interfaces) are settled!)

Hi,

Thanks. Still a bit hard to follow, but much more scoped!
I think a comparison that would be useful is taking the toy analysis and run it on a full multiGB (better if multiTB) sample compressed, saved locally on a fast medium, in single and multi core (say O(128)), with baseline ROOT and with the novel approach.
That would be something that starts to approach HEP IO patterns for analyis or data processing.

Thanks again for all the details.

Best,
Danilo

Thank you for taking a quick look at the last comment. we appreciate your engagement.

Tagma’s problem‑solving scope is narrowly focused on indexing: a very thin C++ layer that replaces only index structures (hash maps, B‑trees) with coordinate arithmetic. This is intentionally minimal, it sits beneath the existing stack, keeping everything else unchanged.

The gap does widen with scale, as the O(1) arithmetic path becomes increasingly dominant over hash‑table degradation. Let us work and see again.

One more: SSCCS Foundation is a non‑profit open‑source full stack computing model initiative (Swiss foundation, legally in progress). We are pursuing a win-win contribution model: providing production‑grade contributions to domain‑leading open‑source projects, and feeding real‑world feedback back into our infrastructure primitives (synTagma).

We hope to establish a long‑term exchange with CERN and the ROOT team, and would welcome continued discussion, advice, and collaboration; for example even an official advisory invitation down on the road.

We will keep you updated as the work progresses, or any official enquiries.

Best,

Taeho Lee, Founder of SSCCS Foundation

Dear Taeho,

Thanks for your reply.
The ROOT project is open to exchanges and conversations like this one, as well as collaborations that can benefit the way in which HEP (and not only) data is processed.

Given the narrow scope of Tagma, which is completely fair and understandable, maybe what could be done is to be clearer with the statements made. The initial post explicitly mentions “The CERN ROOT TTree Problem” and offers a table calling ROOT main branch “Legacy” and citing 69x and 181.7x speedups in a way which, without context, could be misleading. TTree I/O was not optimised to be factors faster by Tagma.

Even if the priorities of Tagma do not foresee this for the short term, I am looking forward to see how the new approach can represent an advantage for ROOT and HEP data processing at large, for example benchmarking a workflow like the one described in my previous post.

Best,
Danilo

Thanks Danilo, we really appreciate the thoughtful reply and the openness of the ROOT project to conversations like this.

You raise a fair point about the framing, and it’s well taken. The title and the “Legacy” label in the table were meant to describe the structural pattern being targeted, not to imply that ROOT’s I/O is generally slow. TTree I/O has been optimised over many years, and the speedup numbers reported are specific to the narrow scope of replacing index traversal with coordinate arithmetic. I apologize for the potential misunderstanding caused by my choice of words.

So the document is already being revised based on this discussion, with more care taken to present the context clearly. The core idea remains valid (for workloads dominated by scattered index traversal, replacing that traversal with arithmetic gives a measurable gain) but this is a focused contribution, not a general critique of ROOT which has daily updating for sure.

On the benchmarking side, the suggestion is fully aligned with what needs to happen next. The current test is still at a hello‑world level of demonstration. The plan is to scale it up to multi‑GB and multi‑TB samples on fast local storage, and to run both single‑core and multi‑core evaluations. As originally planned, a typical HEP analysis workflow will be included, and results will be shared once ready.

This exchange has been very helpful, and we look forward to continuing it. If there is a specific dataset or analysis script you would recommend, we would be happy to include it.

Best,
Taeho Lee, SSCCS Foundation