Alright, I am copying my solution here for anyone curious. I did not pre-convert into field tokens or use RNTupleProcessor due to the complexity of the framework I’m developing.
Why not? for those curious
Some further explanation if you’re curious, please look at the mock-up framework if you want to look at the code details.
I do not want the processors to have to know if they are reading from a file or not, so I centrally-manage the fields that are available within a “transient model” that is not connected to a source or sink. Reading from an RNTuple just binds values into the transient model for later observation; however, I want to be able to ignore some fields (Ignore fields when reading an RNTuple) in order to improve performance in some use cases of the framework (e.g. just filling histograms of a few columns). I also want to be able to read all of the fields in other use cases of the framework (e.g. a larger reconstruction pass where the framework copies all of the columns from the input file into the output file so I can delete the original input file that doesn’t have the new columns added).
I don’t think this use case should cause any design changes to the RNTuple code - it is fairly complicated and multi-featured so it makes sense to me that I have to write more complicated code to support this feature set.
This solution is able to support the possibility of pre-defining a set of fields that need to be read (and are the only ones to be read, i.e. I can incorporate “ignoring” fields like from Ignore fields when reading an RNTuple) at the cost of code complexity and a decent number of string-comparisons when transitioning between files.
// spy on the first file to get a model for reading
// this is also where I implement the ignoring of certain fields if applicable
auto reading_model = ROOT::RNTupleModel::Create();
{
ROOT::RNTupleDescriptor::RCreateModelOptions create_model_options;
create_model_options.SetCreateBare(true);
auto reader = ROOT::RNTupleReader::Open(create_model_options, TUPLENAME, FILEPATHS[0]);
const auto& desc = reader->GetDescriptor();
for (const auto& field_desc : desc.GetTopLevelFields()) {
// filter fields to ignore here
reading_model->AddField(field_desc.CreateField(desc));
}
}
// we aren't going to connect the reading_model to a source or sink
// but we freeze it anyways to make sure later code does not alter
// the set of fields we will be reading
reading_model->Freeze();
// get pointers to fields from the reading_model that will stay valid
// through all of the files processed
auto ifile_ptr = reading_model->GetDefaultEntry().GetPtr<int>("ifile");
auto value = reading_model->GetDefaultEntry().GetPtr<int>("value");
for (int ifile{0}; ifile < FILEPATHS.size(); ifile++) {
auto new_model = ROOT::RNTupleModel::Create();
{
// using the descriptor create a NEW model that WILL be connected
// to a source but use the reading_model to determine which fields
// to include and the address of the values for those fields
ROOT::RNTupleDescriptor::RCreateModelOptions create_model_options;
create_model_options.SetCreateBare(true);
auto spy = ROOT::RNTupleReader::Open(create_model_options, TUPLENAME, FILEPATHS[ifile]);
const auto& desc = spy->GetDescriptor();
for (const auto& value : reading_model->GetDefaultEntry()) {
std::string field_name{value.GetField().GetFieldName()};
auto field_desc_id = desc.FindFieldId(field_name);
if (field_desc_id == ROOT::kInvalidDescriptorId) {
throw std::runtime_error("File does not have a field required to be read by model, file mismatch");
}
const auto& field_desc = desc.GetFieldDescriptor(field_desc_id);
new_model->AddField(field_desc.CreateField(desc));
new_model->GetDefaultEntry().BindValue<void>(field_name, value.GetPtr<void>());
}
}
auto reader = ROOT::RNTupleReader::Open(std::move(new_model), TUPLENAME, FILEPATHS[ifile]);
for (auto ientry : *reader) {
reader->LoadEntry(ientry);
std::cout << *ifile_ptr << " -> " << *value << std::endl;
}
}
The full file includes writing some test files and a check that this code will throw the “File does not have a field required” exception. multi-read.cxx (3.4 KB)