How can multiple (MPI) processes write to one ROOT file in parallel?

[ahesam]

Hello,

I am having some difficulties when multiple processes try to write to the same ROOT file. I have an MPI application that spawns N processes that at some point want to write some events to a TTree in the ROOT file.

I made a small reproducer of what I’m trying to achieve (see the end of this post).

First attempt

My first attempt was to (naively) use the default TFile::Write() method. This ended up in the errors:

Error in <TFile::ReadBuffer>: error reading all requested bytes from file test.root, got 63 of 300
Error in <TFile::Init>: test.root failed to read the file type data.
Warning in <TFile::Write>: file test.root not opened in write mode

Which I assume is the result of two processes trying to access one file at the same time.

Second attempt

In my second attempt, I tried using the TParallelMergingFile, which according to the reference of TBufferMerger should be able to “write data in parallel to a single output file … using processes that connect to a network socket”. As you can see in the reproducer below I tried using the default constructor as such. Using the “UPDATE” option I get the errors:

Error in <TMemFile::Init>: test.root not a ROOT file
Error in <TMemFile::Init>: test.root not a ROOT file
Warning in <TParallelMergingFile::Write>: file test.root not opened in write mode
Warning in <TParallelMergingFile::Write>: file test.root not opened in write mode

Using the “NEW” or “RECREATE” option, I get the errors:

SysError in <TUnixSystem::UnixTcpConnect>: connect (localhost:11111) (Connection refused)
Error in <TParallelMergingFile::UploadAndReset>: Could not contact the server localhost:11111

SysError in <TUnixSystem::UnixTcpConnect>: connect (localhost:11111) (Connection refused)
Error in <TParallelMergingFile::UploadAndReset>: Could not contact the server localhost:11111

I am guessing that I’m missing the step of setting up some sort of server that takes care of the merging. If that is the case, is there an example of how to do so?

Reproducer

#include "TFile.h"
#include "TObject.h"
#include "TParallelMergingFile.h"
#include "TTree.h"

#include <iostream>

#include "mpi.h"

int main(int argc, char **argv) {
  if (argc != 2) {
    std::cout << "./main <an_integer>" << std::endl;
    return 1;
  }
  int rank, size;
  MPI_Init(&argc, &argv);
  MPI_Comm_rank( MPI_COMM_WORLD, &rank );
  MPI_Comm_size( MPI_COMM_WORLD, &size );

  TObject obj;
  obj.SetUniqueID(rank);

  // TFile tfile("test.root", "UPDATE");
  TParallelMergingFile tfile("test.root?pmerge=localhost:11111", "UPDATE");

  TTree *tree = static_cast<TTree *>(tfile.Get("someTree"));

  // If tree doesn't exist yet; make a new one
  if (!tree) {
    tree = new TTree("someTree", "");
    tree->Branch("someBranch", &obj);
  } else { // else we modify the existing one
    TObject *obj_ptr = &obj;
    tree->SetBranchAddress("someBranch", &obj_ptr);
  }

  tree->Fill();
  tfile.Write();
  MPI_Finalize();
  return 0;
}

Any help is welcome!

---

ROOT Version: 6.13/08
Platform: Ubuntu
_Compiler: mpic++ (relies on g++ 5.5.0)

---

[etejedor]

Hi,
You can have a look at the net ROOT tutorials:

https://root.cern/doc/master/group__tutorial__net.html

There is an example of usage of TParallelMergingFile (client, server and test).

[amadio]

Writing from multiple processes to the same file is not supported as far as I know. There is someone working on developing an MPI equivalent of TBufferMerger (which can use threads or TBB tasks for now), you can ask @pcanal about it when he is back from vacations, as he knows more about it. At the moment, I think you’d have to implement an MPI version of TBufferMerger by hand while the other implementation is not yet merged. You can use TMemFiles, create TBufferFiles from your MPI tasks and then merge them with TFileMerger. You can probably use most of the TBufferMerger and TBufferMergerFile code as an example for how to do it.

[ahesam]

Thanks, @etejedor, the tutorials were quite helpful. Somehow my searches didn’t lead me to them. I seem to have achieved what I was looking for by doing:

Terminal 1 (server)

root -l $ROOTSYS/tutorials/net/fastMergeServer.C

Terminal 2 (MPI clients):

mpirun -n 2 reproducer

If you use the same port (changed 11111 to 9090), this results in a test.root file that contains two TObjects with each a distinct UniqueID (based on the MPI process’ rank, as according to the reproducer).

@amadio, you’re mentioning that someone is busy on implementing the MPI-equivalent of TBufferMerger, but what do you think is wrong with using TParallelMergingFile for this purpose?

[amadio]

I think using TParallelMergingFile is ok, just wanted to let you know someone is working on an MPI file merger.

[ahesam]

Just out of curiosity, how would the MPI file merger be different from TParallelMergingFile (i.e. what is the motivation for doing so)?

[amadio]

I’m not involved in the development, so I don’t know the details, sorry. I just remember it being discussed in the ROOT I/O meeting. You can find more details here: https://indico.cern.ch/event/696021. Cheers,