Disable TTree::AutoSave when calling TTree::Fill to write tree in parallel

Description of Problem

I am working from my previous example here. If I add a TTree to the writer, and call TTree::Fill(), then I get a race condition with AsyncFlush. The documentation tells me that TTree:Fill() will call TTree::AutoSave:

The baskets are flushed and the Tree header saved at regular intervals

I’m trying to disable this functionality, so that I can TTree::Fill() from one thread while writing the file to disk in another thread.

The Question

Is there a way to disable all automatic writing of TTrees to disk?

I can infer from this post that disabling auto saves is possible, but I cannot find any documentation about how to accomplish this.

Environment

Component Value
Operating system ubuntu
Kernel Version 4.4.0-43-Microsoft
CMake Version 3.5.1
GCC Version gcc (Ubuntu 5.4.0-6ubuntu1~16.04.6) 5.4.0 20160609
ROOT Version 6.13/01
GSL Version 2.4

Hi,
you are looking for TTree::SetAutoSave.
You can set it to std::numeric_limits<Long64_t>::max() or something like that to disable autosaving completely. You might also want to disable autoflushing (see TTree::SetAutoFlush).

Cheers,
Enrico

@eguiraud : I have been setting TTree::SetAutoSave(), but it seems like I haven’t set it high enough! Good call on auto flushing. I’ll check this tomorrow and report back. What happens in the undefined case that I set TTree::SetAutoSave(0)? The manual only mentions positive or negative numbers.

Thanks!

Hi,
looking at the source code for TTree::Fill, specifically TTree.cxx:4524, it seems that setting fAutoSave to 0 has a similar effect to setting fAutoFlush to 0 (it disables autosaving), although it’s not documented.

@pcanal is probably the right person to confirm or deny my conclusions, but you can easily test this yourself.

Cheers,
Enrico

@eguiraud : You are correct. I reviewed TTree.cxx:4524. Setting both fAutoFlush = 0 and fAutoSave = 0 is the only way to disable them. Sadly, this doesn’t solve the problem.

I’ve come up with an example that I can get to crash pretty regularly. The problem lies in TTree::Fill(). The script fails with a segmentation fault. The segmentation fault always happens when trying to call TFile::Write(). I sometimes get an error :

Warning in <TBasket::WriteBuffer>: Possible memory corruption due to compression algorithm, wrote 31963 bytes past the end of a block of 37 bytes. fNbytes=32000, fObjLen=31929, fKeylen=71

I’ve tried all combinations of options for TTree::AutoSave(). The only thing that I can find that lets me consistently run to completion is removing TTree::Fill(). I’m starting to think that TTree::Fill() isn’t a thread safe operation.

Example Script

///@file example-28337.C
///@brief A program that will generate data and asynchronously write it to disk, then crash when the TTree::Write()
/// calls TTree::AutoSave()
/// https://root-forum.cern.ch/t/disable-ttree-autosave-when-calling-ttree-fill/28337
///@author S. V. Paulauskas
///@date March 16, 2018
///@copyright Copyright (c) 2018 S. V. Paulauskas.
///@copyright All rights reserved. Released under the Creative Commons Attribution-ShareAlike 4.0 International License
#include <TFile.h>
#include <TH1D.h>
#include <TTree.h>
#include <TNtuple.h>

#include <iostream>
#include <mutex>
#include <random>
#include <thread>

struct DataStructure {
    double px;
    double py;
    double pz;
    double random;
    int i;
};

void AsyncFlush(TFile *file, TTree *tree, std::mutex *lock, unsigned int *loopId) {
    std::cout << "AsyncFlush - Now calling tree->AutoSave" << std::endl;
    tree->AutoSave();
    std::cout << "AsyncFlush - Finished calling tree->AutoSave" << std::endl;
    std::cout << "AsyncFlush - Now calling file->Write" << std::endl;
    file->Write(0, TObject::kWriteDelete);
    std::cout << "AsyncFlush - Now unlocking the file for writing in loop " << *loopId << std::endl;
    lock->unlock();
}

void WriteToDisk(TFile *file, TTree *tree, unsigned int *loopID, std::mutex *lock) {
    if (lock->try_lock()) {
        std::cout << "WriteToDisk - we're creating the new thread in loop number " << *loopID << endl;
        std::thread worker0(AsyncFlush, file, tree, lock, loopID);
        worker0.detach();
    }
}

void GenerateData() {
    std::mutex lock;
    TFile *f = new TFile("test.root", "RECREATE");
    TH1D *hist = new TH1D("hist", "", 100, -2, 2);
    TTree *tree = new TTree("tree", "");
    static DataStructure data;
    tree->Branch("data", &data, "px/D:py:pz:random:i/I");
    tree->SetAutoSave(0);
    tree->SetAutoFlush(0);

    std::default_random_engine generator;
    std::normal_distribution<double> distribution(0.0,2.0);

    unsigned int loopCounter = 0;
    while (loopCounter < 1000) {
        cout << "Working on loop number " << loopCounter << endl;
        hist->FillRandom("gaus", 10000);
        WriteToDisk(f, tree, &loopCounter, &lock);
        loopCounter++;

        for (int i = 0; i < 500; i++) {
            data.px = distribution(generator);
            data.py = distribution(generator);
            data.random = distribution(generator);
            data.pz = data.px * data.px + data.py * data.py;
            data.i = i;
            tree->Fill();
        }
    }
    while (!lock.try_lock())
        sleep(1);
    f->Write(nullptr, TObject::kWriteDelete);
    f->Close();
}

Hi,
TTree::Fill is certainly not thread-safe.
TDataFrame::Snapshot is the high-level interface to write TTrees from multiple threads,
while TBufferMerger is the low-level interface.

@eguiraud : I guess I’m confused on what you mean by “write” TTrees. In my example only one thread is writing the TTree to disk, AsyncWrite(), the other thread is filling the TTree. I’ve worked through a number of variants and find the error to occur when the baskets are written.

I reviewed the TDataFrame::Snapshot examples, but don’t see any clear method to accomplish what I’m doing. i.e. none of them seem to generate a tree and write it to disk. Do you have any examples that could help with this?

Hi,
we are moving away from the original topic of this thread, but just to be 100% clear: calling TTree::Fill concurrently on the same TTree object from multiple threads is not safe, and neither is calling TTree::Fill in one thread while you are calling TTree::AutoSave() from another. Both operations write to the TTree internal state. Pretty much no object can be used like that in ROOT 6.

There are a few more things in your snippet that do not look right at first sight (I might be missing something): for example you might never WriteToDisk the last bunch of entries (in case the previous AsyncFlush has not terminated yet) and loopCounter is not atomic but you write to it from one thread while you read it from another.

But actually, if you have access to ROOT v6.12, you don’t have to write multi-threaded code yourself for this task: TDataFrame::Snapshot is the high-level interface to write columns of a TDataFrame to a ROOT file in the form of a TTree, and it can do so by creating the TTree entries and flushing them to disk from multiple threads. It does not fill the tree in one thread and flush it from another (that is not thread-safe), but it creates chunks of TTree entries in each thread, then each thread adds these chunks (i.e. buffers) to a thread-safe writing queue, and the queue is consumed by a single “writer” worker. The end result is an important speed-up in the creation of the data-set, if the actual creation and serialization+compression of the entries is a heavy enough workload (it should be, for all but trivial cases).
The low-level details of concurrent writing of a ROOT file are actually taken care of by TBufferMerger.

Your use-case would look similar to this (modulo typos and, possibly, optimizations):

#include <ROOT/TDataFrame.hxx>
#include <random>
#include <chrono>
using namespace ROOT::Experimental; // this is where TDF lives

struct DataStructure {
    double px;
    double py;
    double pz;
    double random;
    int i;
};

void GenerateData() {
  ROOT::EnableImplicitMT(); // tell TDF to use multiple threads
  TDataFrame d(1000*500); // create a TDF with 1000*500 entries (and no columns yet)

  // now we add a column that contains your data structure
  // we need a random engine per worker thread
  const auto nWorkers = ROOT::IsImplicitMTEnabled() ? ROOT::GetImplicitMTPoolSize() : 1;
  std::vector<std::default_random_engine> generators(nWorkers);
  std::normal_distribution<double> distribution(0., 2.);
  auto makeDataStructure = [&generators, &distribution](unsigned int slot, ULong64_t entry) {
     auto &gen = generators[slot]; // each worker thread is guaranteed to receive a different slot number
     auto px = distribution(gen);
     auto py = distribution(gen);
     auto random = distribution(gen);
     auto pz = px * px + py * py;
     auto i = int(entry);
     return DataStructure{px, py, pz, random, i};
  };
  auto d2 = d.DefineSlotEntry("data", makeDataStructure, {});

  // nothing has been executed so far
  // this Snapshot call actually triggers the event loop
  const auto startTime = std::chrono::high_resolution_clock::now();
  d2.Snapshot<DataStructure>("tree", "test.root", {"data"});
  const auto endTime = std::chrono::high_resolution_clock::now();
  const auto msElapsed = std::chrono::duration_cast<std::chrono::milliseconds>(endTime - startTime).count();
  std::cout << "event loop duration: " << msElapsed << std::endl;
}

On my laptop this event loop takes about 4.5s without ROOT::EnableImplicitMT() and 2.9s with.
EDIT: the first timings were with a debug version of ROOT.
Switching to a release version and rearranging the program so that it’s compiled rather than interpreted, the event loop runs in about .6 seconds with multi-threading and 1 second without.
Increasing the number of entries by a factor 10, we are at 4.2 seconds vs 11 seconds. Not bad, considering my laptop has 2 physical cores (with 2 threads per core).

Hope this helps,
Enrico

The problem is not with AutoSave nor AutoFlush per say. The issue is that TTree::Fill might at arbitrary time ‘fill up’ the buffer associated with a branch and we currently then write this buffer immediately to the disk. So technically you would have to take the mutex around the call to Fill.

To better parallelize, you may want to consider using the TBufferMerger (See the v6.10 release notes and tutorials/multicore/mt103_fillNtupleFromMultipleThreads.C) or maybe better looking into TDF as Enrico recommended.

Cheers,
Philippe.

@eguiraud and @pcanal : Thanks for the clarification and examples! This looks like an excellent route for me to take. I’ve updated the title of the thread to cover us getting off topic. I apologize for that, these two issues were coupled in my noodle and one logically ran with the other!

I’ve found that Filling histograms is incredibly robust when writing the file from another thread. Maybe I’m just getting lucky?

I’ve also marked @eguiraud’s post as the solution to this topic as it answers the OP.

Sorta or maybe you did not notice :). Since the histogram does not ‘move/reallocate’ any of its internal during its Fill, there would never be a crash. However, what could happen (likely at a low rate) is that one of the value of the internal array is being changed while being written, in this case you might write ‘garbage’. What is more likely is that the histogram is updated during its I/O operation. So after the first bin is written but before the statistics (e.g. total number of entries) is wirtten, the histogram might be updated, consequently the written histogram might be inconsistent (the bins values cover n entries but the statistics covers n+1 entries).

Cheers,
Philippe.

I figured something like that could potentially happen. Glad to hear that the internals would be stable. Sounds like filling histograms shouldn’t cause me to lose sleep. Thank you guys again for your help!!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.