Opening a file in hadoop

ROOT
1

Hi,

We have a hadoop file system in our lab mounted through fuse. I cannot open a TFile in it for unknown reasons:

root [0] TFile* f = TFile::Open("/hadoop/myfile.root", "RECREATE")
SysError in <TFile::WriteBuffer>: error writing to file /hadoop/myfile.root (-1) (Input/output error)

Directly writing to it through C++ std::ofstream goes smoothly. I have used the following code and produced a 300MB file:

#include <fstream>
#include <iostream>
#include <iomanip>

using namespace std;

int main() {
  const char file[] = "/hadoop/kk";


  cout << "Opening " << file << endl; 
  ofstream os(file);

  cout << "Writing to " << file << endl;
  for (unsigned int i = 0; i < 100000000; i++) {
    if (i%1000 == 0)
      cout << "\b\b\b\b\b\b\b\b\b" << setw( 9 ) << setfill( '0' ) << i 
	   << flush;
    os << "kk" << endl;
  }

  cout << endl << "Closing " << file << endl;
  os.close();

  return 0;
}

Any hint on how to fix or otherwise debug the issue are welcome.

Isidro

2

By the way, in case some expert wants to test something we can provide access to our computing infrastructure :wink:

A hint on where to start to dig is also welcome.

3

Hi,

The ROOT hadoop file system support is readonly as hadoop does not support the seek operation in write mode needed by ROOT. To use hadoop copy ROOT file on it via scp or any other method.

4

Thanks, Fons, for your reply. Do you foresee any change on this caveat. Reading is, of course, very nice, but having the possibility to write to hadoop would be great. I know we can hack our code (CMS code, you know) to write locally and then copy to hadoop, but that is quite inconvenient some times.

Cheers,

Isidro

5

Hi Isidro,

It’s a limitation of HDFS itself. The underlying file system does not support random seeks during write, nor does it support file modification. ROOT requires random seeks in its file format, while the ostream test you did does not.

We run HDFS for our T2: the easiest thing to do is write the output file to local disk, then copy to HDFS. It’s probably a good idea for any distributed file system, as you don’t consume the shared site-wide resource until after the application has successfully completed.

Brian

6

Hi Brian,

Thanks. I undertand that is the current limitation on HDFS. Sometimes, as you know, Even if it might be better for a big cluster to save data locally and then copy the file, one has to provide extra knowledge to the users (or the tools). I mean, people will need to modify their scripts that work somewhere else to handle this limitation. And I would have to make sure they really cleanup the local space they use (ex. /tmp) which sometimes (WNs behind batch systems) might not be directly accessible to them. Nothing impossible, nothing new… still a bit annoying to me.

One of the possible solutions I was thinking was a “hack” in TFile (or probably somwhere else in the hierarchy) that whenever a ROOT file is open in a hadoop filesystem if fact creates it in /tmp. Then, when the TFile::Close() method is invoked, the file is copied to the original place. Is this reasonable to you, Fons, Brian? Or do you think it is a completely silly idea?

Cheers,

Isidro

7

Hi Isidro,

to keep this simplest I would propose to still write to a local file and then use TFile::Cp() or any other file transfer mechanism to upload the file to Hadoop. The same will be the case for our new I/O plugins to Amazon S2 and Google Storage.

Cheers, Fons.