Reading Large File: >6 GB

ROOT
1

Hi,

I am currently running ROOT Version 4.01 on an i686 machine with Red Hat Release 9.

I would like to read an ASCII file containing what is basically an N-tuple. The file is approx. 7 GB and the exact same code works perfectly on a similar input file of approx. 1 GB. However, it does not work with the large file, and “fopen” returns a NULL.

I have read the few previous posts regarding the use of “fopen” with large files, but cannot implement these suggestions, and most include the use of a #define _LARGEFILE_SOURCE statement with the #include<stdio.h>.

Is this solution supposed to be compatible with ROOT’s CINT??

How can I open the 7GB ASCII file and read it??

Many Thanks!
David

2

Use fopen64 instead of fopen

Rene

3

Thanks Rene,

the way I got it to work was to include

#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE

before the

#include<stdio.h>

and then to use fopen() as usual (since with this implementation, fopen becomes fopen64 automatically).

The problem was with the platform or the libraries on the machine I was running the analysis originally…changing systems solved the problem, although I still do not know why

thanks
david

4

Using

#define _LARGEFILE_SOURCE #define _LARGEFILE64_SOURCE should be used, in my opinion, only as a short term expedient.
The major problem is that now fopen but more importanly pos_t means something for your compilation unit and something totally different from another compilation unit. So it works if and only if the functions (related to the FILE) that you are calling all comes from stdio.h.

Philippe.

5

Thanks Phillipe. I had read something about the limited ability of this way around my problem in the documentation for it. Since I do only use fopen, and in fact only use it once, it works just fine for me.

Can you suggest a more general solution to the problem?

David

6

Explictily use fopen64

Philippe.

7

So I posted this a long time ago, but for some reason, my solution has stopped working. My solution was to use:

#define __LARGEFILE_SOURCE #define __LARGEFILE64_SOURCE

It may be a problem related to the bug in GCC 3.2 (discussed in http://root.cern.ch/root/roottalk/roottalk04/0100.html)

I have however, tried to use ‘fopen64’ as suggested by both Rene and Phillipe. The problem is that I get the error message

Error: Function fopen64(logfile,"r") is not defined in current scope FILE:EventTree64_4.C LINE:77

Can someone point me to how to fix this problem? Just to be clear, I use ROOT Version 4.03/04 on a dual-boot Linux OS running SLC3 and with the current GCC 3.2.3-52 that comes with it.

Thanks a lot!

8

Use open64 not fopen64

Rene

9

Unfortunately, Rene, I get the same error:

And just to be explicit, I am trying to run a named script using the “.x script.C” method.

Is there another option?

Thanks,
David

10

see example of use in TFile::SysOpen and TFile::TFile

Rene

11

Also you probably should compile your code (.x script.C+) instead of interpreting it.

Cheers,
Philippe

12

So I finally got it working.

What I did was exactly what Philippe subsequently suggested, I compiled the code with ACLiC.

Just to note, though, part of the reason this worked is that for some reason ACLiC had NO PROBLEM with

while CINT would not even recognize it as a function. As a result I did not need

#define _LARGEFILE_SOURCE #define _LARGEFILE64_SOURCE

Thanks for all of the help!
~David