Currently I work on some astronomy .fits file to do some analysis. I wrote a macro to read the HDU from the .fits files. But when I access a HDU for a very big data file (around 4.8 Gb), then nothing happens. I have my code below.
void galaxy_pair()
{
TString dir = gSystem->DirName(gInterpreter->GetCurrentMacroName());
TFITSHDU *hdu_locate = new TFITSHDU("DR7_QSO_ALL_INDEX.fit[1]");
[b]TFITSHDU *hdu_spec = new TFITSHDU("DR7_QSO_ALL_CONTINUUM_106.fit[1]");[/b]
if (hdu_locate == 0 || hdu_spec == 0) {
cout << "ERROR: could not access the HDU" << endl;
return;
}
hdu_locate->Print("F+");
hdu_spec->Print("F+");
TVectorD *RA1 = hdu_locate->GetTabRealVectorColumn("RA");
TVectorD *DEC1 = hdu_locate->GetTabRealVectorColumn("DEC");
long RA_size = RA1->GetNoElements();
cout << "size " << RA_size << " " << endl;
std::vector<double> RA;
std::vector<double> DEC;
for (int i = 0; i < RA_size; i++) {
RA.push_back((*RA1)[i]);
DEC.push_back((*DEC1)[i]);
}
delete RA1;
delete DEC1;
delete hdu_locate;
delete hdu_spec;
}
The problem is:
if I do not add any information of this HDU hdu_spec, then it runs very well. If I have it in the code, like the one I post here, then it will run, no error message will pop up and nothing happens. What is the problem with the code or the file? I will have a snapshot of the HDU in the attachment.
Hi,
I’m the author of the TFITSIO module. Your code is pretty simple and indeed it seems there’s something wrong with the module. As of NASA’s CFITSIO library’s documentation, in “CFITSIO Size Limitations” chapter, (https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node32.html):
You are trying to load a huge file. Please, could you check if your installed version of CFITSIO library was compiled using the flags described above?
Regards
EDIT:
The TFITSIO module uses 4-byte signed integers to index data. This may be an issue too if the huge file size is due to many rows/columns of data rather than due to cells’ content data size.
Thank you very much for your help. I already checked the CFITSIO Makefile, it does contain those two flags.
-D_LARGEFILE_SOURCE=1 -D_FILE_OFFSET_BITS=64
if I access the first HDU of the huge data file, which contains nothing. Then it will run. What can I do to access the huge data HDU? Thank you very much.
TFITSHDU *hdu_spectrum = new TFITSHDU(“DR7_QSO_ALL_CONTINUUM_106.fit[]”);
Total: 2 HDUs
[0] IMAGE (PRIMARY)
SIMPLE = T / Dummy Created by MWRFITS v1.8
BITPIX = 8 / Dummy primary header created by MWRFITS
NAXIS = 0 / No data is associated with this header
EXTEND = T / Extensions may (will!) be present
Total: 2 HDUs
[0] IMAGE (PRIMARY)
SIMPLE = T / Dummy Created by MWRFITS v1.8
BITPIX = 8 / Dummy primary header created by MWRFITS
NAXIS = 0 / No data is associated with this header
EXTEND = T / Extensions may (will!) be present
As I suspected, the problem is how the TFITS module indexes data. Internal variables are 4-byte signed integers and therefore we are surely having overflow issues. Your data has 47460 x 107194 = 5087427240 cells, which obviously exceeds the 4-byte boundary.
Provided that you have the huge file to test with, could you try to review the code in order to replace the needed variables’ types to the appropriate 64-bit ones?
Thank you very much. But it seems I do not have the freedom to set the returned variables types. Like TFITSHDU, this module opens the HDU, nothing can be changed.
TVectorD *GetArrayRow(UInt_t row), this one returns the rows as a vector, can be changed to be TVectorD *GetArrayRow(UInt_t row), but this is not the case we want.
I meant modifying ROOT’s source code. Haven’t you compiled ROOT on your computer? Then you can access ROOT’s source code and recompile it. TFITS sources are located under graf2d/fitsio directory. You should update two files:
graf2d/fitsio/src/TFITS.cxx
graf2d/fitsio/inc/TFITS.h
After updating these files you must recompile ROOT as you probably did and then test your code again.
I can do these changes and send you the files, if you prefer, but I thought it’d be faster you to make the changes and tests.
Hi Huanian,
I’ve made some changes to TFITS.cxx. Please backup your current file and replace it by the one attached to this post. Then recompile ROOT and run your code again.
I haven’t compiled nor tested these changes, so feel free to review them (by comparing the old and new versions of TFITS.cxx) and tell me if you encounter any trouble.
