Loading data from CSV file and performing FFT on it

[Tardec1234]

So what I would like to do is the following:
I have a CSV file that contains 15 columns and some $10^{6}$ lines of data. Here is a sample of the data:

1,-1,112,0,599,8,128,-981,433,-459,880,-901,682,-913,506
2,-1,107,0,611,8,136,-981,410,-459,882,-901,682,-913,506
3,-1,107,0,612,8,142,-981,378,-459,887,-901,734,-913,519
4,-1,111,0,608,8,142,-981,106,-459,887,-901,744,-913,533
5,-1,113,0,601,8,141,-981,106,-459,867,-901,730,-913,533
6,-1,112,0,601,8,141,-981,234,-459,855,-901,692,-913,509
7,-1,108,0,600,8,148,-981,350,-459,856,-901,692,-913,506
8,-1,108,0,599,8,158,-981,350,-459,869,-901,656,-913,486
9,-1,108,0,601,8,158,-981,925,-459,869,-901,639,-913,443
10,-1,109,0,608,8,137,-981,957,-459,886,-901,642,-913,443
11,-1,109,0,608,8,132,-981,941,-459,907,-901,662,-913,484

I would like to load all the odd numbered columns (1,3,…) into vectors and then pass those vectors to the FFT algorithm in order to draw some conclusions from that.

What I have done to achieve this is the following:

• I have found the tutorial https://root.cern.ch/root/html/tutorials/tree/basic.C.html which describes how to load data from an ASCII file to an ntuple. However, I am uncertain as to how to modify this code to read from the CSV file that I have. I think a modification (maybe replacing the space with a comma?) should be brought to the lines of code:

dir.ReplaceAll("basic.C",""); dir.ReplaceAll("/./","/")

Is that correct?

• I have found the tutorial https://root.cern.ch/root/html/tutorials/fft/FFT.C.html which describes how to use the FFT algorithm for data arrays in the second part. However, I would like to modify this in order to have on one graph the real, imaginary and complex modulus of the discrete fourier transform of the input data. Again, I am unsure as to how should I go about this.

Thanks

[moneta]

Hi,

You have few possibilities to import CSV files in ROOT. Then one you suggest is fine.
however I think it is more convenient , in case of large files to import them in a TTree using the function TTree::ReadFile.
Once you have in a TTRee you can extract the arrays using the functions GetV1, GetV2 or GetVal(int) of Tree

The second convenient possibilities is for example to use Python and the CSV package. Once you have in Python you can transfer to ROOT and use the FFT via PyROOT

Then once you have in an array you can use directly the FFT as in the tutorial. You can see the FFT documentation on https://root.cern.ch/doc/master/classTVirtualFFT.html

Lorenzo

[Tardec1234]

Thank you for your reply. I have managed to write this so far:

#include "TH1D.h"
#include "TVirtualFFT.h"
#include "TF1.h"
#include "TCanvas.h"
#include "TMath.h"
 #include "Riostream.h"
#include "TString.h"
#include "TFile.h"
#include "TTree.h"
#include "TSystem.h"

void DA()
{
    // First I want to load the data from the CSV file to the TTree. 
   
    //Create TTree
    TTree *tree = new TTree("data", "Magnetic field readings");
    //Create branches in TTree using ReadFile
    tree->ReadFile("/Home/tarabostes-delectus/build/PicoLog data/basic.csv", "" ,",");
    
    
    //Prepare the canvas for drawing
   TCanvas *myc = new TCanvas("myc", "Fast Fourier Transform", 800, 600);
   myc->SetFillColor(45);
   TPad *c1_1 = new TPad("c1_1", "c1_1",0.01,0.67,0.49,0.99);
   TPad *c1_2 = new TPad("c1_2", "c1_2",0.51,0.67,0.99,0.99);
   TPad *c1_3 = new TPad("c1_3", "c1_3",0.01,0.34,0.49,0.65);
   TPad *c1_4 = new TPad("c1_4", "c1_4",0.51,0.34,0.99,0.65);
   TPad *c1_5 = new TPad("c1_5", "c1_5",0.01,0.01,0.49,0.32);
   TPad *c1_6 = new TPad("c1_6", "c1_6",0.51,0.01,0.99,0.32);
   c1_1->Draw();
   c1_2->Draw();
   c1_3->Draw();
   c1_4->Draw();
   c1_5->Draw();
   c1_6->Draw();
   c1_1->SetFillColor(30);
   c1_1->SetFrameFillColor(42);
   c1_2->SetFillColor(30);
   c1_2->SetFrameFillColor(42);
   c1_3->SetFillColor(30);
   c1_3->SetFrameFillColor(42);
   c1_4->SetFillColor(30);
   c1_4->SetFrameFillColor(42);
   c1_5->SetFillColor(30);
   c1_5->SetFrameFillColor(42);
   c1_6->SetFillColor(30);
   c1_6->SetFrameFillColor(42);

   c1_1->cd();
   TH1::AddDirectory(kFALSE);
   
   //Define variables needed for FFT algorithm. 
   //Put in vector in the values of one Branch from the tree
     Int_t n=25;
     Double_t *in = new Double_t[2*((n+1)/2+1)];
     Double_t re_2,im_2;
    

   //Make our own TVirtualFFT object (using option "K")
   //Third parameter (option) consists of 3 parts:
   //-transform type:
   // real input/complex output in our case
   //-transform flag:
   // the amount of time spent in planning
   // the transform (see TVirtualFFT class description)
   //-to create a new TVirtualFFT object (option "K") or use the global (default)
   Int_t n_size = n+1;
   TVirtualFFT *fft_own = TVirtualFFT::FFT(1, &n_size, "R2C ES K");
   if (!fft_own) return;
   fft_own->SetPoints(in);
   fft_own->Transform();

   //Copy all the output points:
   fft_own->GetPoints(in);
   //Draw the real part of the output
   c1_5->cd();
   TH1 *hr = 0;
   hr = TH1::TransformHisto(fft_own, hr, "RE");
   hr->SetTitle("Real part of the 3rd (array) tranfsorm");
   hr->Draw();
   hr->SetStats(kFALSE);
   hr->GetXaxis()->SetLabelSize(0.05);
   hr->GetYaxis()->SetLabelSize(0.05);
   c1_6->cd();
   TH1 *him = 0;
   him = TH1::TransformHisto(fft_own, him, "IM");
   him->SetTitle("Im. part of the 3rd (array) transform");
   him->Draw();
   him->SetStats(kFALSE);
   him->GetXaxis()->SetLabelSize(0.05);
   him->GetYaxis()->SetLabelSize(0.05);

   myc->cd();
   
      TH1 *hm =0;
   TVirtualFFT::SetTransform(0);
   hm = hsin->FFT(hm, "MAG");
   hm->SetTitle("Magnitude of the 1st transform");
   hm->Draw();
}

Ofcourse, it does not work. And the errors that I get when running this are:

In file included from input_line_24:1:                                                                                     
/home/tarabostes-delectus/build/macros/DA.C:19:82: error: cannot initialize a parameter of type 'char' with an lvalue of type 'const char [2]'                                                                                                        
    tree->ReadFile("/Home/tarabostes-delectus/build/PicoLog data/basic.csv", "" ,",");                                     
                                                                                 ^~~                                       
/home/tarabostes-delectus/build/include/TTree.h:488:99: note: passing argument to parameter 'delimiter' here               
   virtual Long64_t        ReadFile(const char* filename, const char* branchDescriptor = "", char delimiter = ' ');        
                                                                                                  ^
In file included from input_line_24:1:
/home/tarabostes-delectus/build/macros/DA.C:98:9: error: use of undeclared identifier 'hsin'
   hm = hsin->FFT(hm, "MAG");

I think the last error is because I just copy-pasted some parts of the FFT.C tutorial (and deleted some parts of it) so that hsin does not exist anymore. Also, my csv file has the first three rows in the following format:

Time,B_x ,B_y,B_z,Thermometer LMTZ70,B0_x,B0_y,B0_z
$\Delta$Seconds, ( mV ), ( mV ), ( mV ), ( mV ), ( mV ), ( mV ), ( mV )
$\Delta$0, -1, 114, 0, 559, 8 ,128, -981, 486, -459, 990, -901, 682, -913, 506

Do you think the first error has something to do with this? Should I try and put a # sign in front of these first few lines in the csv file? Is there some way to instruct ROOT to pick up the column corresponding to a particular name ( like load the values corresponding to B_x into branch called Bx)? Also, how do I copy a branch from a tree to a vector so that I can pass it to the FFT algorithm?

Thanks

[behrenhoff]

Your first error is very simple and the error message is even trying to tell you what is wrong:
The third parameter of ReadFile is of type char but you are passing a string (to be precise a const char array of length 2) instead (a string with one character in it + the \0 at the end)!

Use single quotes to pass a char, i.e. ',' instead of ",".

[Tardec1234]

Thank you for your answer. I have modified the code as you suggested, ended up with even more errors and decided to take things one step at a time.

Thus, I decided to focus on getting the data from the csv file into a tree and then copying specific branches into ntuples. After reading the reference class for TTree, I have the following question: Is there some syntax that I can pass to the branch descriptor of the TTree:ReadFile command so that branches are created only for odd numbered columns from the csv file? Would something like x::y skip the second column?

I want this because the description by which the tree is made, which is found on the first line of the csv file, does not correspond to the rest of the data. There are more columns of data than the description says, and thus the command ignores a chunk of the data.

[behrenhoff]

Wouldn’t it be easier to fix the input file?

To remove the incorrect header and to extract only odd columns (I assume first col is 1, not 0, and thus odd) from the rest of the file you could use this quick&dirty script:
tail -n +2 infile.csv | perl -nE'chomp;@c=split/,/;say join ",", @c[grep !($_&1), 0..$#c]' > outfile.csv

If you need specific cols (not all odd cols independent of the number of columns), you can simply use cut -d, -f 1,3,5,7,9,11,13,15 < infile.csv > outfile.csv

As soon as your input csv file is correct, you should be able to use TTree::ReadFile.

[Tardec1234]

Indeed it would be. While waiting for the answer I managed to load the columns into the tree by specifying the following syntax to the branch descriptor

"Time::Bx::By::Bz::T::B0x::B0y::B0z"

In this way it created branches for all the columns of the csv file and named the odd numbered ones which were of interest to me. I will also try and use the command suggested by you in order to eliminate the unnecessary branches.

I also tried creating ntuples from the branches of interest and making sure that the data was indeed loaded in them. An example of the syntax I used is:

Double_t *Time = tree->GetVal(0);
Time->Print();

This resulted in this error: member reference base type 'Double_t' (aka 'double') is not a structure or union Time->Print(); ~~~~^ ~~~~~
I do not know how to interpret the error. I assumed that maybe I was using the wrong command for displaying an ntuple. Thereafter I tried to use the command:

TGraph *gs = new TGraph(n, Time, Bx);

which resulted in the error:

Error in <TGraphPainter::PaintGraph>: illegal number of points (0)

I have been unable to find another command for creating ntuples from specific branches of a tree. Are you aware of such a command?

Also, is it possible to go around this problem and pass directly tree branches directly to the FFT algorithm?

Thank you