Is it possible to parallelize Garfield?

Pete · October 1, 2020, 10:02pm

Hi,

I tried your suggestion and still get the 2nd error with this extra stuff:

Occasionally, I will also get the segmentation violation and the code would crash.

hschindl · October 2, 2020, 7:27am

Then I’m afraid I don’t have an immediate solution. We’ll need to make MediumMagboltz thread-safe. I’ll put it on my to-do list…

Pete · October 2, 2020, 4:37pm

Thank you for your help so far. Just out of curiosity, do you think this is something that can be resolved relatively quickly? I’m sure you have other things to take care of, so I just want to know an estimate to decide if I should wait or start to try some other things.

hschindl · October 5, 2020, 4:34pm

Hi,
sorry for the delay in getting back to you. I’ve added a guard lock in the Mixer function of the MediumMagboltz class.

Not sure if this is enough to solve the issue, but can you give it a try and let me know if it works?

Pete · October 5, 2020, 7:27pm

Hi,

Your solution seems to have fixed the errors for MediumMagboltz. However, I am still getting warnings(?) from AvalancheMicroscopic:

AvalancheMicroscopic::TransportElectron: Got collision rate <= 0 at 7.0351e-08 eV (band 0).
AvalancheMicroscopic::TransportElectron: Got collision rate <= 0 at 2.26634e-07 eV (band 0).
AvalancheMicroscopic::TransportElectron: Got collision rate <= 0 at 1.36955e-06 eV (band 0).
AvalancheMicroscopic::TransportElectron: Got collision rate <= 0 at 1.93251e-06 eV (band 0).

I get the same warnings when AvalancheMicroscopic aval is inside or outside of the for loop.

hschindl · October 5, 2020, 7:54pm

Hi,
ah, that’s probably because you give it an initial energy of (close to) zero. Can you try with something like 0.1 eV instead?
Great that the parallelized loop works now by the way!

Pete · October 5, 2020, 9:25pm

Hi,

Yes! The parallelized loop seems to work now. Thank you so much.

I changed the initial energy to 0.1-0.5, but the warnings are still there. I did notice that the warnings only show up when I try to run things in parallel. If I comment out the “#pragma omp parallel for shared(electrons)”, the warnings just go away. One other thing is that while the code is running, I noticed that at the beginning when the warnings are being printed out, multiple cores are used simultaneously. However, at some point (maybe halfway through), the warnings stop, and then only 1 core is used for the rest of the execution.

hschindl · October 6, 2020, 10:41am

Can you upload a (minimal version of) your program?

Pete · October 6, 2020, 3:11pm

Hi,

Here’s the program. I have taken out the unimportant parts.
parallel.C (6.0 KB)

hschindl · October 10, 2020, 7:43pm

Hi,
sorry for the delay… I made a couple of small modifications to the program,

initialising MediumMagboltz (i. e. calculating the cross-section table) upfront,
moving DriftLineRKF also inside the loop.

I did a quick test and it seems to run without crashing now. Can you give it a try?

parallel.C (6.0 KB)

Pete · October 11, 2020, 4:12am

Hi,

Thank you so much! You seem to have fixed the issues. I have run the program several times with different kinetic energy and so far have not any of the previous errors/warnings. I also noticed that all threads were used throughout the execution unlike previously!

I have just one more follow up question. With the way driftline and aval are being defined inside of the for loop, how should I call for the ViewSignal, ViewCell, and ViewDrift?

hschindl · October 11, 2020, 1:28pm

Hi,
for ViewSignal and ViewCell you shouldn’t need to do anything special.
To plot the electron and ion drift lines, you would need to create a ViewDrift object outside the loop, and then switch on plotting for each of the AvalancheMicroscopic and DriftLineRKF objects inside the loop, something like this

ViewDrift driftView;
// ...
#pragma omp parallel for
for (int k = 0; k < index; k++) {
  AvalancheMicroscopic aval;
  aval.SetSensor(&sensor);
  aval.EnableSignalCalculation();
  aval.EnablePlotting(&driftView);
  // ...
  DriftLineRKF drift;
  drift.SetSensor(&sensor);
  drift.EnableSignalCalculation();
  drift.EnablePlotting(&driftView);
}

driftView.Plot();

Plotting the drift lines can be quite memory-hungry though and might slow down your program a lot, so I would only do it for debugging or illustration purposes.

PS: I noticed that you are simulating a 100 TeV proton. Out of curiosity: what’s the application?

Pete · October 11, 2020, 8:50pm

Hi,

I am not planning to plot the program every time. So, as you mentioned, I usually plot when I need a “picture” for presentation or to make sure that I placed the track where I actually want. I tried what you suggested and everything seems fine for plotting the drift lines.

Regarding the signals, I am still having some issues. Currently, this is how I am plotting the signals:

// Plot the induced current.
  ViewSignal signalView_left;
  signalView_left.SetSensor(&sensor);
  TCanvas c1("c1", "", 800, 600);
  signalView_left.SetCanvas(&c1);
  signalView_left.SetLabelY("left anode raw [fC]");
  signalView_left.PlotSignal("al");

  ViewSignal signalView_right;
  signalView_right.SetSensor(&sensor);
  TCanvas c2("c2", "", 800, 600);
  signalView_right.SetCanvas(&c2);
  signalView_right.SetLabelY("right anode raw [fC]");
  signalView_right.PlotSignal("ar");

  // Convolute with the transfer function and plot again.
  sensor.SetTransferFunction(transfer);
  constexpr bool fftl = true;
  sensor.ConvoluteSignals(fftl);
  TCanvas c7("c7", "", 800, 600);
  signalView_left.SetCanvas(&c7);
  signalView_left.SetLabelY("convolute signal left [mV]");
  signalView_left.PlotSignal("al");

  sensor.SetTransferFunction(transfer);
  constexpr bool fftr = true;
  sensor.ConvoluteSignals(fftr);
  TCanvas c8("c8", "", 800, 600);
  signalView_right.SetCanvas(&c8);
  signalView_right.SetLabelY("convolute signal right [mV]");
  signalView_right.PlotSignal("ar");

For some reason, the plots are just showing up empty (i.e. so signals shown inside of graphs).

Also, I am not using 100 TeV for my application. I set the kinetic energy that high so that only a few electrons would get created, which speed up the execution. For what I am doing (charge particle detection), the particles usually have energy in the range of MeV.

Pete · October 12, 2020, 1:14am

This is the warnings that I get when I try to plot:

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c1 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c1 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c2 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c2 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c7 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c7 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c8 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c1 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c2 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c7 height changed from 0 to 10

Warning in <TCanvas::ResizePad>: Inf/NaN propagated to the pad. Check drawn objects.
Warning in <TCanvas::ResizePad>: c8 height changed from 0 to 10

couet · October 12, 2020, 1:11pm

You data contain NaN and/or Inf

hschindl · October 12, 2020, 1:22pm

I’ll take a look…

hschindl · October 14, 2020, 9:25pm

Sorry again for the delay… I’ve added a few more guard locks. Can you git pull and give it another try?

Pete · October 14, 2020, 10:33pm

Hi,

No need to be sorry about the delay. Thank you for helping me with this!

I did a git pull, then cmake garfield again. So far (I have only run the code once), sensor does not output nan results anymore, and I am actually getting signals now. I will run the code multiple times and make sure, but I think you fixed it!

hschindl · October 15, 2020, 8:20am

Great! I’m sure there are still many places in the code that are not thread safe, so if you encounter any issues, please shout…

system · October 29, 2020, 8:20am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.