Hi ROOTers, I would appreciate your help/advice in a topic related to performance.
I have a data acquisition system which reads data from a PCIe digitizer card (via a kernel module provided by the vendor). This program runs on a high performance PC (3990x, 64 cores and 256 GiB). The program uses the vendor’s API to copy data from the digitizer internal buffer to the computer RAM memory.
Now the problem. I did some tests (acquisition of 48 blocks of 256 MiB from the digitizer, until filling 12 GiB of the RAM) and noticed something weird. If I compile my program using “make” and then run the test, I get a data transfer speed that is always the same (48 ms copy time for blocks of 256 MiB). However, when I compile my program with “make -j6” and run the test, then I get sometimes 48ms, sometimes 53 ms, so there is a jitter. This is a big issue because I really need to stay every time below 50ms (5 GiB) so that I do not lose any data from the digitizer (which works at 2.5 GSPS, with 2 bytes per sample).
I was scratching my head on how the compilation cores might be affecting running performance at all, which sounds very weird. But then I noticed something. If I a compile using “make” and then I wait a couple of minutes before running my test, then I obtain the same jitter (48ms-53ms) than if I compile with “make -j6” and run it straight away.
So now I am speculating that, somehow, “make” is warming up one of the cores, which then is working at full speed, compared to “make -j6” that distributes the load among more cores. Does this make sense? Is there a way to solve this to make this jitter disappear (I tried with “nice”, to no avail)? Or does it maybe have to do with kernel task switching / cache / … ?
I am using a TThread for the data acquisition thread. Is there any way to “warm it up” before starting the data acquisition or give it high priority?
Maybe related: multithreading - Do C++ std::threads have a warm-up period? - Stack Overflow
Thanks in advance!
------------------------------------------------------------------
| Welcome to ROOT 6.25/01 https://root.cern |
| (c) 1995-2021, The ROOT Team; conception: R. Brun, F. Rademakers |
| Built for linuxx8664gcc on May 27 2021, 15:48:46 |
| From heads/meta_nullptr@v6-25-01-1092-gf684721d6d |
| With |
| Try '.help', '.demo', '.license', '.credits', '.quit'/'.q' |
------------------------------------------------------------------