Hello again experts,
I have a question regarding read speeds/disk usage when using RDataFrame in python. Apologies ahead of time that this is quite technical. I’m hoping to hear about any similar experiences others may have had as opposed to expecting a specific solution
I have an 8 core/16 thread modern desktop CPU that I’ve been running RDataFrame jobs on. I’ve copied CMS NanoAOD data and simulation sets to the machine on a 6 TB HDD that runs at 5400 RPM with a cache of (I think) 256 MB, connected via SATA (6Gbps), and formatted as NTFS. I read from this drive and write histograms, snapshots, etc to an SSD.
If I benchmark the HDD, I get speeds of ~180 MB/s and if I preform a real reading task (copy from the drive to another), I get 80-100 MB/s.
I’ve been having some unexpected speed issues when reading large groups of NanoAOD files from this HDD where the allotted CPU threads jump to 100% and then after 15-30 seconds fall to 0-50% of their processing capacity for the remainder of the run (so, not a CPU bottleneck). When viewing these in htop, I can see that the status of each sub-process is switching between ‘R’ (running) and ‘D’ (disk sleep) with sometimes all of the process sitting in the ‘D’ state for a second or two. Viewing iotop shows that the actual read speeds are only 5-10 MB/s. This value will also go lower depending upon which set I’m reading (haven’t figured out a correlation there just yet).
I’ve tried using fewer and more threads and running concurrent jobs (on different simulation sets but on the same drive) but the highest read speeds I’ve reached are around 10 MB/s.
I also moved one set (~370 GB total) to a different HDD (1TB, 7200 RPM, 64MB cache, Ext4) and have the same experience. If I run that concurrently with a set on the 6 TB drive, I can get total read speeds of ~15 MB/s.
So I believe the drives are healthy and I know the CPU has more processing power to use but I’m not sure what’s causing the slowdown. I plan on moving some files to an SSD for testing (just need to make some room ). Has anyone had any similar experiences though? Maybe this is a python issue?
Thanks!
Lucas
ROOT Version: v6.20
Platform: Ubuntu 18.04 LTS
Compiler: Not Provided