Hello all,
We are experiencing some problems when our analysis framework have to copy a huge amount of data to the proof workers (time out basically: already at 2000s > 30’!). I am wondering if there is some “DataUploader” scripts/routine/macro/… that can do the job of distributing new data among PROOF workers wisely without the need of the whole an analysis framework.
Thanks for any hint.
Cheers,
Jordi
Hi,
I am not sure what you mean by ‘new data’: are these the data to be analysed or data used to set up your code?
Data to be analysed should be downloaded on the machines using a system like xrootd, which can be setup to ‘stage’ (i.e. bring-in) files from any remote back-end via dedicated scripts.
The best way to upload ‘data’ used by the code is to encapsulated them into a PAR package and upload the package. At run time you can use the TProof::AddInputData facility to have big objects distributed via file to the machines and available in the selector via the input list.
Check root.cern.ch/drupal/content/cont … putObjects .
G. Ganis