Change environment variables in proof

ROOT
1

Dear ROOTers,

My proof job need load some libraries spreaded in many directories, and those libraries depend on others. I can use “gSystem->SetDynamicPath” to change the path search list for “gSystem->Load(libA)”. But this does not work if libA depends on libB even libB is located within gSystem->GetDynamicpath(). To make it work in ROOT, libB has to be under a directory within the $LD_LIBRARY_PATH defined prior to ROOT start. But I do not think that we can change the parent environment variables within ROOT.

I found that there is a solution, i.e., define LD_LIBRARY_PATH in $ROOTSYS/bin/proofserv. But it may not work for my case if answer is no to my following question: Can it be configured per user or per job? And does a new proofserv start everytime a user connected to the proof server? Or is it already running and just to accept connections from users? I am not sure of the proof mechanism.

Or do you have good solution for my case?

–Shuwei

2

Hi Shuwei,

a solution is to make a libA.rootmap file containing the list of libraries on which libA depends. Then doing gSystem->Load(“libA”) will first load libB, etc. using gSystem->Load() which will search in gSystem->SetDynamicPath(). For example .rootmap files see $ROOTSYS/lib.

Cheers, Fons.

3

Hi Fons,

Thanks for your suggestion. I am not familiar with the rlibmap tool. In my case, libA depends on many libraries,which again depends on others. And there is no LinkDef.h available for me to use. It would be painful for me to manually prepare such a LinkDef.h file. We do have a overall rootmap for each project, says Project.rootmap, but with no corresponding Project.so file. Does this Project.rootmap help?

Can you help explain how PROOF works? As I know, there is a daemon running on each worker node, accepting message sent from master node. How does the daemon on worker node do after receiving a message? Does it start a new session for each message it receives? If so, can I ask the new session run a specific startup shell script before run root job? Sorry for so many questions.

–Shuwei

4

Dear Shuwei,

The daemon starts a dedicated process which is basically like a local ROOT application, except that it accepts input from a socket instead of stdin.

No, the session is started only once at beginning. During a ‘Process’, the relevant information is transferred to the master session which then takes care of distributing the work to the worker nodes and collect / merge the results.

Yes, you can run an inistialization script before starting the session. Suppose that the script full path is ‘/some/path/setmyenv.sh’ . Then you can have executed it before starting the session with

// Set the path of the script defining the special environment
TProof::AddEnvVar("PROOF_INITCMD","/some/path/setmyenv.sh")

// Start the session
p = TProof::Open("<master>")

The only current limitations is that we do not have (yet) a generic file upload facility in PROOF, so the script has to be made available on the cluster using an alternative channel.

Gerri

5

Hi Gerri,

Following your example to set initialization script file:

// Set the path of the script defining the special environment
TProof::AddEnvVar("PROOF_INITCMD","/some/path/setmyenv.sh")

// Start the session
p = TProof::Open("<master>") 

// check the setting on both master and slave nodes
p->Exec(".! echo $LD_LIBRARY_PATH",kTRUE)

Where in “/some/path/setmyenv.sh”, I define my LD_LIBRARY_PATH. Its content is something like:

#!/bin/bash
export LD_LIBRARY_PATH=path1:path2:path3:...

log=/some/path/setmyenv.log-`hostname`
touch $log
echo "LD_LIBRARY_PATH=",$LD_LIBRARY_PATH >> $log

But I did not see any change in the output of p->Exec(".! echo $LD_LIBRARY_PATH",kTRUE), and did not see the log files /some/path/setmyenv.log-* either.

Did I do something wrong? I am using ROOT-5.17.08 on client, and root-5.17.09-PROOF.00 on proof servers.

–Shuwei[/code]

6

Hi Gerri,

I have understood why “PROOF_INITCMD” does not work. The PROOF_INITCMD is evaluated in a pair of grave accent (``):

if [ -n "$PROOF_INITCMD" ]; then
   # echo "init with $PROOF_INITCMD" >> /tmp/proofserv.log
   eval `$PROOF_INITCMD`
fi

In order to make the change to environment work, it can not be executed in a new session. After I changed it into a pair of single quote and set PROOF_INITCMD to “source /some/path/setmyenv.sh”, not “/some/path/setmyenv.sh”:

// Set the path of the script defining the special environment
TProof::AddEnvVar("PROOF_INITCMD","source /some/path/setmyenv.sh")

Now it works as I expected.

There is still a related problem: how to chose a specific root version, on which I will start a new topic.

–Shuwei

7

Hi Shuwei,

The PROOF_INITCMD is evaluated, so it should contain the command to be executed to setup the environment. I realize that I did not explain well enough this.
Sorry for that.

You have basically three cases:

  1. Simple variable setting: just use ‘echo export VAR=value’, e.g.
    root [] TProof::AddEnvVar("PROOF_INITCMD", "echo export LD_LIBRARY_PATH=/some/new/libpath:$LDLIBRARY_PATH")
  1. The setup is defined by a script, e.g. /some/path/setup-env.sh: the script should be sourced:
    root [] TProof::AddEnvVar("PROOF_INITCMD","source /some/path/setup-env.sh")
This is your case.
  1. If the script outputs a (complex) command to be executed, e.g. /some/path/getscram.sh: just put the script path:
    root [] TProof::AddEnvVar("PROOF_INITCMD","/some/path/getscram.sh")
But we need to put grave accents, because we need to execute what is inside and 'eval' the output.

For what relates to ROOT version settings, you can either override ROOTSYS (and PATH, and LD_LIBRARY_PATH) in the script, or use the ‘xpd.rootsys’ directive in the xrootd configuration file to define the list of available versions (see root.cern.ch/twiki/bin/view/ROOT/XpdROOTVersion).

You can also set the version with

    root [] TProof::AddEnvVar("PROOF_INITCMD","echo export ROOTSYS=<anotherROOTSYS>; export LD_LIBRARY_PATH=$ROOTSYS/lib:$LD_LIBRARY_PATH")

where ‘anotherROOTSYS’ is the path (on the cluster machines) to the alternative ROOT distribution.

I am going to upload in SVN some documentation and improved notification in the bin/proofserv file.

Gerri

8

Hi Gerri,

Let us suppose that we have “setup-env.sh”:

export LD_LIBRARY_PATH=/some/new/libpath:$LDLIBRARY_PATH
With the grave accents, the following will fail because there is no output from the script:

eval `source /some/path/setup-env.sh`

and the environment setting of LD_LIBRARY_PATH in that script will not be propagated to the parent session.

–Shuwei

9

Hi Gerri,

I still think that it is better to use double quotation "$PROOF_INITCMD" instead of grace accent `$PROOF_INITCMD` for eval in proofserv. 

Assuming PROOF_INITCMD=/some/path/setup-env.sh" and the shell script "setup-env.sh" reads:

[quote]export VAR1=value1
export VAR2=value2
[/quote]

Using grace accent will do what expected:

eval `$PROOF_INITCMD`
The environment variables “VAR1” and “VAR2” would be undefined using grace accent.

While using double quotation accent will do what we want:

eval "$PROOF_INITCMD" 

How do you think?

–Shuwei