gLitePROOF on DPM+xrootd storage element

ROOT
1

Hi

Thanks to Anar and our sys admin :slight_smile:, we have setup xrootd on our DPM storage element (for local ATLAS use) and gLitePROOF. I am trying now to do some benchmarks using
root.cern.ch/twiki/bin/view/ROOT/ProofBench

{
//gEnv->SetValue(“Proof.StatsHist”,1);
//gEnv->SetValue(“Proof.StatsTrace”,1);
//gEnv->SetValue(“Proof.SlaveStatsTrace”,1);

// Creating a TDset of files to analyze
TDSet *set = new TDSet( “TTree”,“T” );

// Files to analyze
for (int i=1; i<=20; i++)
{
char ch[20];
sprintf(ch,"%d",i);
std::string hmm=“root://marsedpm.in2p3.fr//dpm/mrs.grid.cnrs.fr/home/atlas/cppm/bernardet/Event_”+string(ch)+".root";
set->Add( hmm.c_str() );
}
// connecting to local PROOF server
p = TProof::Open( “localhost” );
p->SetLogLevel(5) ;

p->UploadPackage(“event”);
p->EnablePackage(“event”);

//p->AddFeedback(“PROOF_ProcTimeHist”);
//p->AddFeedback(“PROOF_LatencyHist”);
//p->AddFeedback(“PROOF_EventsHist”);

//TDrawFeedback fb§;

//gROOT->Time();
set->Process( “EventTree_Proc.C” );
}

I use 10 workers nodes and 20 input files having 10 000 events (20 times the same file). If you look at the logfiles, proof “sees” 10 slaves but the job seems to be only run on 2-3 slaves (and look at the processing rate plot).

Thanks for any advice

Karim
xrdlog.gz (4.07 KB)


master-0-marwn01-1205411280-8128.log.gz (57.6 KB)

2

I suspec that the following is the problem:

[quote]13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::ValidateFiles: sent to slave-0.0 (localhost.localdomain) via 0x83342e0 GETENTRIES on tree root://marsedpm.in2p3.fr//dpm/mrs.grid. … ent_1.root / T
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 1
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::ValidateFiles: sent to slave-0.1 (localhost.localdomain) via 0x8335f98 GETENTRIES on tree root://marsedpm.in2p3.fr//dpm/mrs.grid. … ent_2.root / T
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
OBJ: TObject marsedpm.in2p3.fr MySlaveCount 0 SlaveCount 2
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::NextNode: Reached Slaves per Node Limit (2)
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::ValidateFiles: waiting for 2 slaves:
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::ValidateFiles: slave-0.0 (localhost.localdomain)
13:28:12 8128 Mst-0 | Info in TPacketizerAdaptive::ValidateFiles: slave-0.1 (localhost.localdomain)[/quote]
gLitePROOF setups workers on the Grid nodes, but it makes PROOF master to think, that all of his workers are on the localhost.
I could be mistaking, but I think that PROOF by default makes as many workers on the host as CPU available or something like that. This is why it doesn’t distribute packages to the other 8 workers.

3

Maybe this will help.

[quote]PROOF_MaxSlavesPerNode
Type: int Description: Parameter for the packetizers. Limit the number of slaves accessing data on any single node.
Default Value: In TPacketizer the default value is 4. In TPacketizerAdaptive and TPacketizerProgressive? it is 2.
Example: proof->SetParameter(“PROOF_MaxSlavesPerNode”, 2); [/quote]
Please try and let me know. I will also think how to automate it in gLitePROOF.

4

I have just checked carefully the nodes
in fact for one job with 10 workers, these are for example 5 different nodes :
3 slaves on node wn43
2 on node wn48
2 on node wn46
2 on wn45
1 on wn44

this can explain the xrootd logfile

5

[quote=“karim”]I have just checked carefully the nodes
in fact for one job with 10 workers, these are for example 5 different nodes :
3 slaves on node wn43
2 on node wn48
2 on node wn46
2 on wn45
1 on wn44

this can explain the xrootd logfile[/quote]
If I understood your post correctly, that would mean you are mistaking a bit.

gLitePROOF makes the PROOF server to think that ALL it’s workers are on the localhost. The PROOF server don’t actually know that real workers are on node wn48-wnXX or something.
Look what it writes: sent to slave-0.0 (localhost.localdomain).
gLitePROOF hides remote PROOF workers from the PROOF server and acts as a “proxy”.

And since default value for PROOF_MaxSlavesPerNode is 2, therefore only 2 slaves get packages. Since all slaves (for PROOF server) are on the localhost, the other 8 workers won’t get packages.

Please try to use something like that:

Just to check.

6

Just another 50 cents :slight_smile:

In my previous post I meant, that for this problem (subject), I think, it doesn’t really matter how gLitePROOF distributes workers. But it is important how this workers look like for PROOF server. And for the server they are a local workers. Just as you would start a PROOF session with 10 workers on a single machine.

7

Thanks for the precision Anar,

66834 R SL4 100.00 1.0 DE atl082 atlas marwn45 1 1:11:56:10 Fri Mar 14 17:15:28 - atl082 Karim Bernardet
66835 R SL4 99.59 1.0 DE atl082 atlas marwn44 1 1:11:56:00 Fri Mar 14 17:15:34 - atl082 Karim Bernardet
66836 R SL4 99.59 1.0 DE atl082 atlas marwn43 1 1:11:56:19 Fri Mar 14 17:15:34 - atl082 Karim Bernardet
66837 R SL4 99.59 1.0 DE atl082 atlas marwn48 1 1:11:56:17 Fri Mar 14 17:15:34 - atl082 Karim Bernardet
66838 R SL4 99.59 1.0 DE atl082 atlas marwn46 1 1:11:56:14 Fri Mar 14 17:15:34 - atl082 Karim Bernardet
66839 R SL4 100.00 1.0 DE atl082 atlas marwn43 1 1:11:56:01 Fri Mar 14 17:15:38 - atl082 Karim Bernardet
66840 R SL4 100.00 1.0 DE atl082 atlas marwn45 1 1:11:56:20 Fri Mar 14 17:15:38 - atl082 Karim Bernardet
66841 R SL4 100.00 1.0 DE atl082 atlas marwn44 1 1:11:56:04 Fri Mar 14 17:15:38 - atl082 Karim Bernardet
66842 R SL4 100.00 1.0 DE atl082 atlas marwn43 1 1:11:56:01 Fri Mar 14 17:15:38 - atl082 Karim Bernardet
66843 R SL4 100.00 1.0 DE atl082 atlas marwn48 1 1:11:56:21 Fri Mar 14 17:15:38 - atl082 Karim Bernardet

marwn452
marwn433
marwn461
marwn442
marwn48*2

{
// Creating a TDset of files to analyze
TDSet *set = new TDSet( “TTree”,“T” );

// Files to analyze
for (int i=1; i<=20; i++)
{
char ch[20];
sprintf(ch,"%d",i);
std::string hmm=“root://marsedpm.in2p3.fr//dpm/mrs.grid.cnrs.fr/home/atlas/cppm/bernardet/Event_”+string(ch)+".root";
set->Add( hmm.c_str() );
}

// connecting to local PROOF server
p = TProof::Open( “localhost” );
p->SetLogLevel(5) ;

p->SetParameter(“PROOF_MaxSlavesPerNode”, 10);
//p->SetParameter(“PROOF_Packetizer”, “TPacketizerAdaptive”);

p->UploadPackage(“event”);
p->EnablePackage(“event”);

// Processing our test analysis
set->Process( “EventTree_Proc.C” );
}

maybe I am wrong, but it seems I still have this 2 MaxSlavesPerNode limit


master-0-marwn01-1205511713-32402.log.gz (35.9 KB)

8

Please change an according code line to the following:

instead of

a ROOT specific issue.

Please try and let me know.

9

:exclamation:
BTW, I hope you remember, that for this kind of changes you shouldn’t resubmit gLitePROOF jobs again. :wink:
gLitePROOF supports reconnections and if you just need to change something in your analysis script or need to restart a local ROOT session, just make your changes, restart ROOT and connect to your old workers. :smiley: gLitePROOF will automatically manage that :wink:

10

Thanks Anar it solves the problem

I run the jobs for 100 files (10 000 events each) for a different number of slaves. It seems I reach a limit of 3000 evts/s.

Cheers

Karim



11

[quote=“karim”]Thanks Anar it solves the problem
[/quote]
Don’t mention it.
Actually I have to thank you for being a customer of the product. :slight_smile:

Is it good or bad for you? :slight_smile:
I mean, does the fact that you use gLitePROOF helps in your case or it doesn’t?

12

Of course it helps :slight_smile:

13

I am really glad to read that :unamused:
But of course most of the credits should go to ROOT/PROOF and XROOTD developers, and managers. :smiley:

BTW, please, install new version of gLitePROOF. As I told you already, I have several important fixes there. Anyway, since the subject of the topic is resolved, we could continue per email.
All the best!

14

Forgotten comments …
bandwidth between workers and DPM storage element is Gb/s.
slaves number | processing rate MB/s
10 83.3
15 98.9
20 97.8
30 98.6
35 96.6

and looking at the plot attached before (pics near 840 Mb/s), I may reach the network limit between the SE and the slaves