Weights update in TMultiLayerPerceptron

ROOT
1

Hi,

Since the initial values of the weights between neurons affect the training results, in order to get the same result everytime, I use the following code to get the initial weights (saved in “weighti.txt”),

TMultiLayerPerceptron *mlp = new TMultiLayerPerceptron(“x1,x2:7:type”,
simu,&train,&test);
mlp->Randomize();
mlp->DumpWeights(“weighti.txt”);
mlp->Train(ntrain, “+,text,graph,update=10”);
mlp->DumpWeights(“weightf.txt”);

Then I want to repeat the result, so I load the previous initial weights and run the following code

TMultiLayerPerceptron *mlp = new TMultiLayerPerceptron(“x1,x2:7:type”,
simu,&train,&test);
mlp->LoadWeights(“weighti.txt”);
mlp->Train(ntrain, “+,text,graph,update=10”);
mlp->DumpWeights(“weightf1.txt”);

but what I found is that the weights after training in the two files weightf.txt and weightf1.txt are different, Is this reasonable?

What else will affect the training results if the initial weights are fixed, there is no change on the dataset or net structure?

Thanks a lot
Yu

2

Hi,

I’am surprised, since I would expect your procedure to work fine.

Can you give me which version of ROOT you are using ?

Which training method are you using ? If you use the simple stochastic method, then the result may differ because events in the TTree are first shuffled.

Could you also make the following tests ?

  • try with ntrain=0
  • try to dump the weights without training the network (randomize, dump, load, dump again and compare with the first dump).

I’ll try to reproduce the problem on my side with the tutorial, and let you know when I have results (but I’m rather busy this week).

Cheers,
Christophe.

3

Hi, Thank you very much for your reply.

The root version I used is 4.00.04, and the default training method(kBFGS?).

I did what you suggested, when ntrain=0, the two files with dumped final weights are exactly the same. When I try ntrain=10, they are slightly different, but for ntrain=1000, the difference are increased, as I shown in the attachment.

I also did the same procedure by using mlpHiggs.C in the tutorial and found the same problem.

Thanks a lot
Yu
weightf1.txt (298 Bytes)
weightf.txt (299 Bytes)

4

Hello,

I cannot reproduce the problem. I attached the modified mlpHiggs.C that I used, as well as the output files. As you can see, the weights are strictly the same.

I’m using the HEAD version of ROOT, so I can only suggest you to switch to a newer version (even if I’m not aware of modifications that may have influenced this).

Cheers,
Christophe.
weights.tar (10 KB)
mlpHiggs.C (4.65 KB)

5

Hi, Thank you very much, from your code I know how to fix my problem,
The weights should be loaded right after the first dump like this

mlp->Randomize();
mlp->DumpWeights(“weighti.txt”);
mlp->LoadWeights(“weighti.txt”); //
mlp->Train(ntrain, “+,text,graph,update=10”);
mlp->DumpWeights(“weightf.txt”);

Thanks again,
Yu