One user killed his proof jobs at his client side, then I restarted proofd daemon on all machines, and then removed everything under the directory /localdisk/proofbox/username/* at all WNs. Then the user resubmit the job again, he sees that his jobs processing speed turns to be very strange, suddently jumpted from 0% to ~90%, then starts to go on slowly.
Any idea what caused this? Something on the machines left uncleaned or ?
By default, killing the client terminates the server side associated processes only after the ongoing processing is finished. So, my hypothesis is that the master was still doing something, so when the client reconnected it assumed it was just coming back.
Removing the directories will results in errors inside the processes, but not automatically terminate the processes.
In this cases it may help to run a full Reset:
root [] TProof::Reset("<master>", kTRUE);
(replace with the URL of the master).
This should force termination of all processes associated with the client. Please allow for a couple of minutes after Reset is done for the dameon to finish gracefully the clean-up.