while analyzing some positions with Houdini 3.0 multiple times, I realized that the variation of the evaluation was sometimes bigger or even much bigger than expected.
Of course, when analyzing a position with multiple Threads, the calculation is not deterministic because of the timing of the thread scheduling.
But I did not expect that the variations are sometimes very big, for example in a non-tactical position reaching from -0.05 to +0.27.
In some positions, all results are close together, except one which is completely different.
So I decided to make an investigation of this case.
I analyzed two different games (backwards) 10 times with 4 threads, a fixed depth of 20 and 1GB Hash.
I made two runs, the first with hash cleared after every move, the second with keeping the hash.
Additionally, I determined the number of different moves played in every position.
The result I put into one diagram for each game:
To the left with clearing hash, to the right without clearing the hash.
The number of different moves played is also in the diagram.
The evaluation is for the position after the move to the left.
Interpretation of the diagrams:
First diagram to the left, first position after e2e4:
Two different moves played, eval between -0.26 and -0.14, 8 different results (of 10).
I don't know if the results do mean something, if the variations are bigger than to be expected, or bigger than with other chess engines.
But because I never saw such an analysis, I decided to show it to the community, so that a discussion is possible.
Perhaps the depth of 20 is too low, and a bigger depth/time gives more identical results, in this case I will repeat it with a bigger depth.
Thomas
