investigating why stockfish is strong idea

[Uri Blass]

I suggest the following idea to try to find out why stockfish is so strong.

1)start with some simple program based on the code of stockfish
but simple alpha beta with no pruning and some simple evaluation(only piece square table evaluation)

compare this simple program with another program that use the same evaluation and the same search with a different data structure to see that it has a similar rating so the simple version has no special secrets.

2)Test all possible simple changes that make the code closer to the code of stockfish to see what is the most important idea to improve stockfish.

Write this idea and call it idea 1 and give estimate for rating imrovement for idea 1.

repeat the process for simple stockfish with idea 1 and get
simple stockfish+idea 1+idea 2

continue in this way and you may get a list of versions from simple to complex with elo points for every version

for example:
1)simple stockfish 2000
2)simple stockfish with stockfish hash implementation in the main search 2200
3)simple stockfish with null move pruning and hash implementation in the main search 2300
4)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation 2370
5)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation and hash in the qsearch 2430

For every change that you test you may need to play games when in the beginning you can play less games and the main rule should be simply not to have standard error that is more than 1/10 of the improvement that you measure.

I believe that if somebody care to investigate this problem and make the results public then it may help programmers to get a significant improvement in computer chess.

[lucasart]

I suggest the following idea to try to find out why stockfish is so strong.

1)start with some simple program based on the code of stockfish
but simple alpha beta with no pruning and some simple evaluation(only piece square table evaluation)

compare this simple program with another program that use the same evaluation and the same search with a different data structure to see that it has a similar rating so the simple version has no special secrets.

2)Test all possible simple changes that make the code closer to the code of stockfish to see what is the most important idea to improve stockfish.

Write this idea and call it idea 1 and give estimate for rating imrovement for idea 1.

repeat the process for simple stockfish with idea 1 and get
simple stockfish+idea 1+idea 2

continue in this way and you may get a list of versions from simple to complex with elo points for every version

for example:
1)simple stockfish 2000
2)simple stockfish with stockfish hash implementation in the main search 2200
3)simple stockfish with null move pruning and hash implementation in the main search 2300
4)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation 2370
5)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation and hash in the qsearch 2430

For every change that you test you may need to play games when in the beginning you can play less games and the main rule should be simply not to have standard error that is more than 1/10 of the improvement that you measure.

I believe that if somebody care to investigate this problem and make the results public then it may help programmers to get a significant improvement in computer chess.

Good idea. Why don't you do it and show us your results?

[S.Taylor]

Oh

[Henk]

Maybe if you pay someone 3000-4000 Euro per month, perhaps someone is willing to do some research. Or find some millionaire that has nothing to do and can program and is very interested in this topic.

[Vinvin]

This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests

[Uri Blass]

This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests

The testing system is very good to get improvement
It is not good for understanding or evaluating the elo advantage of different patches.

If you want to evaluate elo advantage for different patches the only way is to test them with a big fixed number of games and not with SPRT(SPRT means that you do not know if a patch gives 1 elo improvement or 5 elo improvement) but stockfish already started from a good starting point so it is not the most interesting question and the most interesting question is what are the main factors that helped stockfish to get the good starting point and are still in stockfish.

[Vinvin]

This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests

The testing system is very good to get improvement
It is not good for understanding or evaluating the elo advantage of different patches.

If you want to evaluate elo advantage for different patches the only way is to test them with a big fixed number of games and not with SPRT(SPRT means that you do not know if a patch gives 1 elo improvement or 5 elo improvement) but stockfish already started from a good starting point so it is not the most interesting question and the most interesting question is what are the main factors that helped stockfish to get the good starting point and are still in stockfish.

26-04-14 gl master diff

ELO: 51.04 +-2.0 (95%) LOS: 100.0%
Total: 40000 W: 10092 L: 4258 D: 25650

40000 @ 60+0.05 th 3 3 thread regression test after "speed picking of killers"

[Uri Blass]

This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests

The testing system is very good to get improvement
It is not good for understanding or evaluating the elo advantage of different patches.

If you want to evaluate elo advantage for different patches the only way is to test them with a big fixed number of games and not with SPRT(SPRT means that you do not know if a patch gives 1 elo improvement or 5 elo improvement) but stockfish already started from a good starting point so it is not the most interesting question and the most interesting question is what are the main factors that helped stockfish to get the good starting point and are still in stockfish.

26-04-14 gl master diff

ELO: 51.04 +-2.0 (95%) LOS: 100.0%
Total: 40000 W: 10092 L: 4258 D: 25650

40000 @ 60+0.05 th 3 3 thread regression test after "speed picking of killers"

I know that there is a big improvement and I know that they test the elo value of all the improvements.

It is not enough to know what are the main factors of the improvement. or more important than it,the main factors for stockfish strength and exactly how much to expect from adding every one of them by some order.


The idea to investigate it is to get some map for developing new chess programs like the following:

step 1:start from simple algorithm with a simple piece square table evaluation and get level of program X.
step 2:add hash tables get 200 elo
step 3:add null move pruning get additional 100 elo.
...
step 10:add passed pawn evaluation and get additional 20 elo improvement

After having this information the programmer may use it to test his program and if he does not get the elo improvement that he expects then he may understand that there is probably a bug in his code that he needs to fix or maybe his values are not good and he needs to tune them.

[PK]

I'm afraid it's not so simple. I remember adding static null move, coded in very Stockfish-like manner, to my engine. First time it failed miserably. However, I repeated that attempt about a year later, when poor little Rodent was just a bit less poor - and got a small improvement. Moreover, I strongly suspect that taking this code away now would lose more Elo than initial gain from adding it. It's all about synergy: some prunings/reductions work only because engine manages to reach sufficient depth for them to work. Stockfish team probably managed to gain a lot from synergy, while reducing negative interactions between pieces of code.

[Henk]

Maybe best is take an alfa beta or a standard minimax algorithm as a basis and test each feature in isolation.

So minimax + hash gives .. extra ELO. Minimax + futility pruning gives .. etc.