I suggest the following idea to try to find out why stockfish is so strong.
1)start with some simple program based on the code of stockfish
but simple alpha beta with no pruning and some simple evaluation(only piece square table evaluation)
compare this simple program with another program that use the same evaluation and the same search with a different data structure to see that it has a similar rating so the simple version has no special secrets.
2)Test all possible simple changes that make the code closer to the code of stockfish to see what is the most important idea to improve stockfish.
Write this idea and call it idea 1 and give estimate for rating imrovement for idea 1.
repeat the process for simple stockfish with idea 1 and get
simple stockfish+idea 1+idea 2
continue in this way and you may get a list of versions from simple to complex with elo points for every version
for example:
1)simple stockfish 2000
2)simple stockfish with stockfish hash implementation in the main search 2200
3)simple stockfish with null move pruning and hash implementation in the main search 2300
4)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation 2370
5)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation and hash in the qsearch 2430
For every change that you test you may need to play games when in the beginning you can play less games and the main rule should be simply not to have standard error that is more than 1/10 of the improvement that you measure.
I believe that if somebody care to investigate this problem and make the results public then it may help programmers to get a significant improvement in computer chess.
investigating why stockfish is strong idea
Moderators: hgm, Rebel, chrisw
-
- Posts: 10281
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: investigating why stockfish is strong idea
Good idea. Why don't you do it and show us your results?Uri Blass wrote:I suggest the following idea to try to find out why stockfish is so strong.
1)start with some simple program based on the code of stockfish
but simple alpha beta with no pruning and some simple evaluation(only piece square table evaluation)
compare this simple program with another program that use the same evaluation and the same search with a different data structure to see that it has a similar rating so the simple version has no special secrets.
2)Test all possible simple changes that make the code closer to the code of stockfish to see what is the most important idea to improve stockfish.
Write this idea and call it idea 1 and give estimate for rating imrovement for idea 1.
repeat the process for simple stockfish with idea 1 and get
simple stockfish+idea 1+idea 2
continue in this way and you may get a list of versions from simple to complex with elo points for every version
for example:
1)simple stockfish 2000
2)simple stockfish with stockfish hash implementation in the main search 2200
3)simple stockfish with null move pruning and hash implementation in the main search 2300
4)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation 2370
5)simple stockfish with null move pruning and hash implementation in the main search and mobility evaluation and hash in the qsearch 2430
For every change that you test you may need to play games when in the beginning you can play less games and the main rule should be simply not to have standard error that is more than 1/10 of the improvement that you measure.
I believe that if somebody care to investigate this problem and make the results public then it may help programmers to get a significant improvement in computer chess.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
-
- Posts: 8514
- Joined: Thu Mar 09, 2006 3:25 am
- Location: Jerusalem Israel
-
- Posts: 7216
- Joined: Mon May 27, 2013 10:31 am
Re: investigating why stockfish is strong idea
Maybe if you pay someone 3000-4000 Euro per month, perhaps someone is willing to do some research. Or find some millionaire that has nothing to do and can program and is very interested in this topic.
-
- Posts: 5228
- Joined: Thu Mar 09, 2006 9:40 am
- Full name: Vincent Lejeune
Re: investigating why stockfish is strong idea
This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests
-
- Posts: 10281
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: investigating why stockfish is strong idea
The testing system is very good to get improvementVinvin wrote:This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests
It is not good for understanding or evaluating the elo advantage of different patches.
If you want to evaluate elo advantage for different patches the only way is to test them with a big fixed number of games and not with SPRT(SPRT means that you do not know if a patch gives 1 elo improvement or 5 elo improvement) but stockfish already started from a good starting point so it is not the most interesting question and the most interesting question is what are the main factors that helped stockfish to get the good starting point and are still in stockfish.
-
- Posts: 5228
- Joined: Thu Mar 09, 2006 9:40 am
- Full name: Vincent Lejeune
Re: investigating why stockfish is strong idea
26-04-14 gl master diffUri Blass wrote:The testing system is very good to get improvementVinvin wrote:This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests
It is not good for understanding or evaluating the elo advantage of different patches.
If you want to evaluate elo advantage for different patches the only way is to test them with a big fixed number of games and not with SPRT(SPRT means that you do not know if a patch gives 1 elo improvement or 5 elo improvement) but stockfish already started from a good starting point so it is not the most interesting question and the most interesting question is what are the main factors that helped stockfish to get the good starting point and are still in stockfish.
ELO: 51.04 +-2.0 (95%) LOS: 100.0%
Total: 40000 W: 10092 L: 4258 D: 25650
40000 @ 60+0.05 th 3 3 thread regression test after "speed picking of killers"
-
- Posts: 10281
- Joined: Thu Mar 09, 2006 12:37 am
- Location: Tel-Aviv Israel
Re: investigating why stockfish is strong idea
I know that there is a big improvement and I know that they test the elo value of all the improvements.Vinvin wrote:26-04-14 gl master diffUri Blass wrote:The testing system is very good to get improvementVinvin wrote:This should be done by the dev team.
The testing system is very good to evaluate the Elo perf : http://tests.stockfishchess.org/tests
It is not good for understanding or evaluating the elo advantage of different patches.
If you want to evaluate elo advantage for different patches the only way is to test them with a big fixed number of games and not with SPRT(SPRT means that you do not know if a patch gives 1 elo improvement or 5 elo improvement) but stockfish already started from a good starting point so it is not the most interesting question and the most interesting question is what are the main factors that helped stockfish to get the good starting point and are still in stockfish.
ELO: 51.04 +-2.0 (95%) LOS: 100.0%
Total: 40000 W: 10092 L: 4258 D: 25650
40000 @ 60+0.05 th 3 3 thread regression test after "speed picking of killers"
It is not enough to know what are the main factors of the improvement. or more important than it,the main factors for stockfish strength and exactly how much to expect from adding every one of them by some order.
The idea to investigate it is to get some map for developing new chess programs like the following:
step 1:start from simple algorithm with a simple piece square table evaluation and get level of program X.
step 2:add hash tables get 200 elo
step 3:add null move pruning get additional 100 elo.
...
step 10:add passed pawn evaluation and get additional 20 elo improvement
After having this information the programmer may use it to test his program and if he does not get the elo improvement that he expects then he may understand that there is probably a bug in his code that he needs to fix or maybe his values are not good and he needs to tune them.
-
- Posts: 893
- Joined: Mon Jan 15, 2007 11:23 am
- Location: Warsza
Re: investigating why stockfish is strong idea
I'm afraid it's not so simple. I remember adding static null move, coded in very Stockfish-like manner, to my engine. First time it failed miserably. However, I repeated that attempt about a year later, when poor little Rodent was just a bit less poor - and got a small improvement. Moreover, I strongly suspect that taking this code away now would lose more Elo than initial gain from adding it. It's all about synergy: some prunings/reductions work only because engine manages to reach sufficient depth for them to work. Stockfish team probably managed to gain a lot from synergy, while reducing negative interactions between pieces of code.
Pawel Koziol
http://www.pkoziol.cal24.pl/rodent/rodent.htm
http://www.pkoziol.cal24.pl/rodent/rodent.htm
-
- Posts: 7216
- Joined: Mon May 27, 2013 10:31 am
Re: investigating why stockfish is strong idea
Maybe best is take an alfa beta or a standard minimax algorithm as a basis and test each feature in isolation.
So minimax + hash gives .. extra ELO. Minimax + futility pruning gives .. etc.
So minimax + hash gives .. extra ELO. Minimax + futility pruning gives .. etc.