Page 2 of 7
Re: Chessprogams with the most chessknowing
Posted: Fri Dec 19, 2014 11:06 pm
by bob
Adam Hair wrote:Laskos wrote:Laskos wrote:Henryval wrote:Which Chessprogram(s) have the most implemented chessknowing. And are there more than a few proofs out there in the chesscommunity?
I wonder why we don't have any ranking about this issue.
I am not sure how to define this. Fixed depth play at depth=1 shows Komodo as the most knowledgeable engine. Generally, modern engines seem to have better knowledge than older ones.
I did some fast testing with fixed depth=1 for several top modern engines and engines reputed to have much knowledge. I included the ancient Shredder 6PB (year 2002 IIRC), which also claimed to have lots of knowledge at that time.
RR in cutechess-cli:
Code: Select all
Rank Name ELO Games Score Draws
1 Komodo 8 92 1000 63% 18%
2 Houdini 4 78 1000 61% 26%
3 Hannibal 1.4 56 1000 58% 23%
4 SF 14122014 43 1000 56% 20%
5 Hiarcs 14 -22 1000 47% 16%
6 Shredder 6PB (2002) -302 1000 15% 14%
Finished match
Modern engines seem to prevail, especially those whose eval passed at some time through Larry's hands.
I think that at least part of the difference is due to Shredder not searching as many nodes as Komodo and Houdini.
I tried the same experiment this afternoon, primarily because I have been thinking about how to measure which engines handle various material imbalances better. Here are my results:
Code: Select all
# PLAYER : RATING POINTS PLAYED (%)
1 Houdini 4 : 78.0 1865.5 3029 61.6%
2 Equinox 3.3 : 76.7 1862.0 3040 61.2%
3 Komodo 8 : 52.1 1727.0 3000 57.6%
4 Critter 1.6a : 37.9 1671.0 3000 55.7%
5 Gull 3 : 18.9 1589.5 3040 52.3%
6 Stockfish 141213 : 14.8 1564.0 3000 52.1%
7 Gaviota 1.0 : -89.3 1093.0 3000 36.4%
8 Fire 4 : -189.1 697.0 3029 23.0%
Then I examined what Houdini and Fire do when made to search 1 ply. Houdini searches many more nodes on average. Here is an example:
Houdini
Code: Select all
1419023568.383 GUI->Adapter: position startpos moves e2e4 e7e5 g1f3 g8f6 d2d4 f6e4 f1d3 d7d5 f3e5 b8d7 e5d7 c8d7 e1g1 f8d6 c2c4 c7c6 c4d5 c6d5 b1c3 e4c3 b2c3 e8g8 d1h5 f7f5 h5f3 g8h8 c1d2 d8h4 g2g3 h4g4 g1g2 a8c8 a1b1 b7b6 f1e1 f5f4 f3g4 d7g4 a2a4 g7g6 g2g1 g4d7 d3a6 c8b8 a6b5 b8d8 b5d7 d8d7 b1b5 d6c7 d2f4 c7f4 g3f4 f8f4 e1e5
1419023568.383 Adapter->Engine: position startpos moves e2e4 e7e5 g1f3 g8f6 d2d4 f6e4 f1d3 d7d5 f3e5 b8d7 e5d7 c8d7 e1g1 f8d6 c2c4 c7c6 c4d5 c6d5 b1c3 e4c3 b2c3 e8g8 d1h5 f7f5 h5f3 g8h8 c1d2 d8h4 g2g3 h4g4 g1g2 a8c8 a1b1 b7b6 f1e1 f5f4 f3g4 d7g4 a2a4 g7g6 g2g1 g4d7 d3a6 c8b8 a6b5 b8d8 b5d7 d8d7 b1b5 d6c7 d2f4 c7f4 g3f4 f8f4 e1e5
1419023568.383 GUI->Adapter: go depth 1
1419023568.383 Adapter->Engine: go depth 1
1419023568.393 Engine->Adapter: info multipv 1 depth 1 seldepth 8 score cp 8 time 0 nodes 565 nps 0 tbhits 0 hashfull 0 pv d7c7
1419023568.393 Adapter->GUI: info multipv 1 depth 1 seldepth 8 score cp 8 time 0 nodes 565 nps 0 tbhits 0 hashfull 0 pv d7c7
1419023568.393 Engine->Adapter: bestmove d7c7 ponder 0000
1419023568.393 Adapter->GUI: bestmove d7c7 ponder 0000
Fire
Code: Select all
1419023696.997 GUI->Adapter: position startpos moves e2e4 e7e5 g1f3 g8f6 d2d4 f6e4 f1d3 d7d5 f3e5 b8d7 e5d7 c8d7 e1g1 f8d6 c2c4 c7c6 c4d5 c6d5 b1c3 e4c3 b2c3 e8g8 d1h5 f7f5 h5f3 g8h8 c1d2 d8h4 g2g3 h4g4 g1g2 a8c8 a1b1 b7b6 f1e1 f5f4 f3g4 d7g4 a2a4 g7g6 g2g1 g4d7 d3a6 c8b8 a6b5 b8d8 b5d7 d8d7 b1b5 d6c7 d2f4 c7f4 g3f4 f8f4 e1e5
1419023696.997 Adapter->Engine: position startpos moves e2e4 e7e5 g1f3 g8f6 d2d4 f6e4 f1d3 d7d5 f3e5 b8d7 e5d7 c8d7 e1g1 f8d6 c2c4 c7c6 c4d5 c6d5 b1c3 e4c3 b2c3 e8g8 d1h5 f7f5 h5f3 g8h8 c1d2 d8h4 g2g3 h4g4 g1g2 a8c8 a1b1 b7b6 f1e1 f5f4 f3g4 d7g4 a2a4 g7g6 g2g1 g4d7 d3a6 c8b8 a6b5 b8d8 b5d7 d8d7 b1b5 d6c7 d2f4 c7f4 g3f4 f8f4 e1e5
1419023696.997 GUI->Adapter: go depth 1
1419023696.997 Adapter->Engine: go depth 1
1419023696.997 Engine->Adapter: info time 1 nodes 0 nps 0 tbhits 0 depth 1 score cp 0 pv h8g8
1419023696.997 Adapter->GUI: info time 1 nodes 0 nps 0 tbhits 0 depth 1 score cp 0 pv h8g8
1419023696.997 Engine->Adapter: info time 1 nodes 0 nps 0 tbhits 0 depth 1 score cp 4 pv h8g7
1419023696.997 Adapter->GUI: info time 1 nodes 0 nps 0 tbhits 0 depth 1 score cp 4 pv h8g7
1419023696.998 Engine->Adapter: info time 1 nodes 41 nps 41000 tbhits 0 depth 1 score cp 8 pv f4f3
1419023696.998 Adapter->GUI: info time 1 nodes 41 nps 41000 tbhits 0 depth 1 score cp 8 pv f4f3
1419023696.998 Engine->Adapter: bestmove f4f3
1419023696.998 Adapter->GUI: bestmove f4f3
I suspect that part of the reason that Shredder is scoring much worse in your test is because it is not searching as many nodes.
Searching more or less nodes is the likely reason, but perhaps not for the reason you suspect. Programs treat a single ply differently. I have already listed much of this in another post in this thread, but some programs extend when giving check rather than when evading check. A one ply search means the former will search 2 plies deep while the latter will not. Some programs include checks in the q-search, some do not. Some have a sort of "threat search" between search and q-search, some do not. Some do singular extensions at the root, some do not. It is not just a "set depth = 1" and play a game, there are too many potential differences. See my other post for more details.
Re: Chessprogams with the most chessknowing
Posted: Fri Dec 19, 2014 11:32 pm
by Steve Maughan
It is an interesting question.
I think we can peel the onion a little more and ask the following questions:
1. Which engine has the most knowledge of special positions (like the ones on this thread)
2. Which engine has the most parameters / factors in its evaluation function
3. Which engine has the most accurate static evaluation function
Note that an engine which tops the league in question 2 will not necessarily rank highly in question 3, since the coefficients applied to each factor may be incorrect. And as the number of factors increases and evaluate more subtle elements of a position, it becomes more difficult to establish the right value for the parameter.
And of course the engine which tops the league in question 3 may be not be the strongest engine. As well as search efficiencies, the time required to perform the evaluation may be so long that the engine isn't able to search as deeply as others.
Steve
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 12:19 am
by Laskos
Adam Hair wrote:
I suspect that part of the reason that Shredder is scoring much worse in your test is because it is not searching as many nodes.
I fed the same startpos to Shredder 6 PB, and it searched 63 nodes. It only partly explains the difference, as at fixed nodes one would account for the fact that Shredder 6 is ~3 times slower than Houdini (or Fire). Pity I seem to not be able to use games at fixed nodes with many engines.
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 12:23 am
by Laskos
bob wrote:
This doesn't work very well. depth=1 does NOT mean the same thing to everyone. To some it is simply a 1 ply search followed by q-search. Some allow check extensions before going to q-search. Some allow check extensions AFTER q-search. Some have a couple of plies of threats between basic search and start of q-search. Some extend at the root if in check, some don't. Some extend if giving check at the root, some don't. Etc. I played a match Crafty vs Stockfish a while back to compare evaluations. It took quite a bit of work to make them search the same basic tree space.
I ended up having to modify code in both programs to reach a consistent meaning for "set depth = 1". After all the work was done, the net conclusion from playing 100K games was "no statistical difference".
Yes, correct, but it seems hard to measure comprehensibly the static eval without messing up with the source, where available.
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 12:45 am
by Ferdy
Henryval wrote:Which Chessprogram(s) have the most implemented chessknowing. And are there more than a few proofs out there in the chesscommunity?
I wonder why we don't have any ranking about this issue.
Another position, white can only draw. Score close to zero is better. Fire 4 is impressive so far.
Hash 64mb, threads 1, time 1sec.
[d]7k/R7/7P/6K1/8/8/2b5/8 w - - 0 1
Code: Select all
1 id name Fire 4 x64 (time 749, score 23)
2 id name Critter 1.6a 64-bit (time 860, score 51)
3 id name Stockfish 131214 64 POPCNT (time 859, score 54)
4 id name Houdini 4 x64 (time 1000, score 84)
5 id name Strelka 6 w32 (time 1000, score 103)
6 id name Texel 1.04 64-bit (time 998, score 216)
7 id name Komodo 6 64-bit (time 826, score 279)
8 id name Booot 5.2.0(64) (time 16, score 319)
9 id name HIARCS 14 WCSC (time 874, score 332)
10 id name Equinox 3.30 x64mp (time 890, score 371)
11 id name Naum 4.6 (time 811, score 387)
12 id name spark-1.0 (time 1014, score 392)
13 id name Nebula 2.0 (time 812, score 395)
14 id name cheng4 0.36c (time 914, score 417)
15 id name Maverick 0.51 x64 (time 702, score 437)
16 id name DiscoCheck 5.2.1 (time 795, score 442)
17 id name Atlas 3.70em x64 (time 795, score 447)
18 id name Fruit reloaded 2.1 (time 920, score 450)
19 id name Andscacs 0.71 (time 459, score 459)
20 id name Rodent 1.6 (build 6) (time 702, score 484)
21 id name Rybka 2.3.2a mp (time 671, score 488)
22 id name Senpai 1.0 (time 967, score 491)
23 id name Arasan 17.4 (time 1014, score 496)
24 id name Protector 1.7.0 (time 936, score 500)
25 id name Ruffian 1.0.5 (time 820, score 503)
26 id name Bobcat 3.25 (time 812, score 505)
27 id name GreKo 12.1 (time 687, score 510)
28 id name Gaviota v1.0 (time 936, score 514)
29 id name Vajolet2 1.45 (time 877, score 526)
30 id name Hannibal 1.4x64 (time 1014, score 529)
31 id name Octochess revision 5190 (time 577, score 541)
32 id name Deuterium v14.3.34.130 (time 765, score 561)
33 id name DisasterArea-1.54 (time 999, score 561)
34 id name GNU Chess 5.60-64 (time 873, score 575)
35 id name Quazar 0.4 x64 (time 1046, score 593)
36 id name Rhetoric 1.4.1 x64 (time 639, score 676)
37 id name iCE 2.0 v2240 x64/popcnt (time 203, score 790)
I am using a very nice book.
100 Endgames You Must Know (3rd edition) by Jesus de la Villa
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 12:55 am
by kranium
I think a fast/efficient search and high NPS can often overcome a weak or minimal eval function at ultra-fast/very fast TCs.
But as TCs increase, engines with expensive (large) eval functions will reach sufficient depth to equalize...
and eventually, at a certain point (as TC increases) any previous advantage in NPS and depth is mitigated or nullified.
At that point, the program with the best (most accurate) eval should prove strongest.
This has been clearly shown by Komodo's development and evolution over the past few years.
Larry Kaufman is
the expert in data-mining and quantifying statistical results into chess programming friendly terms and tables.
If we look at his early 1999 publication on material imbalances:
https://home.comcast.net/~danheisman/Ar ... alance.htm
his current contributions to Komodo are surely broader, vastly matured, and superior.
It makes sense to me that today, the strongest chess engine at LTC contains the most (complete and accurate) 'chess knowledge'.
Can there be any other anwser to this question other than Komodo 8 at this time?
http://tcec.chessdom.com/live.php
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 1:02 am
by cdani
kranium wrote:I think a fast/efficient search and high NPS can often overcome a weak or minimal eval function at ultra-fast/very fast TCs.
But as TCs increase, engines with expensive (large) eval functions will reach sufficient depth to equalize...
and eventually, at a certain point (as TC increases) any previous advantage in NPS and depth is mitigated or nullified.
At that point, the program with the best (most accurate) eval should prove strongest.
This has been clearly shown by Komodo's development and evolution over the past few years.
Larry Kaufman is
the expert in data-mining and quantifying statistical results into chess programming friendly terms and tables.
If we look at his early 1999 publication on material imbalances:
https://home.comcast.net/~danheisman/Ar ... alance.htm
his current contributions to Komodo are surely broader, vastly matured, and superior.
It makes sense to me that today, the strongest chess engine at LTC contains the most (complete and accurate) 'chess knowledge'.
Can there be any other anwser to this question other than Komodo 8 at this time?
http://tcec.chessdom.com/live.php
I'm sure it's like this also. The specialized evals are the future.
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 1:11 am
by Ferdy
kranium wrote:I think a fast/efficient search and high NPS can often overcome a weak or minimal eval function at ultra-fast/very fast TCs.
But as TCs increase, engines with expensive (large) eval functions will reach sufficient depth to equalize...
and eventually, at a certain point (as TC increases) any previous advantage in NPS and depth is mitigated or nullified.
At that point, the program with the best (most accurate) eval should prove strongest.
This has been clearly shown by Komodo's development and evolution over the past few years.
Larry Kaufman is
the expert in data-mining and quantifying statistical results into chess programming friendly terms and tables.
If we look at his early 1999 publication on material imbalances:
https://home.comcast.net/~danheisman/Ar ... alance.htm
his current contributions to Komodo are surely broader, vastly matured, and superior.
It makes sense to me that today, the strongest chess engine at LTC contains the most (complete and accurate) 'chess knowledge'.
Can there be any other anwser to this question other than Komodo 8 at this time?
http://tcec.chessdom.com/live.php
There are positions that an engine will not be able to solve even if given more time. Engines that have good estimation of the positions are usually search-efficient.
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 1:21 am
by kranium
Ferdy wrote:
There are positions that an engine will not be able to solve even if given more time. Engines that have good estimation of the positions are usually search-efficient.
Agreed Ferdinand...
I just believe that what separates Komodo from the rest of the pack at this date and time is it's eval (and it's just getting better and better)
When Komodo wins, it's probably not out-searching SF.
Re: Chessprogams with the most chessknowing
Posted: Sat Dec 20, 2014 1:23 am
by Ferdy
Henryval wrote:Which Chessprogram(s) have the most implemented chessknowing. And are there more than a few proofs out there in the chesscommunity?
I wonder why we don't have any ranking about this issue.
One more, white is fine. Fire tops again.
[d]8/8/5k2/8/8/4qBB1/6K1/8 w - - 0 1
Code: Select all
1 id name Fire 4 x64 (time 842, score -12)
2 id name Komodo 6 64-bit (time 748, score -32)
3 id name Texel 1.04 64-bit (time 936, score -72)
4 id name Booot 5.2.0(64) (time 15, score -147)
5 id name Octochess revision 5190 (time 639, score -162)
6 id name Atlas 3.70em x64 (time 859, score -178)
7 id name DiscoCheck 5.2.1 (time 951, score -182)
8 id name Maverick 0.51 x64 (time 717, score -198)
9 id name Ruffian 1.0.5 (time 780, score -207)
10 id name Andscacs 0.71 (time 942, score -221)
11 id name Houdini 4 x64 (time 1000, score -222)
12 id name Strelka 6 w32 (time 1000, score -225)
13 id name Equinox 3.30 x64mp (time 951, score -230)
14 id name Critter 1.6a 64-bit (time 807, score -236)
15 id name Naum 4.6 (time 796, score -256)
16 id name Fruit reloaded 2.1 (time 998, score -285)
17 id name Arasan 17.4 (time 1031, score -296)
18 id name Nebula 2.0 (time 951, score -307)
19 id name Rodent 1.6 (build 6) (time 827, score -308)
20 id name HIARCS 14 WCSC (time 889, score -311)
21 id name Rybka 2.3.2a mp (time 859, score -324)
22 id name Deuterium v14.3.34.130 (time 905, score -335)
23 id name Stockfish 131214 64 POPCNT (time 921, score -337)
24 id name Protector 1.7.0 (time 936, score -349)
25 id name DisasterArea-1.54 (time 936, score -362)
26 id name cheng4 0.36c (time 724, score -369)
27 id name Hannibal 1.4x64 (time 655, score -379)
28 id name Vajolet2 1.45 (time 784, score -390)
29 id name Senpai 1.0 (time 889, score -391)
30 id name GreKo 12.1 (time 983, score -397)
31 id name Gaviota v1.0 (time 764, score -400)
32 id name Bobcat 3.25 (time 905, score -413)
33 id name GNU Chess 5.60-64 (time 876, score -430)
34 id name iCE 2.0 v2240 x64/popcnt (time 343, score -438)
35 id name Rhetoric 1.4.1 x64 (time 359, score -475)
36 id name Quazar 0.4 x64 (time 1061, score -484)
37 id name spark-1.0 (time 1014, score -568)