Page 1 of 2
The influence of the length of openings
Posted: Sun Jul 14, 2013 10:18 pm
by Laskos
With Houdini 3 at 2.5''+0.05'', Komodo 5.1 at 5''+0.1'', Rybka 4.1 at 5''+0.1'', Stockfish 3 at 7.5''+0.15'' to somewhat equalize the strengths, I put them to play from different length opening lines using SWCR.PGN and GM2006.PGN game collections, the opening length being 4, 16 and 30 plies. I expected a more spectacular result, with Houdini showing a better performance at only 4 ply length openings, as it's very good at Chess 960, but it's not the case, all four engines show similar behavior and dependency with the opening length. The longer the opening lines, more are the draws (9% more draws from 4 to 30 plies), and the ranking is compressed (from 76 points difference at 4 plies to 56 points at 30 plies openings). The strength shown subjected to compression remains very stable, so there is no danger of distorting the order in the rankings from longer or shorter openings of testers and testing groups.
Code: Select all
Openings 4 plies
Program Score % Elo + - Draws
1 Komodo 5.1 : 1699.0/3000 56.6 35 10 10 31.3 %
2 Stockfish 3 : 1602.5/3000 53.4 18 10 10 32.4 %
3 Houdini 3 : 1430.0/3000 47.7 -12 11 11 28.4 %
4 Rybka 4.1 : 1268.5/3000 42.3 -41 10 10 31.1 %
Openings 16 plies
Program Score % Elo + - Draws
1 Komodo 5.1 : 1662.5/3000 55.4 28 10 10 36.4 %
2 Stockfish 3 : 1597.5/3000 53.2 17 10 10 34.4 %
3 Houdini 3 : 1429.5/3000 47.6 -12 10 10 33.7 %
4 Rybka 4.1 : 1310.5/3000 43.7 -33 10 10 36.4 %
Openings 30 plies
Program Score % Elo + - Draws
1 Komodo 5.1 : 1646.0/3016 54.6 24 10 10 38.7 %
2 Stockfish 3 : 1605.5/3015 53.3 17 10 10 41.0 %
3 Houdini 3 : 1457.5/3017 48.3 -9 10 10 38.9 %
4 Rybka 4.1 : 1322.0/3014 43.9 -32 10 10 41.0 %
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 7:44 am
by IWB
Laskos wrote:...(from 76 points difference at 4 plies to 56 points at 30 plies openings...
What is the average game length for 4 and 30 plies? If there is a bigger difference in the "self calculated moves" it is like a longer o shorter time control which might increase the draw rate.
But interesting tha besides the higher draw rate notin changes in ranking ...
Interesting, thx
Ingo
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 10:33 am
by Laskos
IWB wrote:Laskos wrote:...(from 76 points difference at 4 plies to 56 points at 30 plies openings...
What is the average game length for 4 and 30 plies? If there is a bigger difference in the "self calculated moves" it is like a longer o shorter time control which might increase the draw rate.
But interesting tha besides the higher draw rate notin changes in ranking ...
Interesting, thx
Ingo
I looked in SCID for the game length, for 4 plies the average was about 70, for 16 about 74, for 30 about 77. And the time control was with increment, so the total effective TC didn't increase with longer openings by more than 2-3%. It cannot account for such an increase in draw rate (from 31% to 40%). Yes, I found surprising that the rating just compresses due to higher draw rate, but the relative ratings remain almost unchanged. Sure, this is with many games and neutral openings, if one begins to set all sorts of traps in the openings, the picture might look different, but I was not interested in that. My gut feeling was that Houdini deals better with the opening phase of the game by itself, without a book, but as it turns out, all the 4 top engines behave the same. And I somehow mistrusted tests with openings longer than 12 moves, it turns out that they are valid, if the openings are neutral, at least for these 4 engines (only that we should expect a higher draw rate).
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 11:11 am
by IWB
Laskos wrote:IWB wrote:Laskos wrote:...(from 76 points difference at 4 plies to 56 points at 30 plies openings...
What is the average game length for 4 and 30 plies? If there is a bigger difference in the "self calculated moves" it is like a longer o shorter time control which might increase the draw rate.
But interesting tha besides the higher draw rate notin changes in ranking ...
Interesting, thx
Ingo
I looked in SCID for the game length, for 4 plies the average was about 70, for 16 about 74, for 30 about 77. And the time control was with increment, so the total effective TC didn't increase with longer openings by more than 2-3%. It cannot account for such an increase in draw rate (from 31% to 40%). Yes, I found surprising that the rating just compresses due to higher draw rate, but the relative ratings remain almost unchanged. Sure, this is with many games and neutral openings, if one begins to set all sorts of traps in the openings, the picture might look different, but I was not interested in that. My gut feeling was that Houdini deals better with the opening phase of the game by itself, without a book, but as it turns out, all the 4 top engines behave the same. And I somehow mistrusted tests with openings longer than 12 moves, it turns out that they are valid, if the openings are neutral, at least for these 4 engines (only that we should expect a higher draw rate).
If the average game length is basically not influenced by the length of the opening the only thing to cause a higher draw rate I can think of is the fact that a longer but equal opening simply leaves less room for engines as it IS already more drawish ...
For a tester this means that he has to find openings which are
1. as short as possible (o let an engine decide and not a book or position)
2. as different as possible (to have a variaty of different chess openings)
Thx for the info
Ingo
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 12:56 pm
by Nelson Hernandez
Games are won when one player or the other makes a blunder or inaccuracy which, with accurate play, can be converted to a win. In long openings that exit into "neutral" positions you have substantially reduced the number of move-opportunities where an engine could potentially make such an inaccurate move, and the achievement of a neutral position after so many moves is inherently very drawish.
It turns out that in a surprising number of games one side is already losing by the 10th or 15th move and never equalizes. You eliminate such cases by exiting to neutral positions.
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 1:26 pm
by lucasart
It is a well known fact that opening books are mostly useless ELO-wise, past the first 3-4 moves.
They are only useful to bring diversity in the games (no opening book would mean engine plays the same as algorithm is deterministic).
In fact, the real risk, is that using a large opening book (compiled automatically out of large database of unverified moves) forces mistakes that the engine wouldn't play. In other words, engines generally play better than large books (which doesn't mean that all moves they play are better. remember that ELO is more determined by your 1% worst moves than your 1% best ones...)
Personally, I use 8-moves, in order to reach a good balance between diversity and engine creativity. There's nothing more boring than these extremely long book lines that effectively start the game into an already drawn endgame.
An engine that I really hated for that was Spike 1.2: you cannot disable the book and it's hardcoded inside the executable! And the book is just endless and always tries to play for boring, blocked, symmetric, drawish positions... It really killed the creatiity and the fun.
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 2:19 pm
by Don
We do all our testing with the very shallow and wide opening books - we have only 5 moves (10 ply.)
Obviously, if the opening book is playing most of the game you are not really testing the engine as much and you give much greater chances to the weaker engine.
Some people use these highly developed opening books that go very deep and that is fine for maximizing the strength on-line or in a tournament setting but not for computer vs computer objective testing.
Laskos wrote:With Houdini 3 at 2.5''+0.05'', Komodo 5.1 at 5''+0.1'', Rybka 4.1 at 5''+0.1'', Stockfish 3 at 7.5''+0.15'' to somewhat equalize the strengths, I put them to play from different length opening lines using SWCR.PGN and GM2006.PGN game collections, the opening length being 4, 16 and 30 plies. I expected a more spectacular result, with Houdini showing a better performance at only 4 ply length openings, as it's very good at Chess 960, but it's not the case, all four engines show similar behavior and dependency with the opening length. The longer the opening lines, more are the draws (9% more draws from 4 to 30 plies), and the ranking is compressed (from 76 points difference at 4 plies to 56 points at 30 plies openings). The strength shown subjected to compression remains very stable, so there is no danger of distorting the order in the rankings from longer or shorter openings of testers and testing groups.
Code: Select all
Openings 4 plies
Program Score % Elo + - Draws
1 Komodo 5.1 : 1699.0/3000 56.6 35 10 10 31.3 %
2 Stockfish 3 : 1602.5/3000 53.4 18 10 10 32.4 %
3 Houdini 3 : 1430.0/3000 47.7 -12 11 11 28.4 %
4 Rybka 4.1 : 1268.5/3000 42.3 -41 10 10 31.1 %
Openings 16 plies
Program Score % Elo + - Draws
1 Komodo 5.1 : 1662.5/3000 55.4 28 10 10 36.4 %
2 Stockfish 3 : 1597.5/3000 53.2 17 10 10 34.4 %
3 Houdini 3 : 1429.5/3000 47.6 -12 10 10 33.7 %
4 Rybka 4.1 : 1310.5/3000 43.7 -33 10 10 36.4 %
Openings 30 plies
Program Score % Elo + - Draws
1 Komodo 5.1 : 1646.0/3016 54.6 24 10 10 38.7 %
2 Stockfish 3 : 1605.5/3015 53.3 17 10 10 41.0 %
3 Houdini 3 : 1457.5/3017 48.3 -9 10 10 38.9 %
4 Rybka 4.1 : 1322.0/3014 43.9 -32 10 10 41.0 %
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 2:27 pm
by Modern Times
I prefer short (8 or less) but even better, chess960 with no book at all.
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 3:54 pm
by PaulieD
This is why engine test suites with colors reversing cannot be beat for engine testing.
Re: The influence of the length of openings
Posted: Mon Jul 15, 2013 6:18 pm
by Laskos
lucasart wrote:It is a well known fact that opening books are mostly useless ELO-wise, past the first 3-4 moves.
They are only useful to bring diversity in the games (no opening book would mean engine plays the same as algorithm is deterministic).
In fact, the real risk, is that using a large opening book (compiled automatically out of large database of unverified moves) forces mistakes that the engine wouldn't play. In other words, engines generally play better than large books (which doesn't mean that all moves they play are better. remember that ELO is more determined by your 1% worst moves than your 1% best ones...)
Personally, I use 8-moves, in order to reach a good balance between diversity and engine creativity. There's nothing more boring than these extremely long book lines that effectively start the game into an already drawn endgame.
An engine that I really hated for that was Spike 1.2: you cannot disable the book and it's hardcoded inside the executable! And the book is just endless and always tries to play for boring, blocked, symmetric, drawish positions... It really killed the creatiity and the fun.
Yes, it seems that a shallow, but still diverse book is optimal. What is curious is that I expected some engines to overperform or underpeform with only 4 ply book compared to 16 or 30 plies books. It's not the case, it seems that engines are playing openings by themselves at similar level to their overall strength. For openings you have to encode many things in the evaluation, similarly to endgames. But in endgames the engines do shift their strength compared to overall play, and it is well known that some engines under- or over-perform in the endgames.
Endgame balanced suite at same TC as in the first post:
Code: Select all
Program Score % Elo + - Draws
1 Stockfish 3 : 1651.0/3021 54.7 24 8 8 61.7 %
2 Komodo 5.1 : 1598.5/3025 52.8 15 8 8 60.3 %
3 Rybka 4.1 : 1436.5/3023 47.5 -13 8 8 60.8 %
4 Houdini 3 : 1362.0/3027 45.0 -26 8 8 57.5 %
The ratings are even more compressed, the draw ratio is very high, but one can say that in the endgames Stockfish overperforms, Houdini underperforms compared to overall rating. No such thing in the openings.