It depends on the opponent, the larger the strength difference, bigger is the Elo contribution. What is important here is the relative contribution of the opening phase to game-play, and it seems to be 15% from 2moves_v1.epd and 22% from 3moves_GM.epd. I expected larger 30-40% opening phase of the game contribution. In human high level chess, this is roughly the attention given to openings.jdart wrote:42 Elo points on an absolute scale is pretty significant.
--Jon
Low impact of opening phase in engine play?
Moderators: hgm, Rebel, chrisw
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Low impact of opening phase in engine play?
-
- Posts: 2204
- Joined: Sat Jan 18, 2014 10:24 am
- Location: Andorra
Re: Low impact of opening phase in engine play?
Thanks for the tests. As commented maybe is due to bad play of Andscacs relative to Stockfish in any phase of the game, but the fast time control sure generates bigger disconnection between the different phases of the games than a slower one.Laskos wrote:It depends on the opponent, the larger the strength difference, bigger is the Elo contribution. What is important here is the relative contribution of the opening phase to game-play, and it seems to be 15% from 2moves_v1.epd and 22% from 3moves_GM.epd. I expected larger 30-40% opening phase of the game contribution. In human high level chess, this is roughly the attention given to openings.jdart wrote:42 Elo points on an absolute scale is pretty significant.
--Jon
Daniel José - http://www.andscacs.com
-
- Posts: 2488
- Joined: Tue Aug 30, 2016 8:19 pm
- Full name: Rasmus Althoff
Re: Low impact of opening phase in engine play?
An engine opening book becomes most valuable if it is actually tuned to the engine. Then it makes the engine head for positions that it will like (beyond horizon) and avoid positions it would be unhappy with lateron.
Using an untuned book with "objectively" good moves can still lead the engine to lost positions if it cannot handle them correctly.
However, tuning is a lot of work. It is not sufficient to filter out all openings that have a good GM score.
Using an untuned book with "objectively" good moves can still lead the engine to lost positions if it cannot handle them correctly.
However, tuning is a lot of work. It is not sufficient to filter out all openings that have a good GM score.
-
- Posts: 204
- Joined: Tue Oct 15, 2013 2:34 am
- Location: US
- Full name: Mike Babigian
Re: Low impact of opening phase in engine play?
I would try reversing it. Have Andscacs play moves 1-12 and Stockfish the rest. See if the 42 point delta still holds.
Just a thought.
Mike
Just a thought.
Mike
-
- Posts: 12541
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Low impact of opening phase in engine play?
There are more than 5000 positions in the 2 moves book with a centipawn value of 50 or more (which is pretty one sided). Large lopsided margins give the weaker engine winning chances.
Here are the top offenders:
rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/2n5/6p1/3P4/5P2/PPP1P1PP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/n7/6p1/8/1P2P3/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/ppppp2p/8/5pp1/8/2P2N2/PP1PPPPP/RNBQKB1R w KQkq -
rnbqkbnr/p1ppp1pp/1p6/5p2/1P2P3/8/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/pppp1p1p/6p1/4p3/6P1/5P2/PPPPP2P/RNBQKBNR w KQkq -
rnbqkbn1/pppppppr/8/7p/8/2P4P/PP1PPPP1/RNBQKBNR w KQq -
rnbqkbnr/ppp1pp1p/8/3p2p1/7P/N7/PPPPPPP1/R1BQKBNR w KQkq -
rnbqkbnr/ppp2ppp/8/3pp3/4P1P1/8/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/pppppp1p/5n2/6p1/3P4/N7/PPP1PPPP/R1BQKBNR w KQkq -
For example:
[d]rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
is likely won for the side to move (sample pv: Be2 Nf6 e5 Ng4 d4 g6 Nh3 d6 Nf4 Kf7 Bc4+ Kg7 e6 c6 Qe2 Qe8 Nd2 Kg8 Nf3 Bh6 Nh3 Bxc1 Rxc1 Nf6 Nfg5 h4 Bb3 Rh5 Nf7 d5 Nf4 Bxe6 Qxe6 Qxf7).
There are also more than 5000 positions with a ce near zero (5 centipawns or less in absolute value).
Here are ten from that segment:
rnbqkbnr/1ppp1ppp/4p3/p7/1P6/P7/2PPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/1p1ppppp/2p5/p7/1P6/8/PBPPPPPP/RN1QKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/4P3/6P1/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/ppp1pppp/5n2/3p4/8/P4N2/1PPPPPPP/RNBQKB1R w KQkq -
rnbqkb1r/pp1ppppp/5n2/2p5/2P5/6P1/PP1PPP1P/RNBQKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/8/3P1N2/PPP1PPPP/RNBQKB1R w KQkq -
rnbqkbnr/ppppp1pp/8/8/3Pp3/8/PPP2PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppp1ppp/2n5/4p3/8/5NP1/PPPPPP1P/RNBQKB1R w KQkq -
rnbqkbnr/pp2pppp/3p4/2p5/4P3/7P/PPPP1PP1/RNBQKBNR w KQkq -
rnbqkbnr/pp1p1ppp/4p3/2p5/5P2/5N2/PPPPP1PP/RNBQKB1R w KQkq -
The zero segment will be very drawish, so that even a weaker engine has pretty good drawing chances.
I am currently of the opinion that a book used for Elo evaluation should have a leaf position which is between 25 and 45 centipawns in absolute value for all exit positions.
Elo values close to zero have a disproportionate number of draws when played.
Elo values at or above 50 lead to more lopsided results for the side with an advantage.
Just a general guideline {opinion} and not rigorously proven.
Here are the top offenders:
rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/2n5/6p1/3P4/5P2/PPP1P1PP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/n7/6p1/8/1P2P3/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/ppppp2p/8/5pp1/8/2P2N2/PP1PPPPP/RNBQKB1R w KQkq -
rnbqkbnr/p1ppp1pp/1p6/5p2/1P2P3/8/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/pppp1p1p/6p1/4p3/6P1/5P2/PPPPP2P/RNBQKBNR w KQkq -
rnbqkbn1/pppppppr/8/7p/8/2P4P/PP1PPPP1/RNBQKBNR w KQq -
rnbqkbnr/ppp1pp1p/8/3p2p1/7P/N7/PPPPPPP1/R1BQKBNR w KQkq -
rnbqkbnr/ppp2ppp/8/3pp3/4P1P1/8/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/pppppp1p/5n2/6p1/3P4/N7/PPP1PPPP/R1BQKBNR w KQkq -
For example:
[d]rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
is likely won for the side to move (sample pv: Be2 Nf6 e5 Ng4 d4 g6 Nh3 d6 Nf4 Kf7 Bc4+ Kg7 e6 c6 Qe2 Qe8 Nd2 Kg8 Nf3 Bh6 Nh3 Bxc1 Rxc1 Nf6 Nfg5 h4 Bb3 Rh5 Nf7 d5 Nf4 Bxe6 Qxe6 Qxf7).
There are also more than 5000 positions with a ce near zero (5 centipawns or less in absolute value).
Here are ten from that segment:
rnbqkbnr/1ppp1ppp/4p3/p7/1P6/P7/2PPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/1p1ppppp/2p5/p7/1P6/8/PBPPPPPP/RN1QKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/4P3/6P1/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/ppp1pppp/5n2/3p4/8/P4N2/1PPPPPPP/RNBQKB1R w KQkq -
rnbqkb1r/pp1ppppp/5n2/2p5/2P5/6P1/PP1PPP1P/RNBQKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/8/3P1N2/PPP1PPPP/RNBQKB1R w KQkq -
rnbqkbnr/ppppp1pp/8/8/3Pp3/8/PPP2PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppp1ppp/2n5/4p3/8/5NP1/PPPPPP1P/RNBQKB1R w KQkq -
rnbqkbnr/pp2pppp/3p4/2p5/4P3/7P/PPPP1PP1/RNBQKBNR w KQkq -
rnbqkbnr/pp1p1ppp/4p3/2p5/5P2/5N2/PPPPP1PP/RNBQKB1R w KQkq -
The zero segment will be very drawish, so that even a weaker engine has pretty good drawing chances.
I am currently of the opinion that a book used for Elo evaluation should have a leaf position which is between 25 and 45 centipawns in absolute value for all exit positions.
Elo values close to zero have a disproportionate number of draws when played.
Elo values at or above 50 lead to more lopsided results for the side with an advantage.
Just a general guideline {opinion} and not rigorously proven.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Low impact of opening phase in engine play?
Thanks Dann, I knew a bit these facts about 2moves, I myself built from this file "balanced" books and "unbalanced" ones. 2moves_v1.epd is a pretty rough file of nonsensical random 2-movers. As we saw here, it can screw opening phase play. Unbalanced positions in the range 0.5-1.3 pawns advantage can be useful in unbalanced play using pentanomial method for error bars and SPRT stop when the draw rate is high. The "golden range" for sensitivity is somewhere close to 45-50% draws. Balanced positions can give 70% and more for draw rate between closely related engines.Dann Corbit wrote:There are more than 5000 positions in the 2 moves book with a centipawn value of 50 or more (which is pretty one sided). Large lopsided margins give the weaker engine winning chances.
Here are the top offenders:
rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/2n5/6p1/3P4/5P2/PPP1P1PP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/n7/6p1/8/1P2P3/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/ppppp2p/8/5pp1/8/2P2N2/PP1PPPPP/RNBQKB1R w KQkq -
rnbqkbnr/p1ppp1pp/1p6/5p2/1P2P3/8/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/pppp1p1p/6p1/4p3/6P1/5P2/PPPPP2P/RNBQKBNR w KQkq -
rnbqkbn1/pppppppr/8/7p/8/2P4P/PP1PPPP1/RNBQKBNR w KQq -
rnbqkbnr/ppp1pp1p/8/3p2p1/7P/N7/PPPPPPP1/R1BQKBNR w KQkq -
rnbqkbnr/ppp2ppp/8/3pp3/4P1P1/8/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/pppppp1p/5n2/6p1/3P4/N7/PPP1PPPP/R1BQKBNR w KQkq -
For example:
[d]rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
is likely won for the side to move (sample pv: Be2 Nf6 e5 Ng4 d4 g6 Nh3 d6 Nf4 Kf7 Bc4+ Kg7 e6 c6 Qe2 Qe8 Nd2 Kg8 Nf3 Bh6 Nh3 Bxc1 Rxc1 Nf6 Nfg5 h4 Bb3 Rh5 Nf7 d5 Nf4 Bxe6 Qxe6 Qxf7).
There are also more than 5000 positions with a ce near zero (5 centipawns or less in absolute value).
Here are ten from that segment:
rnbqkbnr/1ppp1ppp/4p3/p7/1P6/P7/2PPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/1p1ppppp/2p5/p7/1P6/8/PBPPPPPP/RN1QKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/4P3/6P1/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/ppp1pppp/5n2/3p4/8/P4N2/1PPPPPPP/RNBQKB1R w KQkq -
rnbqkb1r/pp1ppppp/5n2/2p5/2P5/6P1/PP1PPP1P/RNBQKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/8/3P1N2/PPP1PPPP/RNBQKB1R w KQkq -
rnbqkbnr/ppppp1pp/8/8/3Pp3/8/PPP2PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppp1ppp/2n5/4p3/8/5NP1/PPPPPP1P/RNBQKB1R w KQkq -
rnbqkbnr/pp2pppp/3p4/2p5/4P3/7P/PPPP1PP1/RNBQKBNR w KQkq -
rnbqkbnr/pp1p1ppp/4p3/2p5/5P2/5N2/PPPPP1PP/RNBQKB1R w KQkq -
The zero segment will be very drawish, so that even a weaker engine has pretty good drawing chances.
I am currently of the opinion that a book used for Elo evaluation should have a leaf position which is between 25 and 45 centipawns in absolute value for all exit positions.
Elo values close to zero have a disproportionate number of draws when played.
Elo values at or above 50 lead to more lopsided results for the side with an advantage.
Just a general guideline {opinion} and not rigorously proven.
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: Low impact of opening phase in engine play?
That GM book I built was not for the purpose of playing well. It simply offered variety and reasonable moves, and I kept it as short as possible, but still to offer variety. 2moves_v1.epd assures variety, but it leads to nonsensical openings, and is screwing a bit the opening play. I wanted to see the contribution from openings in engine's play.Ras wrote:An engine opening book becomes most valuable if it is actually tuned to the engine. Then it makes the engine head for positions that it will like (beyond horizon) and avoid positions it would be unhappy with lateron.
Using an untuned book with "objectively" good moves can still lead the engine to lost positions if it cannot handle them correctly.
However, tuning is a lot of work. It is not sufficient to filter out all openings that have a good GM score.
I played another experiment:
Stockfish - Andscacs ---> 211 Elo points difference
Stockfish - Stockfish first 12 moves + Andscacs the rest ---> 165 Elo points difference
Stockfish - Stockfish first 24 moves + Andscacs the rest ---> 89 Elo points difference
Contribution moves 1-12 to gameplay: 22%
Contribution moves 13-24 to gameplay: 36%
Humans concentrate on what they can tackle: openings and endgames. Openings seem to offer roughly 20% of game-play, endgames 15%. But the "soul" of the game seems to be moves 13-24, with 36% contribution. So, a "midgame book" would offer much more than an "opening book", only that this kind of "midgame book" is almost impossible to build.
-
- Posts: 12541
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Low impact of opening phase in engine play?
I see a pretty startling win difference for a small ce change past 50. For instance, the query {centered around 50 centipawns, +/- 5 cp}:
Gives:
And this query {centered around 125 centipawns, +/- 5 cp}:
gives:
Code: Select all
select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between 50 and 60
Code: Select all
wwa bwa da
0.424612 0.275602 0.299784
Code: Select all
select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between 120 and 130
Code: Select all
wwa bwa da
0.517428 0.250179 0.232392
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 12541
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Low impact of opening phase in engine play?
BTW, I think this result is not surprising:
Because the small advantage allowed will usually be held by white (due to the tempo)
Code: Select all
select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between -5 and 5
Code: Select all
wwa bwa da
0.322439 0.286634 0.390925
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 2488
- Joined: Tue Aug 30, 2016 8:19 pm
- Full name: Rasmus Althoff
Re: Low impact of opening phase in engine play?
Then it is pointless to draw ELO conslusions. Besides, as I said, reasonable moves can still throw an engine into the sewers if it cannot cope with the resulting positions.Laskos wrote:That GM book I built was not for the purpose of playing well.
Example here, I have a hand-tuned book of about 20,000 moves. All of them manually verified for objective soundness (judged by Shredder) and whether they fit the CT800 style. Since this machine is primarily designed to play humans, not computers, it avoids closed positions like the plague because it "knows" that humans will try this trick. This knowledge is part of the asymmetric static eval, but it has to start with the opening book.
So, unless the book is tuned to emphasise the strong points of the engine and avoid its weak spots, the question whether it contributes to playing strength is not really meaningful, IMO.
The basic aspect of an untuned book is "only" time saving.