Page 2 of 3
Re: Low impact of opening phase in engine play?
Posted: Tue Apr 18, 2017 9:00 pm
by Laskos
jdart wrote:42 Elo points on an absolute scale is pretty significant.
--Jon
It depends on the opponent, the larger the strength difference, bigger is the Elo contribution. What is important here is the relative contribution of the opening phase to game-play, and it seems to be 15% from 2moves_v1.epd and 22% from 3moves_GM.epd. I expected larger 30-40% opening phase of the game contribution. In human high level chess, this is roughly the attention given to openings.
Re: Low impact of opening phase in engine play?
Posted: Tue Apr 18, 2017 9:23 pm
by cdani
Laskos wrote:jdart wrote:42 Elo points on an absolute scale is pretty significant.
--Jon
It depends on the opponent, the larger the strength difference, bigger is the Elo contribution. What is important here is the relative contribution of the opening phase to game-play, and it seems to be 15% from 2moves_v1.epd and 22% from 3moves_GM.epd. I expected larger 30-40% opening phase of the game contribution. In human high level chess, this is roughly the attention given to openings.
Thanks for the tests. As commented maybe is due to bad play of Andscacs relative to Stockfish in any phase of the game, but the fast time control sure generates bigger disconnection between the different phases of the games than a slower one.
Re: Low impact of opening phase in engine play?
Posted: Tue Apr 18, 2017 11:44 pm
by Ras
An engine opening book becomes most valuable if it is actually tuned to the engine. Then it makes the engine head for positions that it will like (beyond horizon) and avoid positions it would be unhappy with lateron.
Using an untuned book with "objectively" good moves can still lead the engine to lost positions if it cannot handle them correctly.
However, tuning is a lot of work. It is not sufficient to filter out all openings that have a good GM score.
Re: Low impact of opening phase in engine play?
Posted: Wed Apr 19, 2017 12:47 am
by mbabigian
I would try reversing it. Have Andscacs play moves 1-12 and Stockfish the rest. See if the 42 point delta still holds.
Just a thought.
Mike
Re: Low impact of opening phase in engine play?
Posted: Wed Apr 19, 2017 2:12 am
by Dann Corbit
There are more than 5000 positions in the 2 moves book with a centipawn value of 50 or more (which is pretty one sided). Large lopsided margins give the weaker engine winning chances.
Here are the top offenders:
rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/2n5/6p1/3P4/5P2/PPP1P1PP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/n7/6p1/8/1P2P3/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/ppppp2p/8/5pp1/8/2P2N2/PP1PPPPP/RNBQKB1R w KQkq -
rnbqkbnr/p1ppp1pp/1p6/5p2/1P2P3/8/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/pppp1p1p/6p1/4p3/6P1/5P2/PPPPP2P/RNBQKBNR w KQkq -
rnbqkbn1/pppppppr/8/7p/8/2P4P/PP1PPPP1/RNBQKBNR w KQq -
rnbqkbnr/ppp1pp1p/8/3p2p1/7P/N7/PPPPPPP1/R1BQKBNR w KQkq -
rnbqkbnr/ppp2ppp/8/3pp3/4P1P1/8/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/pppppp1p/5n2/6p1/3P4/N7/PPP1PPPP/R1BQKBNR w KQkq -
For example:
[d]rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
is likely won for the side to move (sample pv: Be2 Nf6 e5 Ng4 d4 g6 Nh3 d6 Nf4 Kf7 Bc4+ Kg7 e6 c6 Qe2 Qe8 Nd2 Kg8 Nf3 Bh6 Nh3 Bxc1 Rxc1 Nf6 Nfg5 h4 Bb3 Rh5 Nf7 d5 Nf4 Bxe6 Qxe6 Qxf7).
There are also more than 5000 positions with a ce near zero (5 centipawns or less in absolute value).
Here are ten from that segment:
rnbqkbnr/1ppp1ppp/4p3/p7/1P6/P7/2PPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/1p1ppppp/2p5/p7/1P6/8/PBPPPPPP/RN1QKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/4P3/6P1/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/ppp1pppp/5n2/3p4/8/P4N2/1PPPPPPP/RNBQKB1R w KQkq -
rnbqkb1r/pp1ppppp/5n2/2p5/2P5/6P1/PP1PPP1P/RNBQKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/8/3P1N2/PPP1PPPP/RNBQKB1R w KQkq -
rnbqkbnr/ppppp1pp/8/8/3Pp3/8/PPP2PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppp1ppp/2n5/4p3/8/5NP1/PPPPPP1P/RNBQKB1R w KQkq -
rnbqkbnr/pp2pppp/3p4/2p5/4P3/7P/PPPP1PP1/RNBQKBNR w KQkq -
rnbqkbnr/pp1p1ppp/4p3/2p5/5P2/5N2/PPPPP1PP/RNBQKB1R w KQkq -
The zero segment will be very drawish, so that even a weaker engine has pretty good drawing chances.
I am currently of the opinion that a book used for Elo evaluation should have a leaf position which is between 25 and 45 centipawns in absolute value for all exit positions.
Elo values close to zero have a disproportionate number of draws when played.
Elo values at or above 50 lead to more lopsided results for the side with an advantage.
Just a general guideline {opinion} and not rigorously proven.
Re: Low impact of opening phase in engine play?
Posted: Wed Apr 19, 2017 2:27 am
by Laskos
Dann Corbit wrote:There are more than 5000 positions in the 2 moves book with a centipawn value of 50 or more (which is pretty one sided). Large lopsided margins give the weaker engine winning chances.
Here are the top offenders:
rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/2n5/6p1/3P4/5P2/PPP1P1PP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/n7/6p1/8/1P2P3/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/ppppp2p/8/5pp1/8/2P2N2/PP1PPPPP/RNBQKB1R w KQkq -
rnbqkbnr/p1ppp1pp/1p6/5p2/1P2P3/8/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/pppp1p1p/6p1/4p3/6P1/5P2/PPPPP2P/RNBQKBNR w KQkq -
rnbqkbn1/pppppppr/8/7p/8/2P4P/PP1PPPP1/RNBQKBNR w KQq -
rnbqkbnr/ppp1pp1p/8/3p2p1/7P/N7/PPPPPPP1/R1BQKBNR w KQkq -
rnbqkbnr/ppp2ppp/8/3pp3/4P1P1/8/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/pppppp1p/5n2/6p1/3P4/N7/PPP1PPPP/R1BQKBNR w KQkq -
For example:
[d]rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
is likely won for the side to move (sample pv: Be2 Nf6 e5 Ng4 d4 g6 Nh3 d6 Nf4 Kf7 Bc4+ Kg7 e6 c6 Qe2 Qe8 Nd2 Kg8 Nf3 Bh6 Nh3 Bxc1 Rxc1 Nf6 Nfg5 h4 Bb3 Rh5 Nf7 d5 Nf4 Bxe6 Qxe6 Qxf7).
There are also more than 5000 positions with a ce near zero (5 centipawns or less in absolute value).
Here are ten from that segment:
rnbqkbnr/1ppp1ppp/4p3/p7/1P6/P7/2PPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/1p1ppppp/2p5/p7/1P6/8/PBPPPPPP/RN1QKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/4P3/6P1/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/ppp1pppp/5n2/3p4/8/P4N2/1PPPPPPP/RNBQKB1R w KQkq -
rnbqkb1r/pp1ppppp/5n2/2p5/2P5/6P1/PP1PPP1P/RNBQKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/8/3P1N2/PPP1PPPP/RNBQKB1R w KQkq -
rnbqkbnr/ppppp1pp/8/8/3Pp3/8/PPP2PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppp1ppp/2n5/4p3/8/5NP1/PPPPPP1P/RNBQKB1R w KQkq -
rnbqkbnr/pp2pppp/3p4/2p5/4P3/7P/PPPP1PP1/RNBQKBNR w KQkq -
rnbqkbnr/pp1p1ppp/4p3/2p5/5P2/5N2/PPPPP1PP/RNBQKB1R w KQkq -
The zero segment will be very drawish, so that even a weaker engine has pretty good drawing chances.
I am currently of the opinion that a book used for Elo evaluation should have a leaf position which is between 25 and 45 centipawns in absolute value for all exit positions.
Elo values close to zero have a disproportionate number of draws when played.
Elo values at or above 50 lead to more lopsided results for the side with an advantage.
Just a general guideline {opinion} and not rigorously proven.
Thanks Dann, I knew a bit these facts about 2moves, I myself built from this file "balanced" books and "unbalanced" ones. 2moves_v1.epd is a pretty rough file of nonsensical random 2-movers. As we saw here, it can screw opening phase play. Unbalanced positions in the range 0.5-1.3 pawns advantage can be useful in unbalanced play using pentanomial method for error bars and SPRT stop when the draw rate is high. The "golden range" for sensitivity is somewhere close to 45-50% draws. Balanced positions can give 70% and more for draw rate between closely related engines.
Re: Low impact of opening phase in engine play?
Posted: Wed Apr 19, 2017 2:50 am
by Laskos
Ras wrote:An engine opening book becomes most valuable if it is actually tuned to the engine. Then it makes the engine head for positions that it will like (beyond horizon) and avoid positions it would be unhappy with lateron.
Using an untuned book with "objectively" good moves can still lead the engine to lost positions if it cannot handle them correctly.
However, tuning is a lot of work. It is not sufficient to filter out all openings that have a good GM score.
That GM book I built was not for the purpose of playing well. It simply offered variety and reasonable moves, and I kept it as short as possible, but still to offer variety. 2moves_v1.epd assures variety, but it leads to nonsensical openings, and is screwing a bit the opening play. I wanted to see the contribution from openings in engine's play.
I played another experiment:
Stockfish - Andscacs --->
211 Elo points difference
Stockfish - Stockfish first 12 moves + Andscacs the rest --->
165 Elo points difference
Stockfish - Stockfish first 24 moves + Andscacs the rest --->
89 Elo points difference
Contribution moves 1-12 to gameplay:
22%
Contribution moves 13-24 to gameplay:
36%
Humans concentrate on what they can tackle: openings and endgames. Openings seem to offer roughly 20% of game-play, endgames 15%. But the "soul" of the game seems to be moves 13-24, with 36% contribution. So, a "midgame book" would offer much more than an "opening book", only that this kind of "midgame book" is almost impossible to build.
Re: Low impact of opening phase in engine play?
Posted: Wed Apr 19, 2017 3:06 am
by Dann Corbit
I see a pretty startling win difference for a small ce change past 50. For instance, the query {centered around 50 centipawns, +/- 5 cp}:
Code: Select all
select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between 50 and 60
Gives:
Code: Select all
wwa bwa da
0.424612 0.275602 0.299784
And this query {centered around 125 centipawns, +/- 5 cp}:
Code: Select all
select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between 120 and 130
gives:
Code: Select all
wwa bwa da
0.517428 0.250179 0.232392
Re: Low impact of opening phase in engine play?
Posted: Wed Apr 19, 2017 3:08 am
by Dann Corbit
BTW, I think this result is not surprising:
Code: Select all
select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between -5 and 5
Code: Select all
wwa bwa da
0.322439 0.286634 0.390925
Because the small advantage allowed will usually be held by white (due to the tempo)
Re: Low impact of opening phase in engine play?
Posted: Wed Apr 19, 2017 11:20 pm
by Ras
Laskos wrote:That GM book I built was not for the purpose of playing well.
Then it is pointless to draw ELO conslusions. Besides, as I said, reasonable moves can still throw an engine into the sewers if it cannot cope with the resulting positions.
Example here, I have a hand-tuned book of about 20,000 moves. All of them manually verified for objective soundness (judged by Shredder) and whether they fit the CT800 style. Since this machine is primarily designed to play humans, not computers, it avoids closed positions like the plague because it "knows" that humans will try this trick. This knowledge is part of the asymmetric static eval, but it has to start with the opening book.
So, unless the book is tuned to emphasise the strong points of the engine and avoid its weak spots, the question whether it contributes to playing strength is not really meaningful, IMO.
The basic aspect of an untuned book is "only" time saving.