Low impact of opening phase in engine play?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 9412
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Low impact of opening phase in engine play?

Post by Laskos » Tue Apr 18, 2017 7:00 pm

jdart wrote:42 Elo points on an absolute scale is pretty significant.

--Jon
It depends on the opponent, the larger the strength difference, bigger is the Elo contribution. What is important here is the relative contribution of the opening phase to game-play, and it seems to be 15% from 2moves_v1.epd and 22% from 3moves_GM.epd. I expected larger 30-40% opening phase of the game contribution. In human high level chess, this is roughly the attention given to openings.

User avatar
cdani
Posts: 2104
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: Low impact of opening phase in engine play?

Post by cdani » Tue Apr 18, 2017 7:23 pm

Laskos wrote:
jdart wrote:42 Elo points on an absolute scale is pretty significant.

--Jon
It depends on the opponent, the larger the strength difference, bigger is the Elo contribution. What is important here is the relative contribution of the opening phase to game-play, and it seems to be 15% from 2moves_v1.epd and 22% from 3moves_GM.epd. I expected larger 30-40% opening phase of the game contribution. In human high level chess, this is roughly the attention given to openings.
Thanks for the tests. As commented maybe is due to bad play of Andscacs relative to Stockfish in any phase of the game, but the fast time control sure generates bigger disconnection between the different phases of the games than a slower one.

Ras
Posts: 1142
Joined: Tue Aug 30, 2016 6:19 pm
Contact:

Re: Low impact of opening phase in engine play?

Post by Ras » Tue Apr 18, 2017 9:44 pm

An engine opening book becomes most valuable if it is actually tuned to the engine. Then it makes the engine head for positions that it will like (beyond horizon) and avoid positions it would be unhappy with lateron.

Using an untuned book with "objectively" good moves can still lead the engine to lost positions if it cannot handle them correctly.

However, tuning is a lot of work. It is not sufficient to filter out all openings that have a good GM score.

mbabigian
Posts: 100
Joined: Tue Oct 15, 2013 12:34 am
Location: US

Re: Low impact of opening phase in engine play?

Post by mbabigian » Tue Apr 18, 2017 10:47 pm

I would try reversing it. Have Andscacs play moves 1-12 and Stockfish the rest. See if the 42 point delta still holds.

Just a thought.
Mike

Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Low impact of opening phase in engine play?

Post by Dann Corbit » Wed Apr 19, 2017 12:12 am

There are more than 5000 positions in the 2 moves book with a centipawn value of 50 or more (which is pretty one sided). Large lopsided margins give the weaker engine winning chances.

Here are the top offenders:
rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/2n5/6p1/3P4/5P2/PPP1P1PP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/n7/6p1/8/1P2P3/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/ppppp2p/8/5pp1/8/2P2N2/PP1PPPPP/RNBQKB1R w KQkq -
rnbqkbnr/p1ppp1pp/1p6/5p2/1P2P3/8/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/pppp1p1p/6p1/4p3/6P1/5P2/PPPPP2P/RNBQKBNR w KQkq -
rnbqkbn1/pppppppr/8/7p/8/2P4P/PP1PPPP1/RNBQKBNR w KQq -
rnbqkbnr/ppp1pp1p/8/3p2p1/7P/N7/PPPPPPP1/R1BQKBNR w KQkq -
rnbqkbnr/ppp2ppp/8/3pp3/4P1P1/8/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/pppppp1p/5n2/6p1/3P4/N7/PPP1PPPP/R1BQKBNR w KQkq -

For example:
[d]rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
is likely won for the side to move (sample pv: Be2 Nf6 e5 Ng4 d4 g6 Nh3 d6 Nf4 Kf7 Bc4+ Kg7 e6 c6 Qe2 Qe8 Nd2 Kg8 Nf3 Bh6 Nh3 Bxc1 Rxc1 Nf6 Nfg5 h4 Bb3 Rh5 Nf7 d5 Nf4 Bxe6 Qxe6 Qxf7).

There are also more than 5000 positions with a ce near zero (5 centipawns or less in absolute value).

Here are ten from that segment:
rnbqkbnr/1ppp1ppp/4p3/p7/1P6/P7/2PPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/1p1ppppp/2p5/p7/1P6/8/PBPPPPPP/RN1QKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/4P3/6P1/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/ppp1pppp/5n2/3p4/8/P4N2/1PPPPPPP/RNBQKB1R w KQkq -
rnbqkb1r/pp1ppppp/5n2/2p5/2P5/6P1/PP1PPP1P/RNBQKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/8/3P1N2/PPP1PPPP/RNBQKB1R w KQkq -
rnbqkbnr/ppppp1pp/8/8/3Pp3/8/PPP2PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppp1ppp/2n5/4p3/8/5NP1/PPPPPP1P/RNBQKB1R w KQkq -
rnbqkbnr/pp2pppp/3p4/2p5/4P3/7P/PPPP1PP1/RNBQKBNR w KQkq -
rnbqkbnr/pp1p1ppp/4p3/2p5/5P2/5N2/PPPPP1PP/RNBQKB1R w KQkq -

The zero segment will be very drawish, so that even a weaker engine has pretty good drawing chances.

I am currently of the opinion that a book used for Elo evaluation should have a leaf position which is between 25 and 45 centipawns in absolute value for all exit positions.

Elo values close to zero have a disproportionate number of draws when played.

Elo values at or above 50 lead to more lopsided results for the side with an advantage.

Just a general guideline {opinion} and not rigorously proven.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
Laskos
Posts: 9412
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Low impact of opening phase in engine play?

Post by Laskos » Wed Apr 19, 2017 12:27 am

Dann Corbit wrote:There are more than 5000 positions in the 2 moves book with a centipawn value of 50 or more (which is pretty one sided). Large lopsided margins give the weaker engine winning chances.

Here are the top offenders:
rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/2n5/6p1/3P4/5P2/PPP1P1PP/RNBQKBNR w KQkq -
r1bqkbnr/pppppp1p/n7/6p1/8/1P2P3/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/ppppp2p/8/5pp1/8/2P2N2/PP1PPPPP/RNBQKB1R w KQkq -
rnbqkbnr/p1ppp1pp/1p6/5p2/1P2P3/8/P1PP1PPP/RNBQKBNR w KQkq -
rnbqkbnr/pppp1p1p/6p1/4p3/6P1/5P2/PPPPP2P/RNBQKBNR w KQkq -
rnbqkbn1/pppppppr/8/7p/8/2P4P/PP1PPPP1/RNBQKBNR w KQq -
rnbqkbnr/ppp1pp1p/8/3p2p1/7P/N7/PPPPPPP1/R1BQKBNR w KQkq -
rnbqkbnr/ppp2ppp/8/3pp3/4P1P1/8/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/pppppp1p/5n2/6p1/3P4/N7/PPP1PPPP/R1BQKBNR w KQkq -

For example:
[d]rnbqkbnr/ppppp1p1/8/5p1p/P3P3/8/1PPP1PPP/RNBQKBNR w KQkq -
is likely won for the side to move (sample pv: Be2 Nf6 e5 Ng4 d4 g6 Nh3 d6 Nf4 Kf7 Bc4+ Kg7 e6 c6 Qe2 Qe8 Nd2 Kg8 Nf3 Bh6 Nh3 Bxc1 Rxc1 Nf6 Nfg5 h4 Bb3 Rh5 Nf7 d5 Nf4 Bxe6 Qxe6 Qxf7).

There are also more than 5000 positions with a ce near zero (5 centipawns or less in absolute value).

Here are ten from that segment:
rnbqkbnr/1ppp1ppp/4p3/p7/1P6/P7/2PPPPPP/RNBQKBNR w KQkq -
rnbqkbnr/1p1ppppp/2p5/p7/1P6/8/PBPPPPPP/RN1QKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/4P3/6P1/PPPP1P1P/RNBQKBNR w KQkq -
rnbqkb1r/ppp1pppp/5n2/3p4/8/P4N2/1PPPPPPP/RNBQKB1R w KQkq -
rnbqkb1r/pp1ppppp/5n2/2p5/2P5/6P1/PP1PPP1P/RNBQKBNR w KQkq -
rnbqkbnr/pp1ppp1p/6p1/2p5/8/3P1N2/PPP1PPPP/RNBQKB1R w KQkq -
rnbqkbnr/ppppp1pp/8/8/3Pp3/8/PPP2PPP/RNBQKBNR w KQkq -
r1bqkbnr/pppp1ppp/2n5/4p3/8/5NP1/PPPPPP1P/RNBQKB1R w KQkq -
rnbqkbnr/pp2pppp/3p4/2p5/4P3/7P/PPPP1PP1/RNBQKBNR w KQkq -
rnbqkbnr/pp1p1ppp/4p3/2p5/5P2/5N2/PPPPP1PP/RNBQKB1R w KQkq -

The zero segment will be very drawish, so that even a weaker engine has pretty good drawing chances.

I am currently of the opinion that a book used for Elo evaluation should have a leaf position which is between 25 and 45 centipawns in absolute value for all exit positions.

Elo values close to zero have a disproportionate number of draws when played.

Elo values at or above 50 lead to more lopsided results for the side with an advantage.

Just a general guideline {opinion} and not rigorously proven.
Thanks Dann, I knew a bit these facts about 2moves, I myself built from this file "balanced" books and "unbalanced" ones. 2moves_v1.epd is a pretty rough file of nonsensical random 2-movers. As we saw here, it can screw opening phase play. Unbalanced positions in the range 0.5-1.3 pawns advantage can be useful in unbalanced play using pentanomial method for error bars and SPRT stop when the draw rate is high. The "golden range" for sensitivity is somewhere close to 45-50% draws. Balanced positions can give 70% and more for draw rate between closely related engines.

User avatar
Laskos
Posts: 9412
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Low impact of opening phase in engine play?

Post by Laskos » Wed Apr 19, 2017 12:50 am

Ras wrote:An engine opening book becomes most valuable if it is actually tuned to the engine. Then it makes the engine head for positions that it will like (beyond horizon) and avoid positions it would be unhappy with lateron.

Using an untuned book with "objectively" good moves can still lead the engine to lost positions if it cannot handle them correctly.

However, tuning is a lot of work. It is not sufficient to filter out all openings that have a good GM score.
That GM book I built was not for the purpose of playing well. It simply offered variety and reasonable moves, and I kept it as short as possible, but still to offer variety. 2moves_v1.epd assures variety, but it leads to nonsensical openings, and is screwing a bit the opening play. I wanted to see the contribution from openings in engine's play.

I played another experiment:

Stockfish - Andscacs ---> 211 Elo points difference
Stockfish - Stockfish first 12 moves + Andscacs the rest ---> 165 Elo points difference
Stockfish - Stockfish first 24 moves + Andscacs the rest ---> 89 Elo points difference

Contribution moves 1-12 to gameplay: 22%
Contribution moves 13-24 to gameplay: 36%

Humans concentrate on what they can tackle: openings and endgames. Openings seem to offer roughly 20% of game-play, endgames 15%. But the "soul" of the game seems to be moves 13-24, with 36% contribution. So, a "midgame book" would offer much more than an "opening book", only that this kind of "midgame book" is almost impossible to build.

Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Low impact of opening phase in engine play?

Post by Dann Corbit » Wed Apr 19, 2017 1:06 am

I see a pretty startling win difference for a small ce change past 50. For instance, the query {centered around 50 centipawns, +/- 5 cp}:

Code: Select all

select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between 50 and 60 
Gives:

Code: Select all

wwa	bwa	da
0.424612	0.275602	0.299784
And this query {centered around 125 centipawns, +/- 5 cp}:

Code: Select all

select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between 120 and 130
gives:

Code: Select all

wwa	bwa	da
0.517428	0.250179	0.232392
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Low impact of opening phase in engine play?

Post by Dann Corbit » Wed Apr 19, 2017 1:08 am

BTW, I think this result is not surprising:

Code: Select all

select sum(white_wins * 1.0)/sum(white_wins+black_wins+draws) as wwa, sum(black_wins * 1.0)/sum(white_wins+black_wins+draws) as bwa, sum(draws * 1.0)/sum(white_wins+black_wins+draws) as da from Epd where ce between -5 and 5

Code: Select all

wwa	bwa	da
0.322439	0.286634	0.390925
Because the small advantage allowed will usually be held by white (due to the tempo)
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Ras
Posts: 1142
Joined: Tue Aug 30, 2016 6:19 pm
Contact:

Re: Low impact of opening phase in engine play?

Post by Ras » Wed Apr 19, 2017 9:20 pm

Laskos wrote:That GM book I built was not for the purpose of playing well.
Then it is pointless to draw ELO conslusions. Besides, as I said, reasonable moves can still throw an engine into the sewers if it cannot cope with the resulting positions.

Example here, I have a hand-tuned book of about 20,000 moves. All of them manually verified for objective soundness (judged by Shredder) and whether they fit the CT800 style. Since this machine is primarily designed to play humans, not computers, it avoids closed positions like the plague because it "knows" that humans will try this trick. This knowledge is part of the asymmetric static eval, but it has to start with the opening book.

So, unless the book is tuned to emphasise the strong points of the engine and avoid its weak spots, the question whether it contributes to playing strength is not really meaningful, IMO.

The basic aspect of an untuned book is "only" time saving.

Post Reply