Perfect book 2023 no longer perfect?

[nmcrazyim5]

With the release of a new official version, I was curious to see how well Stockfish would perform against other top engines using a well-balanced opening book like Perfect Book 2023. So I ran a 30s + 1s 100-pair gauntlet each against the latest Obsidian, Berserk, and Dragon. There may have been some SSS at play here, but the fact that Stockfish took over 10% of the games against each engine (over 20% vs Dragon 3.3) was a bit unexpected. Does this indicate that Perfect Book 2023 is no longer as bulletproof or has Stockfish made a major leap?

GUI: Cutechess
CPU: Ryzen 9 7950x
Threads: 8
Hash: 4 GB each
Tablebase: 5-men Syzygy
Ponder OFF
Time Control: 30 sec + 1 sec/move
Book: Perfect Book 2023

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw 
   0 Stockfish 17.1                 43      16     600   57.3%   85.3%
   1 Obsidian 15.09                -35      21     200   45.0%   90.0%
   2 Berserk dev_250307            -38      22     200   44.5%   89.0%
   3 Dragon 3.3                    -69      36     200   38.5%   77.0%

600 of 600 games finished.

[Dann Corbit]

Perfect book 2023 is just a name. Here is a quote from Sedat's site: "Balsa, Perfect or Unique book's openings would never gain higher points vs Current Top trend books, because I've used to play many various openings.."
It is not clear if SF used the book or not.
At very fast time control all engines, including SF will make bad decisions.

[chesskobra]

I have extracted top 104 lines (at 8-ply) by frequency from the 40h database. Each line has appeared at least 1000 times in the 40h database. How would an opening book based on them be? Would it be quite balanced? The first field on each line is the frequency of a line.

Code: Select all

  38651 e4 c5 Nf3 d6 d4 cxd4 Nxd4 Nf6 *
  32340 e4 e5 Nf3 Nc6 Bb5 a6 Ba4 Nf6 *
  18268 d4 Nf6 c4 g6 Nc3 Bg7 e4 d6 *
  11109 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 Nf6 *
   9455 e4 c5 Nf3 e6 d4 cxd4 Nxd4 a6 *
   8946 e4 c5 Nf3 e6 d4 cxd4 Nxd4 Nc6 *
   7006 d4 Nf6 c4 e6 Nc3 Bb4 e3 O-O *
   6177 d4 Nf6 c4 g6 Nc3 d5 cxd5 Nxd5 *
   5099 e4 c5 Nf3 e6 d4 cxd4 Nxd4 Nf6 *
   5026 e4 e5 Nf3 Nf6 Nxe5 d6 Nf3 Nxe4 *
   4991 e4 e5 Nf3 Nc6 Bb5 Nf6 O-O Nxe4 *
   4407 d4 Nf6 c4 g6 Nc3 d5 Nf3 Bg7 *
   4334 d4 Nf6 c4 e6 Nf3 b6 g3 Ba6 *
   4166 d4 d5 c4 c6 Nf3 Nf6 Nc3 e6 *
   4149 d4 Nf6 c4 e6 Nc3 Bb4 Qc2 O-O *
   3933 e4 e6 d4 d5 Nc3 Bb4 e5 c5 *
   3767 d4 d5 c4 c6 Nf3 Nf6 Nc3 dxc4 *
   3569 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 g6 *
   3349 e4 e5 Nf3 Nc6 Bb5 a6 Ba4 d6 *
   3301 e4 e6 d4 d5 Nc3 Nf6 e5 Nfd7 *
   3189 e4 c6 d4 d5 Nc3 dxe4 Nxe4 Bf5 *
   3081 d4 Nf6 c4 e6 Nf3 b6 g3 Bb7 *
   2974 d4 Nf6 c4 e6 Nc3 Bb4 e3 c5 *
   2964 e4 e5 Nf3 Nc6 Bc4 Bc5 c3 Nf6 *
   2790 e4 e5 Nf3 Nc6 Bb5 a6 Bxc6 dxc6 *
   2743 e4 e6 d4 d5 Nd2 Nf6 e5 Nfd7 *
   2671 e4 c6 d4 d5 e5 Bf5 Nf3 e6 *
   2570 d4 Nf6 c4 e6 Nf3 d5 Nc3 Bb4 *
   2498 d4 Nf6 c4 c5 d5 b5 cxb5 a6 *
   2473 d4 Nf6 c4 c5 d5 e6 Nc3 exd5 *
   2450 d4 Nf6 c4 g6 g3 Bg7 Bg2 O-O *
   2450 d4 Nf6 c4 e6 Nf3 d5 Nc3 Be7 *
   2394 e4 c6 d4 d5 exd5 cxd5 c4 Nf6 *
   2223 d4 d5 c4 c6 Nc3 Nf6 e3 e6 *
   2220 e4 e5 Nf3 Nc6 Bb5 Nf6 d3 Bc5 *
   2203 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 e6 *
   2058 e4 e5 Nf3 Nc6 d4 exd4 Nxd4 Bc5 *
   2031 d4 Nf6 c4 g6 Nf3 Bg7 g3 O-O *
   2025 e4 c5 Nf3 d6 Bb5+ Bd7 Bxd7+ Qxd7 *
   2000 e4 e5 Nf3 Nc6 d4 exd4 Nxd4 Nf6 *
   1999 e4 e6 d4 d5 Nc3 Nf6 Bg5 dxe4 *
   1990 e4 e6 d4 d5 e5 c5 c3 Nc6 *
   1990 e4 c5 Nf3 Nc6 Bb5 g6 O-O Bg7 *
   1910 d4 Nf6 c4 e6 Nc3 Bb4 Qc2 d5 *
   1872 d4 Nf6 Nf3 g6 c4 Bg7 Nc3 O-O *
   1862 d4 d5 c4 c6 Nf3 Nf6 Nc3 a6 *
   1852 Nf3 Nf6 c4 g6 Nc3 Bg7 e4 d6 *
   1815 e4 c5 Nf3 d6 d4 Nf6 Nc3 cxd4 *
   1805 Nf3 Nf6 g3 g6 Bg2 Bg7 O-O O-O *
   1787 d4 Nf6 Nf3 g6 g3 Bg7 Bg2 O-O *
   1775 e4 d6 d4 Nf6 Nc3 e5 Nf3 Nbd7 *
   1750 d4 Nf6 c4 e6 Nc3 Bb4 Qc2 c5 *
   1733 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 e5 *
   1726 d4 Nf6 c4 g6 Nc3 Bg7 e4 O-O *
   1687 Nf3 Nf6 c4 g6 g3 Bg7 Bg2 O-O *
   1684 d4 Nf6 c4 e6 Nc3 Bb4 e3 b6 *
   1673 e4 c5 Nc3 Nc6 g3 g6 Bg2 Bg7 *
   1673 d4 d5 c4 c6 Nf3 Nf6 e3 Bf5 *
   1663 e4 c6 d4 d5 Nd2 dxe4 Nxe4 Bf5 *
   1560 d4 d5 c4 dxc4 Nf3 Nf6 e3 e6 *
   1514 e4 c5 c3 Nf6 e5 Nd5 d4 cxd4 *
   1509 d4 Nf6 c4 e6 Nf3 b6 a3 Bb7 *
   1469 e4 e6 d4 d5 Nc3 Bb4 e5 Ne7 *
   1469 e4 e5 Nf3 Nc6 Bc4 Nf6 d3 Bc5 *
   1467 e4 c5 Nf3 Nc6 Bb5 e6 O-O Nge7 *
   1461 d4 d5 c4 e6 Nc3 Be7 Nf3 Nf6 *
   1439 d4 Nf6 c4 e6 Nc3 d5 cxd5 exd5 *
   1415 d4 d5 c4 e6 Nc3 Nf6 Bg5 Be7 *
   1405 d4 Nf6 c4 e6 Nf3 d5 Nc3 c6 *
   1368 e4 e6 d4 d5 Nd2 c5 exd5 Qxd5 *
   1348 e4 e5 Nf3 Nc6 Bc4 Nf6 d3 Be7 *
   1348 e4 d6 d4 Nf6 Nc3 g6 f4 Bg7 *
   1314 e4 c6 d4 d5 Nc3 dxe4 Nxe4 Nd7 *
   1308 d4 Nf6 c4 g6 Nc3 d5 Bf4 Bg7 *
   1305 e4 e6 d4 d5 Nc3 Nf6 Bg5 Be7 *
   1280 e4 e5 Nf3 Nc6 Nc3 Nf6 d4 exd4 *
   1277 d4 d5 c4 e6 Nc3 Nf6 cxd5 exd5 *
   1265 Nf3 Nf6 c4 g6 Nc3 d5 cxd5 Nxd5 *
   1256 d4 Nf6 c4 e6 Nf3 Bb4+ Bd2 Qe7 *
   1239 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 Qc7 *
   1215 e4 e6 d4 d5 Nd2 c5 exd5 exd5 *
   1199 e4 c5 c3 d5 exd5 Qxd5 d4 Nf6 *
   1195 d4 Nf6 c4 e6 Nf3 c5 d5 exd5 *
   1140 d4 Nf6 c4 c5 d5 e5 Nc3 d6 *
   1138 e4 e6 d4 d5 Nd2 c5 Ngf3 cxd4 *
   1135 d4 Nf6 c4 e6 Nf3 d5 Nc3 dxc4 *
   1131 d4 Nf6 c4 e6 Nf3 b6 Nc3 Bb7 *
   1113 e4 e5 Nf3 Nc6 Bc4 Bc5 O-O Nf6 *
   1099 e4 e5 Nf3 Nf6 d4 Nxe4 Bd3 d5 *
   1078 c4 Nf6 Nc3 g6 e4 d6 d4 Bg7 *
   1077 d4 Nf6 c4 e6 Nf3 d5 g3 Bb4+ *
   1069 d4 Nf6 Nf3 g6 c4 Bg7 g3 O-O *
   1066 d4 Nf6 c4 e6 Nc3 d5 Bg5 Be7 *
   1059 d4 Nf6 c4 e6 Nf3 d5 Nc3 c5 *
   1050 e4 e5 Nf3 Nc6 Bb5 Nf6 d3 d6 *
   1049 e4 d6 d4 Nf6 Nc3 g6 Nf3 Bg7 *
   1031 e4 e6 d4 d5 Nc3 dxe4 Nxe4 Nd7 *
   1026 e4 e6 d4 d5 Nc3 Nf6 Bg5 Bb4 *
   1024 d4 d5 c4 c6 Nc3 Nf6 Nf3 e6 *
   1019 e4 c6 d4 d5 exd5 cxd5 Bd3 Nc6 *
   1018 e4 Nf6 e5 Nd5 d4 d6 Nf3 Bg4 *
   1018 d4 Nf6 c4 e6 g3 c5 d5 exd5 *
   1004 d4 d5 c4 c6 Nc3 Nf6 e3 a6 *
   1000 d4 d5 c4 e6 Nc3 c5 cxd5 exd5 *

[Uri Blass]

With the release of a new official version, I was curious to see how well Stockfish would perform against other top engines using a well-balanced opening book like Perfect Book 2023. So I ran a 30s + 1s 100-pair gauntlet each against the latest Obsidian, Berserk, and Dragon. There may have been some SSS at play here, but the fact that Stockfish took over 10% of the games against each engine (over 20% vs Dragon 3.3) was a bit unexpected. Does this indicate that Perfect Book 2023 is no longer as bulletproof or has Stockfish made a major leap?

GUI: Cutechess
CPU: Ryzen 9 7950x
Threads: 8
Hash: 4 GB each
Tablebase: 5-men Syzygy
Ponder OFF
Time Control: 30 sec + 1 sec/move
Book: Perfect Book 2023

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw 
   0 Stockfish 17.1                 43      16     600   57.3%   85.3%
   1 Obsidian 15.09                -35      21     200   45.0%   90.0%
   2 Berserk dev_250307            -38      22     200   44.5%   89.0%
   3 Dragon 3.3                    -69      36     200   38.5%   77.0%

600 of 600 games finished.

I suspect the problem is that stockfish's opponents are not perfect.

I have 2 ideas to test if you are interested.
1)Stockfish with the perfect book against the same oppponets you chose with no book to see if no book is better for the opponents relative to perfect book or not.

2)Stockfrish against Stockfish when both use the perfect book to see what is the percentage of draws.

[Ciekce]

It is not clear if SF used the book or not.

this book is not used in SF development (not that it matters?)

At very fast time control all engines, including SF will make bad decisions.

30+1 is far from "very fast" for engines

essentially, even though this book is pre-drawn, sf is Just Better™

[nmcrazyim5]

30+1 is far from "very fast" for engines
essentially, even though this book is pre-drawn, sf is Just Better™

Exactly my conclusions too. 30s + 1s is 3x slower than ipman for instance and ipman uses UHO. Perfect book has all low exits (<0.4) and is very much like a top GM repertoire. As chess evolves books and opening theory have to change too and seems like SF is breaking away from the rest a bit.
I remember L Kaufman mentioning over 95% draw rate around the time of Dragon 3 release which was quite true then, I even ran similar tests against SF (v14?) and most balanced book exits were all drawn (~97-98% draws). Seems like chess has more to offer (which is great news for both the players and devs)

[nmcrazyim5]

I have extracted top 104 lines (at 8-ply) by frequency from the 40h database. Each line has appeared at least 1000 times in the 40h database. How would an opening book based on them be? Would it be quite balanced? The first field on each line is the frequency of a line.

Code: Select all

  38651 e4 c5 Nf3 d6 d4 cxd4 Nxd4 Nf6 *
  32340 e4 e5 Nf3 Nc6 Bb5 a6 Ba4 Nf6 *
  18268 d4 Nf6 c4 g6 Nc3 Bg7 e4 d6 *
  11109 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 Nf6 *
   9455 e4 c5 Nf3 e6 d4 cxd4 Nxd4 a6 *
   8946 e4 c5 Nf3 e6 d4 cxd4 Nxd4 Nc6 *
   7006 d4 Nf6 c4 e6 Nc3 Bb4 e3 O-O *
   6177 d4 Nf6 c4 g6 Nc3 d5 cxd5 Nxd5 *
   5099 e4 c5 Nf3 e6 d4 cxd4 Nxd4 Nf6 *
   5026 e4 e5 Nf3 Nf6 Nxe5 d6 Nf3 Nxe4 *
   4991 e4 e5 Nf3 Nc6 Bb5 Nf6 O-O Nxe4 *
   4407 d4 Nf6 c4 g6 Nc3 d5 Nf3 Bg7 *
   4334 d4 Nf6 c4 e6 Nf3 b6 g3 Ba6 *
   4166 d4 d5 c4 c6 Nf3 Nf6 Nc3 e6 *
   4149 d4 Nf6 c4 e6 Nc3 Bb4 Qc2 O-O *
   3933 e4 e6 d4 d5 Nc3 Bb4 e5 c5 *
   3767 d4 d5 c4 c6 Nf3 Nf6 Nc3 dxc4 *
   3569 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 g6 *
   3349 e4 e5 Nf3 Nc6 Bb5 a6 Ba4 d6 *
   3301 e4 e6 d4 d5 Nc3 Nf6 e5 Nfd7 *
   3189 e4 c6 d4 d5 Nc3 dxe4 Nxe4 Bf5 *
   3081 d4 Nf6 c4 e6 Nf3 b6 g3 Bb7 *
   2974 d4 Nf6 c4 e6 Nc3 Bb4 e3 c5 *
   2964 e4 e5 Nf3 Nc6 Bc4 Bc5 c3 Nf6 *
   2790 e4 e5 Nf3 Nc6 Bb5 a6 Bxc6 dxc6 *
   2743 e4 e6 d4 d5 Nd2 Nf6 e5 Nfd7 *
   2671 e4 c6 d4 d5 e5 Bf5 Nf3 e6 *
   2570 d4 Nf6 c4 e6 Nf3 d5 Nc3 Bb4 *
   2498 d4 Nf6 c4 c5 d5 b5 cxb5 a6 *
   2473 d4 Nf6 c4 c5 d5 e6 Nc3 exd5 *
   2450 d4 Nf6 c4 g6 g3 Bg7 Bg2 O-O *
   2450 d4 Nf6 c4 e6 Nf3 d5 Nc3 Be7 *
   2394 e4 c6 d4 d5 exd5 cxd5 c4 Nf6 *
   2223 d4 d5 c4 c6 Nc3 Nf6 e3 e6 *
   2220 e4 e5 Nf3 Nc6 Bb5 Nf6 d3 Bc5 *
   2203 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 e6 *
   2058 e4 e5 Nf3 Nc6 d4 exd4 Nxd4 Bc5 *
   2031 d4 Nf6 c4 g6 Nf3 Bg7 g3 O-O *
   2025 e4 c5 Nf3 d6 Bb5+ Bd7 Bxd7+ Qxd7 *
   2000 e4 e5 Nf3 Nc6 d4 exd4 Nxd4 Nf6 *
   1999 e4 e6 d4 d5 Nc3 Nf6 Bg5 dxe4 *
   1990 e4 e6 d4 d5 e5 c5 c3 Nc6 *
   1990 e4 c5 Nf3 Nc6 Bb5 g6 O-O Bg7 *
   1910 d4 Nf6 c4 e6 Nc3 Bb4 Qc2 d5 *
   1872 d4 Nf6 Nf3 g6 c4 Bg7 Nc3 O-O *
   1862 d4 d5 c4 c6 Nf3 Nf6 Nc3 a6 *
   1852 Nf3 Nf6 c4 g6 Nc3 Bg7 e4 d6 *
   1815 e4 c5 Nf3 d6 d4 Nf6 Nc3 cxd4 *
   1805 Nf3 Nf6 g3 g6 Bg2 Bg7 O-O O-O *
   1787 d4 Nf6 Nf3 g6 g3 Bg7 Bg2 O-O *
   1775 e4 d6 d4 Nf6 Nc3 e5 Nf3 Nbd7 *
   1750 d4 Nf6 c4 e6 Nc3 Bb4 Qc2 c5 *
   1733 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 e5 *
   1726 d4 Nf6 c4 g6 Nc3 Bg7 e4 O-O *
   1687 Nf3 Nf6 c4 g6 g3 Bg7 Bg2 O-O *
   1684 d4 Nf6 c4 e6 Nc3 Bb4 e3 b6 *
   1673 e4 c5 Nc3 Nc6 g3 g6 Bg2 Bg7 *
   1673 d4 d5 c4 c6 Nf3 Nf6 e3 Bf5 *
   1663 e4 c6 d4 d5 Nd2 dxe4 Nxe4 Bf5 *
   1560 d4 d5 c4 dxc4 Nf3 Nf6 e3 e6 *
   1514 e4 c5 c3 Nf6 e5 Nd5 d4 cxd4 *
   1509 d4 Nf6 c4 e6 Nf3 b6 a3 Bb7 *
   1469 e4 e6 d4 d5 Nc3 Bb4 e5 Ne7 *
   1469 e4 e5 Nf3 Nc6 Bc4 Nf6 d3 Bc5 *
   1467 e4 c5 Nf3 Nc6 Bb5 e6 O-O Nge7 *
   1461 d4 d5 c4 e6 Nc3 Be7 Nf3 Nf6 *
   1439 d4 Nf6 c4 e6 Nc3 d5 cxd5 exd5 *
   1415 d4 d5 c4 e6 Nc3 Nf6 Bg5 Be7 *
   1405 d4 Nf6 c4 e6 Nf3 d5 Nc3 c6 *
   1368 e4 e6 d4 d5 Nd2 c5 exd5 Qxd5 *
   1348 e4 e5 Nf3 Nc6 Bc4 Nf6 d3 Be7 *
   1348 e4 d6 d4 Nf6 Nc3 g6 f4 Bg7 *
   1314 e4 c6 d4 d5 Nc3 dxe4 Nxe4 Nd7 *
   1308 d4 Nf6 c4 g6 Nc3 d5 Bf4 Bg7 *
   1305 e4 e6 d4 d5 Nc3 Nf6 Bg5 Be7 *
   1280 e4 e5 Nf3 Nc6 Nc3 Nf6 d4 exd4 *
   1277 d4 d5 c4 e6 Nc3 Nf6 cxd5 exd5 *
   1265 Nf3 Nf6 c4 g6 Nc3 d5 cxd5 Nxd5 *
   1256 d4 Nf6 c4 e6 Nf3 Bb4+ Bd2 Qe7 *
   1239 e4 c5 Nf3 Nc6 d4 cxd4 Nxd4 Qc7 *
   1215 e4 e6 d4 d5 Nd2 c5 exd5 exd5 *
   1199 e4 c5 c3 d5 exd5 Qxd5 d4 Nf6 *
   1195 d4 Nf6 c4 e6 Nf3 c5 d5 exd5 *
   1140 d4 Nf6 c4 c5 d5 e5 Nc3 d6 *
   1138 e4 e6 d4 d5 Nd2 c5 Ngf3 cxd4 *
   1135 d4 Nf6 c4 e6 Nf3 d5 Nc3 dxc4 *
   1131 d4 Nf6 c4 e6 Nf3 b6 Nc3 Bb7 *
   1113 e4 e5 Nf3 Nc6 Bc4 Bc5 O-O Nf6 *
   1099 e4 e5 Nf3 Nf6 d4 Nxe4 Bd3 d5 *
   1078 c4 Nf6 Nc3 g6 e4 d6 d4 Bg7 *
   1077 d4 Nf6 c4 e6 Nf3 d5 g3 Bb4+ *
   1069 d4 Nf6 Nf3 g6 c4 Bg7 g3 O-O *
   1066 d4 Nf6 c4 e6 Nc3 d5 Bg5 Be7 *
   1059 d4 Nf6 c4 e6 Nf3 d5 Nc3 c5 *
   1050 e4 e5 Nf3 Nc6 Bb5 Nf6 d3 d6 *
   1049 e4 d6 d4 Nf6 Nc3 g6 Nf3 Bg7 *
   1031 e4 e6 d4 d5 Nc3 dxe4 Nxe4 Nd7 *
   1026 e4 e6 d4 d5 Nc3 Nf6 Bg5 Bb4 *
   1024 d4 d5 c4 c6 Nc3 Nf6 Nf3 e6 *
   1019 e4 c6 d4 d5 exd5 cxd5 Bd3 Nc6 *
   1018 e4 Nf6 e5 Nd5 d4 d6 Nf3 Bg4 *
   1018 d4 Nf6 c4 e6 g3 c5 d5 exd5 *
   1004 d4 d5 c4 c6 Nc3 Nf6 e3 a6 *
   1000 d4 d5 c4 e6 Nc3 c5 cxd5 exd5 *

Most lines you found through aggregation are the first few plies of standard openings like Spanish, Open Sicilian, Queen's Gambit, Slav etc. This makes sense because those positions are solid for black and make them a good candidate for a balanced book with low exits.

[jefk]

nowadays Gruenfeld defense is considered to be more solid
than King's Indian (with computers/engines playing)

[chesskobra]

I have extracted top 104 lines (at 8-ply) by frequency from the 40h database. Each line has appeared at least 1000 times in the 40h database. How would an opening book based on them be? Would it be quite balanced? The first field on each line is the frequency of a line.

Most lines you found through aggregation are the first few plies of standard openings like Spanish, Open Sicilian, Queen's Gambit, Slav etc. This makes sense because those positions are solid for black and make them a good candidate for a balanced book with low exits.

In fact they are the most frequent lines from a database of human games (40h db of Norman Pollock). So they are standard openings. My thought is that if a line is played 1000s of times by GMs, it is unlikely that it gives significant advantage to one side. Isn't this the objective of the perfect book?