On Opening books in 2015

[Laskos]

Simplified claim: Opening books are pretty useless these days. E.g. Stockfish don't need no book at slow time controls.

I almost agree with you. However, to measure the book effect per se, one needs to depollute the measure from the time biais.
...
Generally, the larger the book, the crappier it is.

These are complete misconceptions of both of you. Are those your opinions? Do you have at least some empirical data? Two weeks ago I played 30 games at 40'/game on pretty strong hardware, the result was +5 =25 -0 for the Stockfish with a good, large book against Stockfish no book. I am curious what miracles would happen at 3 times longer TC like 120'/game? 2 plies deeper the engine realizes it constantly blunders many openings?

A typical win for book Stockfish:

[Event "40min 4cores"]
[Site "?"]
[Date "2015.03.07"]
[Round "7.1"]
[White "Stockfish 020315 64 BMI2 Book"]
[Black "Stockfish 020315 64 BMI2"]
[Result "1-0"]
[ECO "C11"]
[Annotator "0.00;0.23"]
[PlyCount "163"]
[EventDate "2015.03.07"]
[EventType "tourn"]
[Source "Kai"]
[TimeControl "2400"]

{Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz 3598 MHz W=36.7 plies; 8,989kN/s; PlaychessNightmare.ctg B=34.0 plies; 8,035kN/s}
1. e4 {B 0} e6 {0.23/28 64} 2. d4 {B 0} d5 {0.15/30 49} 3. Nc3 {B 0} Nf6 {0.12/29 40} 4. e5 {B 0} Nfd7 {0.09/31 42} 5. f4 {B 0} c5 {0.02/31 46} 6. Nf3 {B 0} Nc6 {0.00/30 36} 7. Be3 {B 0} Qb6 {0.00/32 26} 8. Na4 {B 0} Qa5+ {0.00/33 23} 9. Nc3 {B 0} Qb6 {0.00/36 31} 10. Na4 {B 0} Qa5+ {0.00/41 34} 11. c3 {B 0} cxd4 {0.00/32 38} 12. b4 {B 0} Nxb4 {0.00/33 23} 13. cxb4 {B 0} Bxb4+ {0.39/30 42} 14. Bd2 {B 0} Bxd2+ {0.32/31 43} 15. Nxd2 {B 0} b6 {0.29/34 41} 16. Bd3 {B 0} Ba6 {0.36/33 59} 17. Nb2 {B 0} Nc5 {0.53/33 41} 18. Bxa6 {B 0} Qxa6 {0.55/32 68} 19. Qe2 {B 0} Rc8 {0.57/32 34} 20. Qxa6 {B 0} Nxa6 {0.50/32 38} 21. Rb1 {B 0} O-O {0.58/33 90} 22. Nf3 {0.65/35 55} Nb4 {0.65/34 37}

...

1-0

At moves 6 to 12, the non-book Stockfish thinks it gained ground as black, giving 0.00 eval. While at move 22, exiting the book, it admits it blundered the opening, giving 0.65 to book Stockfish, which just exited the book. In 4 of the 5 wins of book Stockfish, the games went on these lines. Then, there are exiting positions which Stockfish considers equal (close to 0.00) while not being so, usually according to outcome statistics. The burden to prove your naive, misleading statements and LTC miracles (2 plies deeper) is entirely on you. I claim that maybe 5-10 years are needed for top engines to play openings reasonably and not need books. It's not about 2 plies more of the same Stockfish, your miraculous LTC. It's about 20+ plies more, horizon, lack of outcome predictability, unreliability of PV in the openings.

Please calm down .
With some caveats I think you made your main point by now and actual data suggests you are right in principle.
The caveats might be interesting though.
My own tests/games strongly suggest that there mainly is a problem in exactly one variation for nobook right now - the closed French with the black pieces ( no surprise) that Stockfish started to play about 8-10 dev versions ago.
Before that it usually went 1. ..e5 here and dealt with the resulting positions in a competent way IMHO.
Btw - I want to point out an obvious benefit of nobook that went somehow unnoticed so far. E.g. in a recent game with white Stockfish nobook went 1. d4 f5 and replied with 2. e3. This move can't be too bad for obvious reasons and it will most probably leave the opponent more or less on its own, too - in a position that an engine won't like.
Peter

You are right, many of the wins of the book are in the French Classical. I had a power failure, and the planned 100 game match at fixed time per move 30 sec (no time bias) was cut short and I am not going to restart it. The result after 45 games is:


MyFriends 10.1 is a fairly strong bin book I used in cutechess-cli. Although still not very statistically conclusive, a large part of the book engine wins are book wins even at this "unbiased" time control.
Two random French wins, one a complete walkover by the book over non-book Stockfish:

[pgn][Event "?"]
[Site "?"]
[Date "2015.03.19"]
[Round "23"]
[White "Stockfish 17.03.2015 MyFriends"]
[Black "Stockfish 17.03.2015 No Book"]
[Result "1-0"]
[ECO "C11"]
[PlyCount "100"]
[EventDate "2015.??.??"]

1. e4 {book} e6 {-0.17/23 30} 2. d4 {book} d5 {-0.22/23 30} 3. Nc3 {book} Nf6 {-0.12/24 30} 4. e5 {book} Nfd7 {0.00/26 30} 5. f4 {book} c5 {0.00/28 30} 6. Nf3 {book} Nc6 {0.00/28 30} 7. Be3 {book} Qb6 {0.00/28 30} 8. Na4 {book} Qa5+ {0.00/29 30} 9. c3 {book} cxd4 {0.00/27 30} 10. b4 {book} Nxb4 {0.00/27 30} 11. cxb4 {book} Bxb4+ {-0.06/26 30} 12. Bd2 {book} Bxd2+ {0.00/27 30} 13. Nxd2 {book} b6 {-0.22/25 30} 14. Bd3 {book} Ba6 {-0.30/27 30} 15. Nb2 {book} Nc5 {-0.29/26 30} 16. Bxa6 {book} Qxa6 {-0.29/28 30} 17. Qe2 {book} Rc8 {-0.43/26 30} 18. Qxa6 {book} Nxa6 {-0.57/26 30} 19. Kd1 {book} f6 {-0.48/25 30} 20. Nf3 {book} fxe5 {-0.63/24 30} 21. fxe5 {book} Rf8 {-0.62/26 30} 22. Nxd4 {book} Rf2 {-0.62/25 30} 23. Nd3 {book} Rxg2 {-0.62/26 30} 24. Nf4 {book} Rg4 {-0.70/26 30} 25. Nfxe6 {book} Nc5 {-0.71/26 30} 26. h3 {book} Nxe6 {-0.93/27 30} 27. hxg4 {book} Nxd4 {-0.89/29 30} 28. Rxh7 {book} Kf7 {-0.98/26 30} 29. Rc1 {book} Rxc1+ {-0.85/29 30} 30. Kxc1 {book} Nc6 {-0.95/32 30} 31. e6+ {book} Kf6 {-0.92/35 30} 32. Kd2 {book} Ne7 {-1.32/29 30} 33. Ke3 {book} g6 {-1.41/29 30} 34. Rh8 {book} Kxe6 {-1.53/29 30} 35. Kf4 {book} d4 {-1.71/28 30} 36. Ke4 {book} d3 {-1.79/31 30} 37. Kxd3 {1.87/31 30} Nd5 {-1.85/31 30} *[/pgn]

The engine left the book at move 37.
37. Kxd3 {1.87/31 30} Nd5 {-1.85/31 30}
[d]7R/p3n3/1p2k1p1/8/4K1P1/3p4/P7/8 w - - 0 1[/d]
Five minute analysis by Stockfish 170315 64 BMI2:
37.Kxd3
+- (7.54 ++) Depth: 43/72 00:05:18 3735MN
(Kai, 19.03.2015)
Stockfish got destroyed by the book.

Another French win of the book, a bit milder:

[pgn][Event "?"]
[Site "?"]
[Date "2015.03.19"]
[Round "7"]
[White "Stockfish 17.03.2015 MyFriends"]
[Black "Stockfish 17.03.2015 No Book"]
[Result "1-0"]
[ECO "C11"]
[PlyCount "130"]
[EventDate "2015.??.??"]

1. e4 {book} e6 {-0.17/23 30} 2. d4 {book} d5 {-0.12/23 30} 3. Nc3 {book} Nf6 {-0.08/24 30} 4. e5 {book} Nfd7 {0.00/27 30} 5. f4 {book} c5 {0.00/28 30} 6. Nf3 {book} Nc6 {0.00/28 30} 7. Be3 {book} Qb6 {0.00/29 30} 8. Na4 {book} Qa5+ {0.00/30 30} 9. c3 {book} cxd4 {-0.05/26 30} 10. b4 {book} Nxb4 {0.00/29 30} 11. cxb4 {book} Bxb4+ {0.00/27 30} 12. Bd2 {book} Bxd2+ {-0.23/24 30} 13. Nxd2 {book} b6 {-0.30/26 30} 14. Bd3 {book} Ba6 {-0.30/26 30} 15. Nb2 {book} Nc5 {-0.36/27 30} 16. Bxa6 {book} Qxa6 {-0.35/27 30} 17. Qe2 {book} Rc8 {-0.44/25 30} 18. Qxa6 {book} Nxa6 {-0.50/27 30} 19. Kd1 {book} f6 {-0.50/25 30} 20. Nf3 {book} Kd7 {-0.58/25 30} 21. Kd2 {0.67/23 30} Rhf8 {-0.86/23 30} *[/pgn]

The engine left the book at move 21.
21. Kd2 {0.67/23 30} Rhf8 {-0.86/23 30}
[d]2r4r/p2k2pp/np2pp2/3pP3/3p1P2/5N2/PN4PP/R2K3R w - - 0 1[/d]
Five minute analysis by Stockfish 170315 64 BMI2:
21.Kd2 Rhf8
+/- (0.86 ++) Depth: 36/62 00:05:43 2740MN
(Kai, 19.03.2015)
A book win too.

I can upload somewhere the PGN of 45 games at 30s/move for those interested.

The advantages of the book seem to be:
1/ Little time is used in the opening. That gives often 40%-50% more time to spend in the remaining parts of the games. Almost none use fixed time per move, so time consumed in the opening is important.
2/ Book often overplays the engine in the opening even at fixed time per move.

Maybe Stockfish can be tweaked to not play losing lines of the French and such. Maybe it's even easy to tweak it, but as of now, it gets often overplayed by good books.

[zullil]

I think cxd4 was a mistake.
I believe that with a good hardware stockfish can find c4
please check it

[D]r1b1kb1r/pp1n1ppp/2n1p3/q1ppP3/N2P1P2/2P1BN2/PP4PP/R2QKB1R b KQkq - 0 9

With 16 threads and 4 GB hash SF chooses c4 within 3 minutes:

Code: Select all

info depth 32 seldepth 48 multipv 1 score cp -25 lowerbound nodes 2071104160 nps 16742147 hashfull 999 tbhits 0 time 123706 pv c5c4
info depth 32 currmove c5c4 currmovenumber 1
info nodes 3067792506 time 180000
bestmove c5c4

[Laskos]

I think cxd4 was a mistake.
I believe that with a good hardware stockfish can find c4
please check it

[D]r1b1kb1r/pp1n1ppp/2n1p3/q1ppP3/N2P1P2/2P1BN2/PP4PP/R2QKB1R b KQkq - 0 9

With 16 threads and 4 GB hash SF chooses c4 within 3 minutes:

Code: Select all

info depth 32 seldepth 48 multipv 1 score cp -25 lowerbound nodes 2071104160 nps 16742147 hashfull 999 tbhits 0 time 123706 pv c5c4
info depth 32 currmove c5c4 currmovenumber 1
info nodes 3067792506 time 180000
bestmove c5c4

On 4 cores in some 10 minutes:
Analysis by Stockfish 170315 64 BMI2:
9...c4 10.b4
+/= (0.31 --) Depth: 36/55 00:10:54 3818MN

"Let's check" has a Komodo 8 depth=36 (very long analysis) with cxd4.

[Leto]

I have a book so big that it gives me a huge advantage in time on playchess. In this game I'm in now my engine's first move out of book was at move 57, my opponent's first move out of book was at move 32. When I finally went out of book I had two more minutes than my opponent in a 5 minute blitz game. My opponent managed to get a draw but it's still a significant advantage to have.

Then again most big tournaments (such as TCEC) and engine testers use small books so the engine itself is tested rather than the book. So even if we start solving opening lines it'd still be very important to improve an engine's handling of the opening phase of the game.

[Peter Berger]

If there are a lot of losses in a single variation for nobook, this can lead to wrong impressions on book value though.

I'd like to show a counter example where nobook looks pretty smart to me:

r1bqnrk1/1pp2pb1/3p2pp/p1nPp3/2P1P3/P1N1B2P/1P1N1PP1/R2QKB1R w KQ - 0 12

Stockfish nobook played 12. g4 here and went on to win easily - a move that would be unlikely to be found in an existing opening book.

[Event "120m +10s"]
[Site "2* Celeron @2.16GHz"]
[Date "2015.03.21"]
[Round "?"]
[White "Stockfish 200315 64 POPCNT nobook"]
[WhiteElo ""]
[Black "Crafty 24.1"]
[BlackElo ""]
[Result "1-0"]

1. d4 Nf6 2. Nf3 g6 3. c4 Bg7 4. Nc3 O-O 5. e4 d6 6. h3 e5 7. d5
Na6 8. Bg5 h6 9. Be3 Nc5 10. Nd2 a5 11. a3 Ne8 12. g4 Qh4 13. Rg1
Qe7 14. b4 axb4 15. axb4 Rxa1 16. Qxa1 Na6 17. Qa3 Kh8 18. c5
dxc5 19. bxc5 f5 20. exf5 gxf5 21. gxf5 Bxf5 22. Nde4 Qh4 23. c6
bxc6 24. dxc6 Nb8 25. Bc5 Rg8 26. Bc4 Bxe4 27. Bxg8 Kxg8 28. Qb3+
Kh7 29. Qf7 Qf4 30. Qxf4 exf4 31. Nxe4 Nxc6 32. Ke2 Bb2 33. Kf3
Be5 34. Rc1 Bb2 35. Rc2 Be5 36. Nc3 Kg6 37. Nd5 Kf5 38. Be3
Nf6 39. Nxf4 Nd8 40. Rc5 Ne4 41. Rd5 Ng5+ 42. Ke2 Ngf7 43. Nd3
Ke6 44. Ra5 Bd6 45. Nc5+ Ke7 46. Ne4 Kd7 47. Nxd6 cxd6 48. Rh5
Nc6 49. Bxh6 Ke6 50. Be3 Ne7 51. Ra5 Nf5 52. Ra4 Ng7 53. h4
Nh5 54. Bg5 Ne5 55. f4 Ng4 56. Ra2 Nh2 57. Kf2 Kf7 58. Kg2 Ke6
59. Kxh2 Kf7 60. f5 Kg8 61. f6 Nxf6 62. Bxf6 Kf7 63. Ba1 Ke7
64. h5 Ke6 65. h6 d5 66. h7 d4 67. h8=Q Kd5 68. Qxd4+ Kc6 69. Qc4+
Kd6 70. Rd2+ Ke7 71. Qc7+ Ke6 72. Qd7# 1-0

Of course I could also give many examples about bad book moves I saw at my home that would weaken Stockfish compaired to nobook but this isn't necessary as this is would be about book quality only. As I don't have access to the books you are using I can't judge on their quality by myself.

Peter