The scaling with time of opening books

[Laskos]

There is not much talk here about the importance of opening books, although online events and WCCC are heavily dependent on them. I was curious a bit about their impact. I saw two dismissive opinions made here:

1/ Opening books only save time.
2/ With increased time control their importance diminishes, as engines themselves become better at openings.

I took a good (and small) Polyglot opening book BookX.bin by Adam Hair with Komodo and played at different time controls against Komodo no book. Games are from ultra-fast 20s+0.2s to blitz 360s+3.6s on one core. The results seem to contradict both 1/ and 2/.

ELO difference:

20s+0.2s --> 61.7
60s+0.6s --> 59.6
180s+1.8s --> 70.4
360s+3.6s --> 79.5

Code: Select all

20s+0.2s
Score of K2 vs K1: 105 - 34 - 261  [0.589] 400
ELO difference: 61.68 +/- 20.17

60s+0.6s
Score of K2 vs K1: 109 - 41 - 250  [0.585] 400
ELO difference: 59.64 +/- 20.62

180s+1.8s
Score of K2 vs K1: 20 - 4 - 56  [0.600] 80
ELO difference: 70.44 +/- 40.59

360s+3.6s
Score of K2 vs K1: 22 - 4 - 54  [0.613] 80
ELO difference: 79.53 +/- 42.10

Opening books do indeed save time, about 15-20% from total time used. But the gain in the case 360s+3.6s is about equal to doubling in time (100% difference), which is about 80 ELO points for Komodo at this blitz time control. So, book does more than saving time. Also, the importance of the book with longer time control seems to increase, not decrease.

[Dan Cooper]

Perhaps time spent in the opening has a disproportionate elo cost compared to time spent elsewhere. Also of interest is the constant observation of how Komodo plays so quickly in the opening in TCEC, yet appears to achieve fantastic results with that time management.

I think it would be interesting to rerun this test, using a dual time control. Assuming your original test had a 8-move book, do something like 8-16 seconds for the first 8 moves, then 360s +3.6 for the rest of the game.

[Ras]

Opening books do two more things.

1) the opening decides what the overall game will look like. An engine should strive to avoid positions that it cannot handle well, and that begins with avoiding certain opening book lines.

2) maybe not relevant for chess engines with the primary goal of being strongest, but certainly for any commercial project: game variety. Human users want each game to be something different. Repeating games is a great way to annoy one's customers.

[Laskos]

Perhaps time spent in the opening has a disproportionate elo cost compared to time spent elsewhere. Also of interest is the constant observation of how Komodo plays so quickly in the opening in TCEC, yet appears to achieve fantastic results with that time management.

I think it would be interesting to rerun this test, using a dual time control. Assuming your original test had a 8-move book, do something like 8-16 seconds for the first 8 moves, then 360s +3.6 for the rest of the game.

Well, if that is the case, then Komodo team could significantly improve the time management in Komodo, if by 20% saving in time spent given by book, they can have 100% boost in time-wise performance. I would guess it's not the case, and I could check with SF, would it have the same time management issue?

[Dann Corbit]

Stockfish also greatly benefits from a book.
My quick stockfish variant test shows that Brainfish outperformed all the other versions, even though the only difference was the brainfish book.

Code: Select all

  Program                          Elo    +   -   Games   Score   Av.Op.  Draws

1 BrainFish_160914_x64_modern    : 3161   21  20   500    75.5 %   2965   48.6 %
2 Cfish-modern                   : 2989   14  14   500    48.5 %   3000   78.2 %
3 Cfish-vanilla                  : 2987   13  13   500    48.1 %   3000   81.0 %
4 Cfish                          : 2986   13  14   500    48.0 %   3000   80.0 %
5 Stockfish-x64-mingw            : 2964   14  14   500    44.1 %   3005   78.2 %
6 Matefinder-x64-mingw           : 2913   15  16   500    35.8 %   3015   70.8 %

So about 170 Elo for a book at time control 1 minute + 1 second time increment with 10 threads at 3.2 GHz.

[Uri Blass]

There is not much talk here about the importance of opening books, although online events and WCCC are heavily dependent on them. I was curious a bit about their impact. I saw two dismissive opinions made here:

1/ Opening books only save time.
2/ With increased time control their importance diminishes, as engines themselves become better at openings.

I took a good (and small) Polyglot opening book BookX.bin by Adam Hair with Komodo and played at different time controls against Komodo no book. Games are from ultra-fast 20s+0.2s to blitz 360s+3.6s on one core. The results seem to contradict both 1/ and 2/.

ELO difference:

20s+0.2s --> 61.7
60s+0.6s --> 59.6
180s+1.8s --> 70.4
360s+3.6s --> 79.5

Code: Select all

20s+0.2s
Score of K2 vs K1: 105 - 34 - 261  [0.589] 400
ELO difference: 61.68 +/- 20.17

60s+0.6s
Score of K2 vs K1: 109 - 41 - 250  [0.585] 400
ELO difference: 59.64 +/- 20.62

180s+1.8s
Score of K2 vs K1: 20 - 4 - 56  [0.600] 80
ELO difference: 70.44 +/- 40.59

360s+3.6s
Score of K2 vs K1: 22 - 4 - 54  [0.613] 80
ELO difference: 79.53 +/- 42.10

Opening books do indeed save time, about 15-20% from total time used. But the gain in the case 360s+3.6s is about equal to doubling in time (100% difference), which is about 80 ELO points for Komodo at this blitz time control. So, book does more than saving time. Also, the importance of the book with longer time control seems to increase, not decrease.

I think that the time control was not long enough for the importance to go down.

Common sense tell me that if the engines can find the right moves by themselves they do not need book so it is obvious that the importance should go down.

I guess that 360+3.6 time control is not long enough for the importance to go down but I believe that the importance go down if you use TCEC time control.

If the time control is long enough books can be even counter productive because engine may find better moves than the book moves.

[Laskos]

Stockfish also greatly benefits from a book.
My quick stockfish variant test shows that Brainfish outperformed all the other versions, even though the only difference was the brainfish book.

Code: Select all

  Program                          Elo    +   -   Games   Score   Av.Op.  Draws

1 BrainFish_160914_x64_modern    : 3161   21  20   500    75.5 %   2965   48.6 %
2 Cfish-modern                   : 2989   14  14   500    48.5 %   3000   78.2 %
3 Cfish-vanilla                  : 2987   13  13   500    48.1 %   3000   81.0 %
4 Cfish                          : 2986   13  14   500    48.0 %   3000   80.0 %
5 Stockfish-x64-mingw            : 2964   14  14   500    44.1 %   3005   78.2 %
6 Matefinder-x64-mingw           : 2913   15  16   500    35.8 %   3015   70.8 %

So about 170 Elo for a book at time control 1 minute + 1 second time increment with 10 threads at 3.2 GHz.

Completely confirmed here. I took the same BookX.bin, because I don't know how to randomize Cerebellum book for variety, it doesn't like "srand" switch in Cutechess-Cli, and might play always the same winning openings against a "naked" Stockfish without a book. Cerebellum book as used by Brainfish has more knowledge than usual "bin" book with Stockfish. So, with BookX.bin, the results are even more conclusive than with Komodo, the improvement is large, and even significantly more to longer time control:

ELO difference:

20s+0.2s --> 98.1
360s+3.6s --> 132.7

Code: Select all

20s+0.2s
Score of SF2 vs SF1: 138 - 28 - 234  [0.637] 400
ELO difference: 98.07 +/- 21.46

360s+3.6s
Score of SF2 vs SF1: 55 - 4 - 81  [0.682] 140
ELO difference: 132.66 +/- 35.78

Practically, at longer time control, Stockfish no book has difficulty winning at all against Stockfish with BookX, losing 40% of the games. The doubling in time is about 80 ELO at this time control, meaning that effective time gain is a factor of 3, or 200%. 20% time saved with a book cannot account for 200% improvement just by itself, the book has other effects too (like simply overplaying the no book SF).

[Dann Corbit]

I think the break even point is probably about 35 plies when comparing with a strong book.

So if the time is long enough for an engine like Komodo or SF to reach 35 plies, then the only difference is lost time. The moves will be of the same quality.

[Laskos]

There is not much talk here about the importance of opening books, although online events and WCCC are heavily dependent on them. I was curious a bit about their impact. I saw two dismissive opinions made here:

1/ Opening books only save time.
2/ With increased time control their importance diminishes, as engines themselves become better at openings.

I took a good (and small) Polyglot opening book BookX.bin by Adam Hair with Komodo and played at different time controls against Komodo no book. Games are from ultra-fast 20s+0.2s to blitz 360s+3.6s on one core. The results seem to contradict both 1/ and 2/.

ELO difference:

20s+0.2s --> 61.7
60s+0.6s --> 59.6
180s+1.8s --> 70.4
360s+3.6s --> 79.5

Code: Select all

20s+0.2s
Score of K2 vs K1: 105 - 34 - 261  [0.589] 400
ELO difference: 61.68 +/- 20.17

60s+0.6s
Score of K2 vs K1: 109 - 41 - 250  [0.585] 400
ELO difference: 59.64 +/- 20.62

180s+1.8s
Score of K2 vs K1: 20 - 4 - 56  [0.600] 80
ELO difference: 70.44 +/- 40.59

360s+3.6s
Score of K2 vs K1: 22 - 4 - 54  [0.613] 80
ELO difference: 79.53 +/- 42.10

Opening books do indeed save time, about 15-20% from total time used. But the gain in the case 360s+3.6s is about equal to doubling in time (100% difference), which is about 80 ELO points for Komodo at this blitz time control. So, book does more than saving time. Also, the importance of the book with longer time control seems to increase, not decrease.

I think that the time control was not long enough for the importance to go down.

Common sense tell me that if the engines can find the right moves by themselves they do not need book so it is obvious that the importance should go down.

I guess that 360+3.6 time control is not long enough for the importance to go down but I believe that the importance go down if you use TCEC time control.

If the time control is long enough books can be even counter productive because engine may find better moves than the book moves.

It's nice and easy to make unverifiable and unfalsifiable claims. May it happen that all engines misplay the openings given them even almost infinite time control? As of now the verified claim is that book factor is important and increases with longer time control, you better present some experimental rebuttal or a solid theory behind the claim.

[Guenther]

There is not much talk here about the importance of opening books, although online events and WCCC are heavily dependent on them. I was curious a bit about their impact. I saw two dismissive opinions made here:

1/ Opening books only save time.
2/ With increased time control their importance diminishes, as engines themselves become better at openings.

I took a good (and small) Polyglot opening book BookX.bin by Adam Hair with Komodo and played at different time controls against Komodo no book. Games are from ultra-fast 20s+0.2s to blitz 360s+3.6s on one core. The results seem to contradict both 1/ and 2/.

ELO difference:

20s+0.2s --> 61.7
60s+0.6s --> 59.6
180s+1.8s --> 70.4
360s+3.6s --> 79.5

Code: Select all

20s+0.2s
Score of K2 vs K1: 105 - 34 - 261  [0.589] 400
ELO difference: 61.68 +/- 20.17

60s+0.6s
Score of K2 vs K1: 109 - 41 - 250  [0.585] 400
ELO difference: 59.64 +/- 20.62

180s+1.8s
Score of K2 vs K1: 20 - 4 - 56  [0.600] 80
ELO difference: 70.44 +/- 40.59

360s+3.6s
Score of K2 vs K1: 22 - 4 - 54  [0.613] 80
ELO difference: 79.53 +/- 42.10

Opening books do indeed save time, about 15-20% from total time used. But the gain in the case 360s+3.6s is about equal to doubling in time (100% difference), which is about 80 ELO points for Komodo at this blitz time control. So, book does more than saving time. Also, the importance of the book with longer time control seems to increase, not decrease.

I think that the time control was not long enough for the importance to go down.

Common sense tell me that if the engines can find the right moves by themselves they do not need book so it is obvious that the importance should go down.

I guess that 360+3.6 time control is not long enough for the importance to go down but I believe that the importance go down if you use TCEC time control.

If the time control is long enough books can be even counter productive because engine may find better moves than the book moves.

It's nice and easy to make unverifiable and unfalsifiable claims. May it happen that all engines misplay the openings given them even almost infinite time control? As of now the verified claim is that book factor is important and increases with longer time control, you better present some experimental rebuttal or a solid theory behind the claim.

Kai, why this harsh answer? I have the same common sense, that at long time controls which are of course much higher than 6m+3.6s (which is still a short time control) there will be a turning point.
It cannot be proven without immense time/work, but there are indications from games like TCEC and SSDF or others, when books produced moves which were weaker than the programs moves, if they had played for themselves.

It also depends of course on book length. The more moves are in a book the more the chance for non optimal moves.
BTW even your test proves not too much so far, because it is based only on 80 games for time controls above 60s+0.6s.

And a last word for the Brainfish test. This cannot be generalized, because the brainfish book is exclusively (and intensively) tuned against/with SF,
which means it does best against SF. (Also its depth is much larger than in your book test - up to ply 50 IIRC)

We can do limited tests up to certain depths and for certain time controls,
but this should also include a few more programs.

Anyway this kind of test is always interesting. Did you mention the depth of that BookX? I cannot find a number.

Thanks for your tests.