Laskos wrote:Uri Blass wrote:Laskos wrote:There is not much talk here about the importance of opening books, although online events and WCCC are heavily dependent on them. I was curious a bit about their impact. I saw two dismissive opinions made here:
1/ Opening books only save time.
2/ With increased time control their importance diminishes, as engines themselves become better at openings.
I took a good (and small) Polyglot opening book BookX.bin by Adam Hair with Komodo and played at different time controls against Komodo no book. Games are from ultra-fast 20s+0.2s to blitz 360s+3.6s on one core. The results seem to contradict both 1/ and 2/.
ELO difference:
20s+0.2s --> 61.7
60s+0.6s --> 59.6
180s+1.8s --> 70.4
360s+3.6s --> 79.5
Code: Select all
20s+0.2s
Score of K2 vs K1: 105 - 34 - 261 [0.589] 400
ELO difference: 61.68 +/- 20.17
60s+0.6s
Score of K2 vs K1: 109 - 41 - 250 [0.585] 400
ELO difference: 59.64 +/- 20.62
180s+1.8s
Score of K2 vs K1: 20 - 4 - 56 [0.600] 80
ELO difference: 70.44 +/- 40.59
360s+3.6s
Score of K2 vs K1: 22 - 4 - 54 [0.613] 80
ELO difference: 79.53 +/- 42.10
Opening books do indeed save time, about 15-20% from total time used. But the gain in the case 360s+3.6s is about equal to doubling in time (100% difference), which is about 80 ELO points for Komodo at this blitz time control. So, book does more than saving time. Also, the importance of the book with longer time control seems to increase, not decrease.
I think that the time control was not long enough for the importance to go down.
Common sense tell me that if the engines can find the right moves by themselves they do not need book so it is obvious that the importance should go down.
I guess that 360+3.6 time control is not long enough for the importance to go down but I believe that the importance go down if you use TCEC time control.
If the time control is long enough books can be even counter productive because engine may find better moves than the book moves.
It's nice and easy to make unverifiable and unfalsifiable claims. May it happen that all engines misplay the openings given them even almost infinite time control? As of now the verified claim is that book factor is important and increases with longer time control, you better present some experimental rebuttal or a solid theory behind the claim.
Kai, why this harsh answer? I have the same common sense, that at long time controls which are of course much higher than 6m+3.6s (which is still a short time control) there will be a turning point.
It cannot be proven without immense time/work, but there are indications from games like TCEC and SSDF or others, when books produced moves which were weaker than the programs moves, if they had played for themselves.
It also depends of course on book length. The more moves are in a book the more the chance for non optimal moves.
BTW even your test proves not too much so far, because it is based only on 80 games for time controls above 60s+0.6s.
And a last word for the Brainfish test. This cannot be generalized, because the brainfish book is exclusively (and intensively) tuned against/with SF,
which means it does best against SF. (Also its depth is much larger than in your book test - up to ply 50 IIRC)
We can do limited tests up to certain depths and for certain time controls,
but this should also include a few more programs.
Anyway this kind of test is always interesting. Did you mention the depth of that BookX? I cannot find a number.
Thanks for your tests.