Very much diminishing benefit of Opening Books at LTC

[Laskos]

I took as one competitor the latest Brainfish with the latest free Cerebellum book, a very competitive book against other books and against no-book engine. The other was no-book latest Brainfish. For diversity of openings, I took Performance.bin general book to 4 moves, then the Cerebellum took control of the first engine, and the second engine started calculating by itself. The average depth the Cerebellum went into the game was 12-14 moves after exiting the opening position. The second engine had to think for itself these 12-14 moves.

The main advantage is the time used: the "naked" engine spent about 30% of the total thinking time in the opening, while the first engine was still in the Cerebellum book.

Having 1.4 time advantage factor ( 100%/(100%-30%) ), means about 0.5 doublings in time. So, if a book has an added value compared to just saving time in the opening, it should perform better than 0.5 doublings advantage, if not, equal or less.

I played games at two time controls on one thread:

First time control:
STC

Code: Select all

15''+ 0.15''

Score of BrainFish_Book vs BrainFish: 295 - 55 - 650 [0.620]
Elo difference: 85.04 +/- 12.34

1000 of 1000 games finished.

The doubling at this time control is about 130 Elo points. So, the advantage of the book is 85/130 ~ 0.65 doublings. This means that the the book has an added advantage to just saving time, probably the engine enabled with the book exits the book in a better position.

Second time control:
LTC

Code: Select all

3600''+ 36'' (1 hour + 36 seconds increment)

Score of BrainFish_Book vs BrainFish: 5 - 3 - 92 [0.510]
Elo difference: 6.95 +/- 19.21

100 of 100 games finished.

This LTC test lasted 3 days, 4 concurrent games on 4 cores (engine on 1 thread per core). The PGN file is here:
http://s000.tinyupload.com/?file_id=037 ... 8209928033

The doubling at this long time control is about 50 Elo points. The book advantage given by time saved while in the book should be again about 0.5 doublings, but it is almost beyond error margins that this is not achieved in LTC, never mind earlier STC 0.65 doublings result. In fact the result here is about 0.15 doublings (7 Elo points / 50 Elo points). The only explanation would be that the exit from the book is not very favorable compared to "naked" no-book engine at LTC (it seems favorable at STC). And the only real advantage is the time saved. Also, all the measures like Elo, Normalized Elo and Wilo show dramatically decreasing performance of the book with long time control.

It should be noted that Cerebellum is a "computer" Stockfish-analyzed book, maybe hand-tuned books using different criteria (say databases) behave a bit differently.

[Rebel]

Nice subject Kai.

Related - http://rebel13.nl/rebel13/experiments.html

Note, page is still under development.

[jdart]

I don't know but I think this effect depends on engine strength.

Very strong engines such as Brainfish seem adept at recovering from subpar postions out of the book. They may even find by search an "only" move that a book would probably contain. Weaker engines will struggle or will fail to find a key move soon of book, and the bad book exit will kill them.

--Jon

[Laskos]

Nice subject Kai.

Related - http://rebel13.nl/rebel13/experiments.html

Note, page is still under development.

Wow, that's good. I used my much earlier results with SF 7 or 8 and Komodo, maybe I have to adjust to a bit lower Elo numbers for doubling, say 120 instead of 130 at 15'' and 40 instead of 50 at 3600''. But the conclusions of my post are the same, there is a drastically diminishing benefit of book-enabled SF against the "naked" SF with time control.

[Laskos]

I don't know but I think this effect depends on engine strength.

Very strong engines such as Brainfish seem adept at recovering from subpar postions out of the book. They may even find by search an "only" move that a book would probably contain. Weaker engines will struggle or will fail to find a key move soon of book, and the bad book exit will kill them.

--Jon

Yes, this is one aspect, it is self-play of very strong engine. Another issue: against a poor book, a good book like Cerebellum performs even better than against no-book at all. I enabled BrainFish with a weak Performance.bin book to full length, and the intermediate result at 15''+ 0.15'' time control looks like the advantage is huge, some 70% larger than against no-book result

Code: Select all

15''+ 0.15''
 
Score of BrainFish_Book vs BrainFish: 251 - 38 - 243 [0.700]
Elo difference: 147.35 +/- 21.77

532 of 1000 games finished.

The conclusion would be: if you have a weaker book, against a stronger book use no-book at all. I didn't check LTC result, but LTC result of BrainFish enabled with Cerebellum is very meager against no-book BrainFish.

[Milos]

The doubling at this long time control is about 50 Elo points. The book advantage given by time saved while in the book should be again about 0.5 doublings, but it is almost beyond error margins that this is not achieved in LTC, never mind earlier STC 0.65 doublings result. In fact the result here is about 0.15 doublings (7 Elo points / 50 Elo points). The only explanation would be that the exit from the book is not very favorable compared to "naked" no-book engine at LTC (it seems favorable at STC). And the only real advantage is the time saved. Also, all the measures like Elo, Normalized Elo and Wilo show dramatically decreasing performance of the book with long time control.

What would be interesting to see is how much naked SF moves differed from Cerebellum moves in those games. I guess not much.

[Rebel]

Nice subject Kai.

Related - http://rebel13.nl/rebel13/experiments.html

Note, page is still under development.

Wow, that's good. I used my much earlier results with SF 7 or 8 and Komodo, maybe I have to adjust to a bit lower Elo numbers for doubling, say 120 instead of 130 at 15'' and 40 instead of 50 at 3600''. But the conclusions of my post are the same, there is a drastically diminishing benefit of book-enabled SF against the "naked" SF with time control.

No surprise, but the good news is doubling speed still gives amazingly much at LTC.

[Ovyron]

Have you considered "fixed time per move" for your tests? The idea is here it'd nullify any time advantage for the book and we'd only see the advantage of better out of book positions, if any.

[jdart]

That is interesting, and makes sense.

I have a pretty decent opening book now but it is large and I am still discovering some places where it exits with a bad score. So it is quite possible a poor book will dump you into a position you wouldn't have gone into using search. That might even be frequent. If you just build a book from a game collection for example you will include many older moves that are now refuted.

On the other hand, if you play with no book you are likely to force a book exit earlier, before you are really in trouble.

--Jon

[corres]

To make an opening book we use games played by older and weaker engines and/or humans with limited time usage. So it is natural that the given benefit of these opening books diminish if we use stronger engines and/or longer time control.