How many Elo points is a book?

[chrisw]

Up to now I've not been using a book (multi-game testing was always using cutechess and a big random book, same for both sides, to give variance).
Since getting relatively trashed in the monthly Dutch blitz tournament, I hurriedly put a book together (semi-random move chooser, similar chooser method to polyglot based on frequency for the moment) and am testing engine plus book vs engine, and seem to be getting 63% win rate, or about 100 Elo or so. Does this figure? 100 Elo points for a primitive book?

Book takes about an hour to construct (Python) pulling positions from about 3 million CCRL PGNs, hashing them and accumulating the game results.

[towforce]

IIRC when CS-Tal came out, it had a revolutionary large and high quality opening book, and when it left the opening book, at that time (mid to late 90s), it used to be significantly ahead on evaluation score.

This would be a good test of how good today's computers are (and IMO a test of how close they are to playing unbeatable chess): playing without a book, how even is the position when playing against a program with a very good book when that program leaves the book?

Anyone done a test like this?

[Joost Buijs]

Up to now I've not been using a book (multi-game testing was always using cutechess and a big random book, same for both sides, to give variance).
Since getting relatively trashed in the monthly Dutch blitz tournament, I hurriedly put a book together (semi-random move chooser, similar chooser method to polyglot based on frequency for the moment) and am testing engine plus book vs engine, and seem to be getting 63% win rate, or about 100 Elo or so. Does this figure? 100 Elo points for a primitive book?

Book takes about an hour to construct (Python) pulling positions from about 3 million CCRL PGNs, hashing them and accumulating the game results.

I do exactly the same to construct my book, a binary tree with hashes and statistics. The difference is that I've written it in C++ and that it goes a lot faster (about a minute or so for 5 million games). I kept it compatible with Polyglot by using the same random numbers and output format. Instead of looking at move frequency only I also look at the WLD ratio, to avoid selecting moves with good statistics and a low frequency I've added some kind of regularization. The games I use are a mix of games from CCRL, Chessbase and Infinity Chess, usually to a depth of 70 plies.

To answer your question, I've never checked how much Elo points my engine gains by using this book, however I think it's quite a lot. It often happens that it plays 35 moves from book, maybe a whole game takes ~70, so it almost doubles the amount of time the engine can spend on the remaining moves, this alone is worth ~60 Elo.

[chrisw]

Up to now I've not been using a book (multi-game testing was always using cutechess and a big random book, same for both sides, to give variance).
Since getting relatively trashed in the monthly Dutch blitz tournament, I hurriedly put a book together (semi-random move chooser, similar chooser method to polyglot based on frequency for the moment) and am testing engine plus book vs engine, and seem to be getting 63% win rate, or about 100 Elo or so. Does this figure? 100 Elo points for a primitive book?

Book takes about an hour to construct (Python) pulling positions from about 3 million CCRL PGNs, hashing them and accumulating the game results.

I do exactly the same to construct my book, a binary tree with hashes and statistics. The difference is that I've written it in C++ and that it goes a lot faster (about a minute or so for 5 million games).

Betcha that writing the book creation code in Python goes a lot faster than writing it in C !! The whole operation, from deciding to do it, to finished book playing program took less than two days. I do as much of my 'outside of engine' stuff in Python as I possibly can.

I kept it compatible with Polyglot by using the same random numbers and output format. Instead of looking at move frequency only I also look at the WLD ratio, to avoid selecting moves with good statistics and a low frequency I've added some kind of regularization.

Yes, I need to take a look at that, currently I just use (2 x W + D) and then random choose the move based on that value. I didn't want to use Polyglot hashes because I have all hashing code mechanics already for two/four years or more in the engine, so wasn't going to reinvent any wheels. I was going to use some function of W-D-L, but quick look at Polyglot docs, and it uses W D only, which seemed logical, so went with that idea.

The games I use are a mix of games from CCRL, Chessbase and Infinity Chess, usually to a depth of 70 plies.

CCRL only, not for any good reason, just that they exist on my HD. Went to 60 plies, no good reason either, just a guess.

To answer your question, I've never checked how much Elo points my engine gains by using this book, however I think it's quite a lot. It often happens that it plays 35 moves from book, maybe a whole game takes ~70, so it almost doubles the amount of time the engine can spend on the remaining moves, this alone is worth ~60 Elo.

My test running now, is down to 55% up, about 40 Elo. Prior version with no book, versus the above described first shot book. I'm using a slightly modified Ed Schroeder 32000.pgn limited to 6 plies to give the initial position range, then relying on the 'book' to randomise those positions into a range of games. So basically, I'm testing engine with book against engine without (I hope).
The question arises how fast does 'engine without' take 'engine with', out of book? Because then engine with does not have the long N=70 move time advantage any longer. Possible this implies time advantage of ~60 ELO doesn't get reached?

Other factor 'for' book, is that theoretically 'engine with' gets better positions, which is +ve Elo.

[Joost Buijs]

Other factor 'for' book, is that theoretically 'engine with' gets better positions, which is +ve Elo.

Most of the games I use for the book are played by top engines, I assume that the engine will get a better position with book.

During tourneys it often happens that the engine plays a repetition draw straight from book, this is a nuisance especially against much weaker opponents, this is still something that I need to fix.

[chrisw]

Other factor 'for' book, is that theoretically 'engine with' gets better positions, which is +ve Elo.

Most of the games I use for the book are played by top engines, I assume that the engine will get a better position with book.

During tourneys it often happens that the engine plays a repetition draw straight from book, this is a nuisance especially against much weaker opponents, this is still something that I need to fix.

Thanks. I better mark that for action. I guess test book moves for 2-rep draw and ask search() instead? Would that work? Couple of code lines, if it would.

[Joost Buijs]

Other factor 'for' book, is that theoretically 'engine with' gets better positions, which is +ve Elo.

Most of the games I use for the book are played by top engines, I assume that the engine will get a better position with book.

During tourneys it often happens that the engine plays a repetition draw straight from book, this is a nuisance especially against much weaker opponents, this is still something that I need to fix.

Thanks. I better mark that for action. I guess test book moves for 2-rep draw and ask search() instead? Would that work? Couple of code lines, if it would.

That would probably work. I still use polyglot to read my book, that makes it difficult, I have to incorporate the book software in my engine to solve this. Another problem is that the engine doesn't know what the rating of the opponent is, if the opponent is stronger it should be happy with a repetition draw. In my case it usually happens against weaker opponents, it could be that the stronger ones already have a fix for this nuisance.

Funny is that in a very distant past I always had the book software inside my engine, after I started using polyglot I removed it. The reason that I made my own software to build a polyglot book is that polyglot has difficulties to build very large books because it is restricted to 32 bits, and that it aborts when there is a move error in one of the games you throw at it.

[D Sceviour]

Other factor 'for' book, is that theoretically 'engine with' gets better positions, which is +ve Elo.

Most of the games I use for the book are played by top engines, I assume that the engine will get a better position with book.

During tourneys it often happens that the engine plays a repetition draw straight from book, this is a nuisance especially against much weaker opponents, this is still something that I need to fix.

Thanks. I better mark that for action. I guess test book moves for 2-rep draw and ask search() instead? Would that work? Couple of code lines, if it would.

That would probably work. I still use polyglot to read my book, that makes it difficult, I have to incorporate the book software in my engine to solve this. Another problem is that the engine doesn't know what the rating of the opponent is, if the opponent is stronger it should be happy with a repetition draw. In my case it usually happens against weaker opponents, it could be that the stronger ones already have a fix for this nuisance.

In some book positions there is an almost forced perpetual check. Any other move would lose instantly. Books should never contain such forced draws, but simply testing for 3-fold position is not enough.
I prepare a root move list to test book legality. There may be only one move in the move list.
Play the board instead of the opponents supposed strength. Suggest: continue with the 3-fold repetition if the static evaluation < -100 for example.

[yurikvelo]

https://www.chessdb.cn/queryc_en/
https://github.com/noobpwnftw/chessdb

Does this figure? 100 Elo points for a primitive book?

depends on standalone strength of engine
for SF-NNUE or Leela it may be few ELO

Brainfish author has stopped development immediately after NNUE release
https://zipproth.de/Brainfish/download/

[Dann Corbit]

https://www.chessdb.cn/queryc_en/
https://github.com/noobpwnftw/chessdb

Does this figure? 100 Elo points for a primitive book?

depends on standalone strength of engine
for SF-NNUE or Leela it may be few ELO

Brainfish author has stopped development immediately after NNUE release
https://zipproth.de/Brainfish/download/

Quite so. From Pohl's site:
https://www.sp-cc.de/

We have this:

Code: Select all

     Program                    Elo    +    -   Games   Score   Av.Op.  Draws

   1 CFish 12 3xCerebellum    : 3715    8    8  7000    86.1 %   3377   27.3 %
   2 Stockfish 200921 avx2    : 3703    8    8  7000    82.6 %   3401   32.5 %
   3 Stockfish 200915 avx2    : 3699    8    8  7000    82.3 %   3401   33.0 %
   4 CFish 12 avx2            : 3692    8    8  7000    84.6 %   3377   29%

So the Cerebellum book added 23 Elo to Cfish.
If you look at his graphs on the same page, you will see that the book added 50 Elo to Stockfish.
So that gap is closing.

I guess also that the value of the book will depend upon a lot of factors.
For instance, at hyper-bullet speed, the engines will make moves that are a lot worse, so it stands to reason that the book will add more than at longer time control.
Imagine if Bojun's data were at 50 plies deep. That book would be a crushing advantage, but it would cost a lot of money to make the book, just in electricity costs today.
Sedat does a lot of book experiments. From here: https://sites.google.com/site/computers ... book-cs-45
You can see the Cerebellum 3 merge book (used above) is 19 Elo behind the book "DON lite 2".

Of course, after each contest, the book authors will do post mortem analysis and patch the holes in their books.