Advice on book building

[Werewolf]

I'm looking for a GUI which can make opening books to be used with an engine to play against another engine.


The Book-building GUI should be:
* (relatively) easy to use
* Able to make books with single move additions, rather than a database dump
* Be able to deal with weights where there are a variety of options. I've used Fritz / Chessbase before but found their system confusing / unreliable on this point

Could anyone suggest something? Thanks.

[Jonathan003]

I use PolyGlot Tolerant for book making: https://chess.massimilianogoi.com/downl ... _tolerant/
You can change the weight of book moves with SCID. You can even add moves with SCID.
With SCID vs PC you can delete moves of a book, but it takes a long time. It is faster if you delete the moves in the pgn and make the bin book again afterwards. You can easily delete the moves in pgn with cpt or Lucas chess, by importing the pgn, and deleting the candidate moves, and then export back to pgn.
Here is an interesting polyglot book making script you can try: http://goo.gl/qfkUUp
The new Banksia software also has some interesting book making functions.

[hgm]

WinBoard / XBoard allows you to display the book moves for any given position, alter their weights, delete moves, or add new moves. The latter can be done by playing them on the board (after you pressed the button 'Play Move' in the edit window). The played move will then be added with weight 1. (And automatically retracted on the board, so you stay in the same position so that you could add more moves.) Book moves can be played on the board by right-clicking them in the edit window. You will have to confirm any change by pressing the 'Save' button; if you leave the position without doing that your edits will be discarded.

Book format is Polyglot. The book that is edited is the one installed as GUI book in the Common Engine Settings dialog.

[jdart]

I use an annotated PGN file for inputting and maintaining manually tuned weights and opening lines. I edit these in Chessbase, export to PGN, and build the opening book from that PGN + selected unannotated PGN from human, computer, and correspondence games. Arasan has a custom format opening book, and its own PGN to book construction software, but this method could also be used to build other format books such as Polygot. (Btw. ChessBase is expensive but I use it very, very heavily - I recommend it to all chess fans, although it does have some quirks).

--Jon

[Ovyron]

I guess I have to mention that the free InfinityChess book format (bix) is a thing that exists.

Benefits over Polyglot:

-Book learning (without needing the engine to do it. This is self-modifying of weights so moves that perform better are tried more often)
-Different Book settings (the same book can play different moves depending on how you set it up)
-Fast move deletion (not needing to wait too long or build the book from scratch to delete a move or entire subtree)

That's it, and now:

The reason you don't want to use Bix Books:

-Confusing system (it does nothing better than Fritz / Chessbase on this aspect)
-Unreliable (sometimes you'll see that it'll show castling and castling as two different moves in the same position, or you'll need to restart the GUI lest you'll be randomly adding new moves to the start position instead of the current position)
-GUI Confinement (you can only use Bix books inside the InfinityChess GUI. And the GUI itself is an abomination worthy of an "alpha" name. Expect random crashes, hanging, errors that cause it to stop working unless you restart it, errors that cause more errors in an infinite loop unless you kill it on task manager, and problems with engines not doing anything at all unless you restart, among surprising new annoyances you didn't see coming. If you can get it to install on the first place. The only thing here is a free alternative to Playchess for playing against someone else's engine... with money prizes)

NOT RECOMMENDED (but this might be info worth knowing about)

[hgm]

Note that Polyglot books also support book learning, without needing any engine involvement. Book learning requires two ingedients:

*The book format must reserve space for storing the required data. (Polyglot books have two 16-bit fields per move, one for the number of plays, and one for the number of half-points scored with it.)
* The entity probing the book must include code to update these fields. WinBoard does not do it (as the GUI book is supposed to serve many different engines), but Polyglot does. You just have to switch it on when using Polyglot as a book adapter.

WinBoard's book editor can also be used to edit the learning fields of the book.

I am not sure what this 'long waiting' problem for deleting a move is; I never noticed it in WinBoard, but perhaps my books were never large enogh. Books that hare hand-tailored this way usually don't have to be very large.

WinBoard allows you to set the weight of a move to 0, though, practically instantly, irrespective of the book size. This is as good as deleting the move from the book, except that it will still occupy the space. (Which, for a single move, hardly can be an issue.)

[Ovyron]

I am not sure what this 'long waiting' problem for deleting a move is; I never noticed it in WinBoard, but perhaps my books were never large enogh.

Move adding can also be slow. Scid basically hangs for a while when I add moves to my 300MB Bin book, to the point using a BIX -> CTG -> BIN books setup has been faster.

WinBoard allows you to set the weight of a move to 0, though, practically instantly, irrespective of the book size. This is as good as deleting the move from the book, except that it will still occupy the space. (Which, for a single move, hardly can be an issue.)

Which is like marking moves red on other book formats. Perhaps all we need is a "hide moves with 0 Weight" GUI option.

[Werewolf]

WinBoard / XBoard allows you to display the book moves for any given position, alter their weights, delete moves, or add new moves. The latter can be done by playing them on the board (after you pressed the button 'Play Move' in the edit window). The played move will then be added with weight 1. (And automatically retracted on the board, so you stay in the same position so that you could add more moves.) Book moves can be played on the board by right-clicking them in the edit window. You will have to confirm any change by pressing the 'Save' button; if you leave the position without doing that your edits will be discarded.

Book format is Polyglot. The book that is edited is the one installed as GUI book in the Common Engine Settings dialog.

Is there a help file on this? I downloaded it and it wasn't immediately obvious. The window was tiny for some strange reason as well and when I dragged it bigger it disconnected from the other windows in the GUI (a small thing to be fair but it's an unfamiliar design)

[hgm]

I guess in book formats that require the moves to be stored in some fixed order it is hard to escape that addition will be slow. You will have to shift up everything stored after the insertion point one place, to create space. Doing this repeatedly would indeed be very inefficient.

One work-around could be to just put the moves to be added in a separate, hand-edited book, and then merge the books.

But the problem could be made to go away at the expense of a tiny bit of extra storage. E.g. we could intentionally add a dummy move with weight 0 every 100 moves or so. That would blow up the book size only by 1%. But it would mean it only has to shift up 50 moves on average to insert something.

I could add an 'Inflate Book' menu item in WinBoard to put in such dummy moves. (They would get position key not yet in the book, and a recognizably invalid move with weight 0.) That would then take about the same time as single-move insertion would take. But once you have done it, other move insertions would be 'instantaneous' for a very long time. A menu item 'Compact Book' could squeeze out the spacer moves again.

An alternative is to automate this: if an move insertion would not find an dummy within 200 moves of the insertion point, it would search ahead a few thousand moves to see if this is typical (e.g. fewer than 5 dummies in a thousand). If it is, it would blow up the book again by 1% by doubling the number of dummy moves. Books would always be about 1% larger than they need to be. (But hey, for those not interested in book learning, Polyglot books are 25% larger than they need to be already...)

[hgm]

Is there a help file on this?

Not really, but it doesn't seem rocket science. What I wrote above pretty much describes all there is to it.

I downloaded it and it wasn't immediately obvious. The window was tiny for some strange reason as well and when I dragged it bigger it disconnected from the other windows in the GUI (a small thing to be fair but it's an unfamiliar design)

I don't know why the window initially was tiny (or what you consider 'tiny'). But WinBoard should remember the size and relative location of all windows together with any other user settings, so next time you launch WinBoard the Edit Book window should open as you prefer it.

And yes, WinBoard's design uses independent windows, which can be positioned anywhere, and individually sized. Some windows are 'sticky', and would move with the board window when the latter is sized or moved, but not the various edit windows.