2 Moves Engine Book

[Lyudmil Tsvetkov]

Probably I will be sorry for posting this, many people will laugh, etc., but that is how things have developed with my relations to opening lines:

I have created a 2 moves opening book in simple doc format, downloadable from here: http://www.freeuploadsite.com/do.php?id=27429. The book is written in simple text following the tree approach, and should contain some 2000 different lines. Now, I would like very much to somehow convert the book to a format (for example bin) easily usable by fans and tournament organizers. My problem is I am having a bit of tough time with programming tools, and, if I manage at all to convert the book to the necessary format, I will have to take great pains with familiarising myself with software and functions that might be a bit difficult to grasp. And even if I am successful in my attempt, there is always the chance that something will go wrong because of my inexperience. Therefore I would like to ask if someone is willing to help with converting the book to a engine-usable format?

Why did I create the book and what would be the possible use of its application? For the simple fact that, by following different engine tournaments, even at LTC, it is easily observable that a large portion of the lines played, frequently going to as much as 50%, are a bit, more, or even very much unequal, favouring some of the sides more than would be creditable for a game between top engines to proceed normally without an unnecessary bias. There are cases where the engine is already lost right after book, and for me that is disastrous.

The second reason is that nowadays engines of top quality play as a rule chess better, or significantly better than humans, so that it would be quite normal to leave a larger portion of the chess game to be authentically played by engines, instead of just relying on premade opening lines of dubious quality.

The third reason is of course the desire to see engines play a bigger share of the game themselves. When a book, even if good, ends at move 8, 10, 15, etc. (but for me, even 5 is too much), this basically means that a significant amount of the game is already played by the book, and usually opening is one of the most interesting stages. Why not leave the opening lines up to the decision of engines themselves? I have observed that, precisely because engines are developed and tuned starting from opening positions maybe after move 8, as a general rule engines play weaker the opening stage if left without book. I.e., it is still possible to tune eval and search significantly better for the opening, where engines lag far behind from the knowledge they have later on.

Concerning the 2 Moves book itself:

- it should contain some 2000 distinct opening lines (probably there are doubles in the positions arising with transpositions after move 2, I do not know how many, it would be nice to delete them)
- I think 95% of the openings should be equal within the range of evaluation of 20-30 centipawns for one of the sides; there are no opening lines that are winning for one of the sides, maybe less than 1%
- 2000 lines would allow testing under most standard conditions
- I have included lines that are very much unpopular among chess players of whatever calibre, but perfectly playable; for example, I have included a line like 1.d3 d5 2. c3 e5, which is by no way lost for white, although some engines might have big black scores, and also 1. b3 and even 1.h3, also perfectly playable, but have absolutely excluded any lines starting with 1. g4, 2. b4, etc., which already are very controversial, as they expose white too much.


Why not proceed with simply creating a 2 moves book out of chess players' games? If GM games are chosen, my feeling is that your 2 move book will end up with fewer than some 300-400 main lines. That could be insufficient for a good testing book. If you choose a wider range of players to create a 2 moves book, with all probability there will be too many suboptimal lines (like 1.g4 and 1. b4 for example), so that you will stilll have to go over all the lines manually to correct the book. The idea is good of course and someone might try it, but I have done this more or less in this book without having recourse to any specific games played.

Any comments, but before all any help with converting the book so that it could later be offered for download for use in engine events, would be very much appreciated. In case of need, you can leave me of course a PM.

PS. I guess you will have some hard time with converting to bin format the opening lines written in a tree approach, as possibly this approach does something not in line with specific protocol requirements, but that is why I need your help. I think the book could be an useful tool if accomplished.

[hgm]

Well, like you already guessed the tree format based on indentation is very non-standard. As 2000 lines is still rather small, I doubt if it would pay to create an automated tool to handle it from scratch.

The most efficient way to handle it would be to enter all the moves by hand into a .bin (Polyglot) book (starting with an empty such book), using WinBoard's Edit Book function. There you can just type weight + move into a text field, and press 'Save' to add this move to the repertoire for the currently displayed position. E.g., if you would type

3 e2e4
6 d2d4
1 g1f3

in the Edit Book dialog in the opening position and press the Save button there, you would get the three mentioned moves in the book as initial move, where e2e4 would be played 30% of the time, etc.

If you think this is helpful, I could add a new button to the dialog, which would have the function "add the last played move to the book as move for the PREVIOUS position, with weight 1". Then you would never have to type anything; you would just play the opening lines on the board, (in Edit Game mode) and press the button when you reach the end (or a new position along the line), then take the moves back (with the Left Arrow key, or clicking the < button), and play another one to add that. Should just take a few hours to do that.

[Lyudmil Tsvetkov]

Well, like you already guessed the tree format based on indentation is very non-standard. As 2000 lines is still rather small, I doubt if it would pay to create an automated tool to handle it from scratch.

The most efficient way to handle it would be to enter all the moves by hand into a .bin (Polyglot) book (starting with an empty such book), using WinBoard's Edit Book function. There you can just type weight + move into a text field, and press 'Save' to add this move to the repertoire for the currently displayed position. E.g., if you would type

3 e2e4
6 d2d4
1 g1f3

in the Edit Book dialog in the opening position and press the Save button there, you would get the three mentioned moves in the book as initial move, where e2e4 would be played 30% of the time, etc.

If you think this is helpful, I could add a new button to the dialog, which would have the function "add the last played move to the book as move for the PREVIOUS position, with weight 1". Then you would never have to type anything; you would just play the opening lines on the board, (in Edit Game mode) and press the button when you reach the end (or a new position along the line), then take the moves back (with the Left Arrow key, or clicking the < button), and play another one to add that. Should just take a few hours to do that.

Thanks for the answer, Harm!

Well, it took me many hours to play the moves on a board before entering them in the file. I guess, if I had to manually replay all the lines, it will take me at least couple of full days. Do not you think of any easier way to a bit automate the process? Is it necessary to always specify how frequently a move is played? In this book all the moves are more or less equal, so I guess the same frequency of play for all lines should be fine. Could not this be done automatically?

I think 2000 lines are an optimum and not at all that few. When you consider that you switch sides, you already have 4000 games ready. How many more do you need for testing? A 1 move book would not of course have sufficient variety, while a 3 moves book would take another unnecessary right of own choice for the engine. I thought the earlier an engine starts thinking, the better.

I would very much appreciate if some tool for an easier handling of the book could be conjured up, or some ideas how to make life much easier in transforming the book suggested. In any case I will be waiting some time for suggestions (maybe someone will lend a helping hand), as the manual reentering of all the moves seems really quite burdensome to me.

In any case, what I know for sure, is that engine testing badly needs some very short and completely equal openings book like that.

[Adam Hair]

Well, like you already guessed the tree format based on indentation is very non-standard. As 2000 lines is still rather small, I doubt if it would pay to create an automated tool to handle it from scratch.

The most efficient way to handle it would be to enter all the moves by hand into a .bin (Polyglot) book (starting with an empty such book), using WinBoard's Edit Book function. There you can just type weight + move into a text field, and press 'Save' to add this move to the repertoire for the currently displayed position. E.g., if you would type

3 e2e4
6 d2d4
1 g1f3

in the Edit Book dialog in the opening position and press the Save button there, you would get the three mentioned moves in the book as initial move, where e2e4 would be played 30% of the time, etc.

If you think this is helpful, I could add a new button to the dialog, which would have the function "add the last played move to the book as move for the PREVIOUS position, with weight 1". Then you would never have to type anything; you would just play the opening lines on the board, (in Edit Game mode) and press the button when you reach the end (or a new position along the line), then take the moves back (with the Left Arrow key, or clicking the < button), and play another one to add that. Should just take a few hours to do that.

Thanks for the answer, Harm!

Well, it took me many hours to play the moves on a board before entering them in the file. I guess, if I had to manually replay all the lines, it will take me at least couple of full days. Do not you think of any easier way to a bit automate the process? Is it necessary to always specify how frequently a move is played? In this book all the moves are more or less equal, so I guess the same frequency of play for all lines should be fine. Could not this be done automatically?

I think 2000 lines are an optimum and not at all that few. When you consider that you switch sides, you already have 4000 games ready. How many more do you need for testing? A 1 move book would not of course have sufficient variety, while a 3 moves book would take another unnecessary right of own choice for the engine. I thought the earlier an engine starts thinking, the better.

I would very much appreciate if some tool for an easier handling of the book could be conjured up, or some ideas how to make life much easier in transforming the book suggested. In any case I will be waiting some time for suggestions (maybe someone will lend a helping hand), as the manual reentering of all the moves seems really quite burdensome to me.

In any case, what I know for sure, is that engine testing badly needs some very short and completely equal openings book like that.

I am working on transforming your book into a pgn file at the moment, Lyudmil. I think I can have it done in 2-3 hours from now.

[hgm]

If it was a PGN file (or in fact if it just contained the full move sequence of each line, rtahre than relying on indentation to repeat initial parts previous move sequences), and each line was preceded by the number 1), WinBoard could read it, and convert it to a bin book. Not all moves would get equal weights, however. But they would get weights that makes reachin geach f the leaves equally likely.

I would very much appreciate if some tool for an easier handling of the book could be conjured up.

The tool exists. MS Word is not it, however.

It would have been far less work for you to type the moves directly into WinBoard's Edit Book dialog than the time you now spent to make the .doc file. With the enhanvement of the 'Add last move' button I proposed here you would not even have had to type anything, and it would have been at least 5 times faster than what you did now.

It is not our fault that you run off into a dead-end street, and then start complaining you now have to travel so far to get where you want to be, and that there are no busses running through the street...

[Lyudmil Tsvetkov]

I am working on transforming your book into a pgn file at the moment, Lyudmil. I think I can have it done in 2-3 hours from now.

Many thanks, Adam!
Just great, I hope you will let know if you manage to do it.

[Adam Hair]

There were 1683 lines in the book. After converting it to pgn format (involved reformatting, then importing into a spreadsheet in order to copy/paste, then using pgn extract), I used a function in Gaviota called 'enddups' to eliminate duplicate positions. This left 913 unique starting positions. While this is probably not the amount of positions you were hoping for, Lyudmil, the amount is still enough for many types of tests.

Here is the link:

***

EDIT
New link: http://www.mediafire.com/download/jskag ... LT_1000.7z

[Lyudmil Tsvetkov]

There were 1683 lines in the book. After converting it to pgn format (involved reformatting, then importing into a spreadsheet in order to copy/paste, then using pgn extract), I used a function in Gaviota called 'enddups' to eliminate duplicate positions. This left 913 unique starting positions. While this is probably not the amount of positions you were hoping for, Lyudmil, the amount is still enough for many types of tests.

Here is the link:

http://www.mediafire.com/download/o0kze ... _LT.pgn.7z

Many thanks Adam!!

You are an absolute whizzard. I knew I should have studied computer science once.
Many thanks also to Miguel, of course, for his precious Gaviota tool. It seems that the mods here are doing an extremely useful job, hats off.

Concerning the book, that is how a computer shines over a human: I never thought there wer so many duplicates, maybe some 4/9ths, I supposed there were some when I entered the moves, but my memory failed me.

Now, one last thing about the book: would you please give me an advice if the pgn format is the best for a book, or maybe there is some other format preferable, for example bin file? What would most engine events use?
Is it possible to also add a bin file to the download option, maybe someone could help with transforming the pgn to bin?

Finally, would you mind if I use your download address to post a brief message, maybe on the main forum, to let users know there is a new book to use?

Maybe thanks again for the incredible help!

[Lyudmil Tsvetkov]

Many thanks Adam!!

You are an absolute whizzard. I knew I should have studied computer science once.
Many thanks also to Miguel, of course, for his precious Gaviota tool. It seems that the mods here are doing an extremely useful job, hats off.

Concerning the book, that is how a computer shines over a human: I never thought there wer so many duplicates, maybe some 4/9ths, I supposed there were some when I entered the moves, but my memory failed me.

Now, one last thing about the book: would you please give me an advice if the pgn format is the best for a book, or maybe there is some other format preferable, for example bin file? What would most engine events use?
Is it possible to also add a bin file to the download option, maybe someone could help with transforming the pgn to bin?

Finally, would you mind if I use your download address to post a brief message, maybe on the main forum, to let users know there is a new book to use?

Maybe thanks again for the incredible help!

Hi Adam.
Just to ask once again if you have some comments on the above questions of mine?

[Adam Hair]

Many thanks Adam!!

You are an absolute whizzard. I knew I should have studied computer science once.
Many thanks also to Miguel, of course, for his precious Gaviota tool. It seems that the mods here are doing an extremely useful job, hats off.

Concerning the book, that is how a computer shines over a human: I never thought there wer so many duplicates, maybe some 4/9ths, I supposed there were some when I entered the moves, but my memory failed me.

Now, one last thing about the book: would you please give me an advice if the pgn format is the best for a book, or maybe there is some other format preferable, for example bin file? What would most engine events use?
Is it possible to also add a bin file to the download option, maybe someone could help with transforming the pgn to bin?

Finally, would you mind if I use your download address to post a brief message, maybe on the main forum, to let users know there is a new book to use?

Maybe thanks again for the incredible help!

Hi Adam.
Just to ask once again if you have some comments on the above questions of mine?

Hi Lyudmil,

I have not checked for a few hours, so I missed your response.

I think the format that would be most used is pgn, but it would probably useful to make it available in various formats. Tomorrow (Sunday) I will upload a folder of the book in pgn, epd, bin, and possibly Arena and ChessGui formats. When I have uploaded the folder, I will share the link. You can freely post the link wherever you like.

I helped out because I admire your contributions to computer chess. I am no programmer, but I saw how I could use a text editor, a spreadsheet, and some computer chess tools to reformat your book. It took less than 4 hours total. However, if the book was 3 moves or more deep, I would have not made the effort. I have better things to do on the weekends