2 Moves Engine Book

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
michiguel
Posts: 6389
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: 2 Moves Engine Book

Post by michiguel » Sun Dec 08, 2013 10:52 pm

hgm wrote:
michiguel wrote:Doing these things in Ruby is way much faster to write it, particularly with string manipulation (not necessarily fewer lines). Anyway, I am sure it could be written more compact in both, C and Ruby than I have done it.
Well, I don't know Ruby at all, so it is hard to judge for me how easy that makes it. But the task at hand seems quite trivial in C. It is basically just a matter of copying the indented lines over the end of the previous non-indented lines, starting at various positions. That is a single strcpy call, and counting some spaces.
It is a different way of thinking. That is because of the many high level tools available. You can think in terms of tokens, rather than chars. When you do

a = s.chomp.trim.split(' ')

You take the string s and you get an array of tokens in a. Carriage returns, tabs, spaces etc. have been removed and tokens were separated by spaces. It does not matter how long the string is, and how long the array will be. Everything is handled dynamically under the hood.

In this particular case, words were not evenly spaced and to make matters less "C" friendly, there were tabs mixed with spaces. Of course I could have done with C, but then I would have had to pay much more attention to the input format.

Miguel

User avatar
Ajedrecista
Posts: 1402
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 2-move engine book.

Post by Ajedrecista » Mon Dec 09, 2013 10:16 am

Hello again:

It seems that nobody tried my small sample of 2moves_LT.cgb book, so anyway I release it today WITHOUT ANY WARRANTY. If it works, perfect; if not, sorry, I tried it and failed. I will not spent more time in this task.

Firstly, I want to describe my strange conversion method from *.pgn to *.cgb standard: I used Grob.cgb (the second post of the thread) and Nimzowitsch-Larsen.cgb (the first post of the thread) as guidelines. I opened them with Notepad (not Notepad++, which will be used later) and I spent some minutes without result until things began to match in my slow mind. I induced a chart of conversion of characters between the two formats. Here are my findings:

Code: Select all

.pgn   .cgb
-----------
  a      .
  b      g
  c      h
  d      O
  e      j
  f      i
  g      B
  h      N
  x      x
  1      Unknown to me.
  2      a
  3      b
  4      c
  5      K
  6      Q
  7      R
  8      d
  K      Unknown to me.
  Q      4
  R      Unknown to me.
  B      6
  N      7
  +
That is, I think that a check (+) is not indicated with any sign in CGB books. A checkmate (#) is also unknown to me. More things: the initial FEN of a chess game is denoted with X-, the space between white and black move is denoted with 1, the space between (n-1)-th black move and n-th white move (moves, not plies) is indicated with 1&-, where & is n in CGB notation (for example, between the first black move and the second white move, the separator is 1a-). Castlings and promotions are unknown to me and the finish of an opening book line is marked with XYa2XYa (Matthias Gemuh could verify or disprove my statements).

I deleted all the PGN tags and 1/2-1/2 results at the end of each line (it was easy with Notepad) but ECO codes were more time consuming to delete because of their variety. Once I got the PGN as I wanted, I copied it in Notepad++ and did the replacements (Notepad++ is much faster than Notepad for this task) with extreme care: I had luck because there were not moves in the rank 1 (I mean, Nb1 for example)... please remember that rank 1 conversion is unknown to me, as well as castlings and other things explained above. I began with initial FENs (X-) and separators between moves. Then I did the replacements by squares (a2 |---> .a and many more) and finally I took care of pieces (I had good luck again because there were neither king moves nor rook moves) in the following way: Bf4, Bf5, etc. were handled with the piece type and their file (Bf) but not their rank, so Bf was converted to 6i and so on. The last task was assign each opening book line to a physical line, therefore I had to force newlines (or line breaks or whatever they are called). I had no clue but Google is too much Google and \r\n did the trick in Notepad++. That is all!

I uploaded the file 2moves_LT.cgb with its duplicate in .txt extension, just to make easy corrections if needed. This link will die 30 days after the last download.

2moves_LT.cgb.rar (3 KB)

2moves_LT.cgb file size is 22.35 KB more less. I will appreciate feedback (basically 'it works!' or 'it does not work') and in the case of positive feedback, mirror links are accepted as expected.

Regards from Spain.

Ajedrecista.

Adam Hair
Posts: 3201
Joined: Wed May 06, 2009 8:31 pm
Location: Fuquay-Varina, North Carolina

Re: 2-move engine book.

Post by Adam Hair » Mon Dec 09, 2013 11:42 am

Hi Jesús,

In NotePad++:

Find
Replace
Check Regular expression

Find:\[ECO\s.+
Replace:

Study regular expressions. I do not know a lot, but regular expressions are quite helpful when trying to modify a text file or extract data.

Also, add the TextFX plugins to NotePad++. Then you can remove all blank lines at one time.

User avatar
Ajedrecista
Posts: 1402
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 2-move engine book.

Post by Ajedrecista » Mon Dec 09, 2013 11:55 am

Hi Adam:
Adam Hair wrote:Hi Jesús,

In NotePad++:

Find
Replace
Check Regular expression

Find:\[ECO\s.+
Replace:

Study regular expressions. I do not know a lot, but regular expressions are quite helpful when trying to modify a text file or extract data.

Also, add the TextFX plugins to NotePad++. Then you can remove all blank lines at one time.
Thanks for your tips, very useful as usual.

I just realize that I did not remove checks (+) so probably my .cgb file does not work under ChessGUI. There are four of them among the 913 opening lines. Fortunately I included the txt copy, so the change can be done easily.

Just as a suggestion, please try the original file of my download and see if works; then remove signs +, change the extension to .cgb and try again. Any feedback will be welcome! Crossing fingers to get it work and you can spread a download link together with other book formats. ;)

Regards from Spain.

Ajedrecista.

Lyudmil Tsvetkov
Posts: 6052
Joined: Tue Jun 12, 2012 10:41 am

Re: 2-move engine book. - please help again

Post by Lyudmil Tsvetkov » Mon Dec 09, 2013 2:15 pm

Hi Adam, Miguel, (Julien), Jesus, and everybody that has the slightest ooportunity to help me fot the last time. I completed the book to 1000 lines - actually 1015 to be sure 1000 will remain after doubles are eliminated; any positions after 1000 might simply be deleted to round things off. Is someone able to kindly help? Below the full text of my message on the tournament forum, where there is a download site for the new pgn file:


Hi again,
as I see that some people are interested in the book, and even Larry Kaufman (why I would like to write Kaufmann with double n?) reacted that he could use the book, I took the pains in the hurry to complete the book to 1000 positions. I added 1.f4, 1. c3, 1. a3, 1.a4 and 1.f3 lines, to make the book more exotic and interesting. Still I think none of those lines is lost; later I might post some games with those lines. I would not vouch however that 1.e4 b6 is not lost for black, and therefore there are not such lines. White has some leeway while playing. The opening advantage might not be there, but the lines are still even, and that was the main purpose: to provide some variety of play in equal lines.

Now, I have entered the new lines in chesspad (thanks to Mark van der Leek for a wonderful tool), but my problem is to check for doubles, unify headings, as well as delete the remaining plus positions over 1000 after the check for doubles is done. I simply can not do that without having to read lots of manuals. So please someone (I would not have the courage to ask Adam again, but maybe Miguel or someone else will be so helpful again) help me with the above job, so that in the end we have a book of 1000 starting positions.

The download link to the completed pgn file is here: http://www.freeuploadsite.com/do.php?id=27609

I do not know if someone will be able to convert the pgn to other formats, I could do probably ctg, but my work is finished here.

User avatar
Ajedrecista
Posts: 1402
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 2-move engine book. Please help me again.

Post by Ajedrecista » Mon Dec 09, 2013 2:55 pm

Hello Lyudmil:
Lyudmil Tsvetkov wrote:Hi Adam, Miguel, (Julien), Jesus, and everybody that has the slightest ooportunity to help me fot the last time. I completed the book to 1000 lines - actually 1015 to be sure 1000 will remain after doubles are eliminated; any positions after 1000 might simply be deleted to round things off. Is someone able to kindly help? Below the full text of my message on the tournament forum, where there is a download site for the new pgn file:


Hi again,
as I see that some people are interested in the book, and even Larry Kaufman (why I would like to write Kaufmann with double n?) reacted that he could use the book, I took the pains in the hurry to complete the book to 1000 positions. I added 1.f4, 1. c3, 1. a3, 1.a4 and 1.f3 lines, to make the book more exotic and interesting. Still I think none of those lines is lost; later I might post some games with those lines. I would not vouch however that 1.e4 b6 is not lost for black, and therefore there are not such lines. White has some leeway while playing. The opening advantage might not be there, but the lines are still even, and that was the main purpose: to provide some variety of play in equal lines.

Now, I have entered the new lines in chesspad (thanks to Mark van der Leek for a wonderful tool), but my problem is to check for doubles, unify headings, as well as delete the remaining plus positions over 1000 after the check for doubles is done. I simply can not do that without having to read lots of manuals. So please someone (I would not have the courage to ask Adam again, but maybe Miguel or someone else will be so helpful again) help me with the above job, so that in the end we have a book of 1000 starting positions.

The download link to the completed pgn file is here: http://www.freeuploadsite.com/do.php?id=27609

I do not know if someone will be able to convert the pgn to other formats, I could do probably ctg, but my work is finished here.
I do not have problems in help you again, but the question is if my CGB version works, which I do not know! Just in case I will start my conversion from PGN to CGB once again without warranty of duplicates and without knowing if my CGB version will work under ChessGUI. Any feedback is welcome!

Stay tuned because I expect to finish this conversion in few hours.

Regards from Spain.

Ajedrecista.

User avatar
Ajedrecista
Posts: 1402
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 2-move engine book. Please help me again.

Post by Ajedrecista » Mon Dec 09, 2013 3:41 pm

Hi again:
Ajedrecista wrote:Hello Lyudmil:
Lyudmil Tsvetkov wrote:Hi Adam, Miguel, (Julien), Jesus, and everybody that has the slightest ooportunity to help me fot the last time. I completed the book to 1000 lines - actually 1015 to be sure 1000 will remain after doubles are eliminated; any positions after 1000 might simply be deleted to round things off. Is someone able to kindly help? Below the full text of my message on the tournament forum, where there is a download site for the new pgn file:


Hi again,
as I see that some people are interested in the book, and even Larry Kaufman (why I would like to write Kaufmann with double n?) reacted that he could use the book, I took the pains in the hurry to complete the book to 1000 positions. I added 1.f4, 1. c3, 1. a3, 1.a4 and 1.f3 lines, to make the book more exotic and interesting. Still I think none of those lines is lost; later I might post some games with those lines. I would not vouch however that 1.e4 b6 is not lost for black, and therefore there are not such lines. White has some leeway while playing. The opening advantage might not be there, but the lines are still even, and that was the main purpose: to provide some variety of play in equal lines.

Now, I have entered the new lines in chesspad (thanks to Mark van der Leek for a wonderful tool), but my problem is to check for doubles, unify headings, as well as delete the remaining plus positions over 1000 after the check for doubles is done. I simply can not do that without having to read lots of manuals. So please someone (I would not have the courage to ask Adam again, but maybe Miguel or someone else will be so helpful again) help me with the above job, so that in the end we have a book of 1000 starting positions.

The download link to the completed pgn file is here: http://www.freeuploadsite.com/do.php?id=27609

I do not know if someone will be able to convert the pgn to other formats, I could do probably ctg, but my work is finished here.
I do not have problems in help you again, but the question is if my CGB version works, which I do not know! Just in case I will start my conversion from PGN to CGB once again without warranty of duplicates and without knowing if my CGB version will work under ChessGUI. Any feedback is welcome!

Stay tuned because I expect to finish this conversion in few hours.

Regards from Spain.

Ajedrecista.
My conversion was much faster than expected and I have just finished it. Adam, thank you very much for your tip of regular expressions: they saved me a lot of time. ;)

Here is the link to 2_moves_LT_(1015_lines).cgb file:

2_moves_LT_(1015_lines).cgb.rar

The compressed file size is less than 3.5 KB; the CGB file size is under 25 KB. I hope that this version will work under ChessGUI although I have no way to check it! Any feedback is welcome.

Regards from Spain.

Ajedrecista.

Lyudmil Tsvetkov
Posts: 6052
Joined: Tue Jun 12, 2012 10:41 am

Re: 2-move engine book. Please help me again.

Post by Lyudmil Tsvetkov » Mon Dec 09, 2013 3:42 pm

Ajedrecista wrote:Hello Lyudmil:
Lyudmil Tsvetkov wrote:Hi Adam, Miguel, (Julien), Jesus, and everybody that has the slightest ooportunity to help me fot the last time. I completed the book to 1000 lines - actually 1015 to be sure 1000 will remain after doubles are eliminated; any positions after 1000 might simply be deleted to round things off. Is someone able to kindly help? Below the full text of my message on the tournament forum, where there is a download site for the new pgn file:


Hi again,
as I see that some people are interested in the book, and even Larry Kaufman (why I would like to write Kaufmann with double n?) reacted that he could use the book, I took the pains in the hurry to complete the book to 1000 positions. I added 1.f4, 1. c3, 1. a3, 1.a4 and 1.f3 lines, to make the book more exotic and interesting. Still I think none of those lines is lost; later I might post some games with those lines. I would not vouch however that 1.e4 b6 is not lost for black, and therefore there are not such lines. White has some leeway while playing. The opening advantage might not be there, but the lines are still even, and that was the main purpose: to provide some variety of play in equal lines.

Now, I have entered the new lines in chesspad (thanks to Mark van der Leek for a wonderful tool), but my problem is to check for doubles, unify headings, as well as delete the remaining plus positions over 1000 after the check for doubles is done. I simply can not do that without having to read lots of manuals. So please someone (I would not have the courage to ask Adam again, but maybe Miguel or someone else will be so helpful again) help me with the above job, so that in the end we have a book of 1000 starting positions.

The download link to the completed pgn file is here: http://www.freeuploadsite.com/do.php?id=27609

I do not know if someone will be able to convert the pgn to other formats, I could do probably ctg, but my work is finished here.
I do not have problems in help you again, but the question is if my CGB version works, which I do not know! Just in case I will start my conversion from PGN to CGB once again without warranty of duplicates and without knowing if my CGB version will work under ChessGUI. Any feedback is welcome!

Stay tuned because I expect to finish this conversion in few hours.

Regards from Spain.

Ajedrecista.
Hi Jesus,

please wait until someone eliminates the duplicates and sets the number of positions to 1000, otherwise will just waste time. Will appreciate ChessGUI version after that.

User avatar
Ajedrecista
Posts: 1402
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 2-move engine book. Please help me again.

Post by Ajedrecista » Mon Dec 09, 2013 4:00 pm

Hello Lyudmil:
Lyudmil Tsvetkov wrote:Hi Jesus,

please wait until someone eliminates the duplicates and sets the number of positions to 1000, otherwise will just waste time. Will appreciate ChessGUI version after that.
Good point! I found only a duplicate in lines 1000 and 1004:

Code: Select all

X-hb1hQ1a-jb17iQXYa2XYa

1.- c3, c6; 2.- e3, Nf6.
It is too late to make an update of my version but just removing line 1004 of my Notepad and change its extension to .cgb should do the job. I used a self-made programme of less than twenty lines of code to find duplicates among those strange strings and it only found that duplicate, so I do not expect more. Then your book has 1014 unique lines unless someone reduces them from 1014 to 1000. I do not judge if some of this positions are loses or not.

It would be good that Adam or someone else could verify my duplicate positive of lines 1000 and 1004.

Regards from Spain.

Ajedrecista.

Adam Hair
Posts: 3201
Joined: Wed May 06, 2009 8:31 pm
Location: Fuquay-Varina, North Carolina

Re: 2-move engine book. Please help me again.

Post by Adam Hair » Mon Dec 09, 2013 6:50 pm

I was having some trouble determining the number of duplicates. I have two methods for finding duplicates, and they did not agree. As it turns out, one method followed the FEN standard strictly. It considers two positions to be distinct if one of the positions has an en passant target square and the other does not, regardless of whether a capture can be made or not. The other method does not consider the positions distinct if no capture can be made. Since it is not possible to make an en passant capture within the first 2 moves, the second method (Gaviota) would be preferable for our purposes.

So, there are 12 duplicates, leaving 1003 unique positions. Here are the game numbers for the duplicates (second number represents the duplicate):

773 1015
1000 1004
575 997
951 981
954 967
774 1010
576 998
713 1012
827 969
815 1014
775 1013
772 1011

Here is the link to a compressed folder containing the book in pgn, epd, bin (Polyglot), abk (Arena), and cgb (Chessgui) formats:
***

EDIT:
It appears that there was a problem in the creation of the ChessGUI book.

EDIT2:
I am now not sure if there is a problem or not. My second attempt at making a .cgb book is identical to the first attempt. There are only 298 lines of text in the .cgb file. My assumption was that since there are 1003 unique positions, there would be 1003 lines in the .cgb file (based in part on Jesus' work). But that assumption could be wrong. Matthias will have to tell us what we should expect.

EDIT3:
New link: http://www.mediafire.com/download/jskag ... LT_1000.7z
Last edited by Adam Hair on Mon Dec 09, 2013 10:03 pm, edited 2 times in total.

Post Reply