2 Moves Engine Book

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
michiguel
Posts: 6388
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: 2 Moves Engine Book

Post by michiguel » Sun Dec 08, 2013 7:31 pm

Lyudmil Tsvetkov wrote:
Adam Hair wrote:
Lyudmil Tsvetkov wrote:
Lyudmil Tsvetkov wrote: Many thanks Adam!!

You are an absolute whizzard. I knew I should have studied computer science once.
Many thanks also to Miguel, of course, for his precious Gaviota tool. It seems that the mods here are doing an extremely useful job, hats off.

Concerning the book, that is how a computer shines over a human: I never thought there wer so many duplicates, maybe some 4/9ths, I supposed there were some when I entered the moves, but my memory failed me.

Now, one last thing about the book: would you please give me an advice if the pgn format is the best for a book, or maybe there is some other format preferable, for example bin file? What would most engine events use?
Is it possible to also add a bin file to the download option, maybe someone could help with transforming the pgn to bin?

Finally, would you mind if I use your download address to post a brief message, maybe on the main forum, to let users know there is a new book to use?

Maybe thanks again for the incredible help!
Hi Adam.
Just to ask once again if you have some comments on the above questions of mine?
Hi Lyudmil,

I have not checked for a few hours, so I missed your response.

I think the format that would be most used is pgn, but it would probably useful to make it available in various formats. Tomorrow (Sunday) I will upload a folder of the book in pgn, epd, bin, and possibly Arena and ChessGui formats. When I have uploaded the folder, I will share the link. You can freely post the link wherever you like.

I helped out because I admire your contributions to computer chess. I am no programmer, but I saw how I could use a text editor, a spreadsheet, and some computer chess tools to reformat your book. It took less than 4 hours total. However, if the book was 3 moves or more deep, I would have not made the effort. I have better things to do on the weekends :)
Many thanks Adam!
I will be waiting for the link.
If you are no programmer, I am unable to read this message. :)
I suppose all the openings now will be played with equal frequency.

I again say the mods are great.
Just to be clear, this is not part of a moderation service :-) Adam is a really helpful member, whether being a mod or not. The rest of the mod team is mean...

Adam beat me to it, but I thought yesterday that the format translation was a nice exercise and I quickly hacked a ruby script. By that time, Adam posted it.

This is the ruby script that will translate your format to pgn (named lt2pgn.rb and execute it like this

ruby lt2pgn.rb book.txt > book.pgn

If Adam uses it again, it will be neede to clean up the moves (de4 ---> dxe4), and later find duplicates, but that is pretty automatic.

Miguel
PS: Ruby is fantastic, it allows me to do this even though my knowledge of ruby is extremely limited.

Code: Select all

def get_lines filename
	lines = []
	File.open filename, "r" do |f|
		lines = f.readlines
	end
	return lines
end

def outputheader
	puts '[Event "?"]'
	puts '[Site "?"]'
	puts '[Date "????.??.??"]'
	puts '[Round "?"]'
	puts '[White "?"]'
	puts '[Black "?"]'
	puts '[Result "*"]'
end

def outputtail
	puts 
end

def outputline a
	if (a != nil)
		o = ''
		a.each do |x|
			if (x == nil)
				o += 'NIL' 
			else
				o += x 
			end
			o += ' '
		end
		puts o + '*'
	else
		puts 'NIL'
	end
end

def getfromto a,i,d
	x = []
	while &#40;i < d&#41; do
		x << a&#91;i&#93;
		i += 1
	end
	x
end

#== PROGRAM STARTS =================================

linearr = get_lines ARGV&#91;0&#93;

current = ''
clen = 0
round = 0

linearr.each do |line|
	if line != nil
		line = line.chomp.strip
		if &#40;line.length > 0&#41;
			a = line.split&#40;' ')

			alen = a.length

			if &#40;round == 0&#41;
				clen = alen
				newcurr = a
			else
				d = clen - alen
				if &#40;d < 0&#41; 
					puts 'length problems'
					exit 0
				end
				newcurr = getfromto&#40;current,0,d&#41;
				newcurr += a
			end

			outputheader
			outputline newcurr
			outputtail
			current = newcurr
			round += 1

		end
	end	
end

User avatar
hgm
Posts: 23718
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: 2 Moves Engine Book

Post by hgm » Sun Dec 08, 2013 7:58 pm

Would that really be any longer in plain C?

User avatar
michiguel
Posts: 6388
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: 2 Moves Engine Book

Post by michiguel » Sun Dec 08, 2013 8:13 pm

hgm wrote:Would that really be any longer in plain C?
Doing these things in Ruby is way much faster to write it, particularly with string manipulation (not necessarily fewer lines). Anyway, I am sure it could be written more compact in both, C and Ruby than I have done it.

Miguel

Adam Hair
Posts: 3201
Joined: Wed May 06, 2009 8:31 pm
Location: Fuquay-Varina, North Carolina

Re: 2 Moves Engine Book

Post by Adam Hair » Sun Dec 08, 2013 8:23 pm

michiguel wrote:
Lyudmil Tsvetkov wrote:
Adam Hair wrote:
Lyudmil Tsvetkov wrote:
Lyudmil Tsvetkov wrote: Many thanks Adam!!

You are an absolute whizzard. I knew I should have studied computer science once.
Many thanks also to Miguel, of course, for his precious Gaviota tool. It seems that the mods here are doing an extremely useful job, hats off.

Concerning the book, that is how a computer shines over a human: I never thought there wer so many duplicates, maybe some 4/9ths, I supposed there were some when I entered the moves, but my memory failed me.

Now, one last thing about the book: would you please give me an advice if the pgn format is the best for a book, or maybe there is some other format preferable, for example bin file? What would most engine events use?
Is it possible to also add a bin file to the download option, maybe someone could help with transforming the pgn to bin?

Finally, would you mind if I use your download address to post a brief message, maybe on the main forum, to let users know there is a new book to use?

Maybe thanks again for the incredible help!
Hi Adam.
Just to ask once again if you have some comments on the above questions of mine?
Hi Lyudmil,

I have not checked for a few hours, so I missed your response.

I think the format that would be most used is pgn, but it would probably useful to make it available in various formats. Tomorrow (Sunday) I will upload a folder of the book in pgn, epd, bin, and possibly Arena and ChessGui formats. When I have uploaded the folder, I will share the link. You can freely post the link wherever you like.

I helped out because I admire your contributions to computer chess. I am no programmer, but I saw how I could use a text editor, a spreadsheet, and some computer chess tools to reformat your book. It took less than 4 hours total. However, if the book was 3 moves or more deep, I would have not made the effort. I have better things to do on the weekends :)
Many thanks Adam!
I will be waiting for the link.
If you are no programmer, I am unable to read this message. :)
I suppose all the openings now will be played with equal frequency.

I again say the mods are great.
Just to be clear, this is not part of a moderation service :-) Adam is a really helpful member, whether being a mod or not. The rest of the mod team is mean...

Adam beat me to it, but I thought yesterday that the format translation was a nice exercise and I quickly hacked a ruby script. By that time, Adam posted it.

This is the ruby script that will translate your format to pgn (named lt2pgn.rb and execute it like this

ruby lt2pgn.rb book.txt > book.pgn

If Adam uses it again, it will be neede to clean up the moves (de4 ---> dxe4), and later find duplicates, but that is pretty automatic.

Miguel
PS: Ruby is fantastic, it allows me to do this even though my knowledge of ruby is extremely limited.

Code: Select all

def get_lines filename
	lines = &#91;&#93;
	File.open filename, "r" do |f|
		lines = f.readlines
	end
	return lines
end

def outputheader
	puts '&#91;Event "?"&#93;'
	puts '&#91;Site "?"&#93;'
	puts '&#91;Date "????.??.??"&#93;'
	puts '&#91;Round "?"&#93;'
	puts '&#91;White "?"&#93;'
	puts '&#91;Black "?"&#93;'
	puts '&#91;Result "*"&#93;'
end

def outputtail
	puts 
end

def outputline a
	if &#40;a != nil&#41;
		o = ''
		a.each do |x|
			if &#40;x == nil&#41;
				o += 'NIL' 
			else
				o += x 
			end
			o += ' '
		end
		puts o + '*'
	else
		puts 'NIL'
	end
end

def getfromto a,i,d
	x = &#91;&#93;
	while &#40;i < d&#41; do
		x << a&#91;i&#93;
		i += 1
	end
	x
end

#== PROGRAM STARTS =================================

linearr = get_lines ARGV&#91;0&#93;

current = ''
clen = 0
round = 0

linearr.each do |line|
	if line != nil
		line = line.chomp.strip
		if &#40;line.length > 0&#41;
			a = line.split&#40;' ')

			alen = a.length

			if &#40;round == 0&#41;
				clen = alen
				newcurr = a
			else
				d = clen - alen
				if &#40;d < 0&#41; 
					puts 'length problems'
					exit 0
				end
				newcurr = getfromto&#40;current,0,d&#41;
				newcurr += a
			end

			outputheader
			outputline newcurr
			outputtail
			current = newcurr
			round += 1

		end
	end	
end
I have two thoughts:

1) Crap! I could have spent my 4 hours doing something else.
2) Great! I have another script that I can modify if necessary.

User avatar
Ajedrecista
Posts: 1398
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 2-move engine book.

Post by Ajedrecista » Sun Dec 08, 2013 8:33 pm

Hello:
Adam Hair wrote:Here is the link to the book in various formats:

http://www.mediafire.com/download/56dcg ... oves_LT.7z

The file contains the book in pgn, epd, bin, and Arena format. Perhaps some else can handle other formats such as ChessGUI, ChessBase, and Shredder.

The bin and Arena books are uniform, meaning that all openings are weighed the same and should occur with the same frequency.

I have provided my part of the work with no strings attached, so feel free to do as you will with the link and files.
I do not have ChessGUI but I took a look to *.cgb notation just opening Grob.cgb and Nimzowitsch-Larsen.cgb with Notepad (both downloaded from Kirill Kryukov's site, probably uploaded by Graham), and I created the full book 2moves_LT.cgb from scratch (913 lines), replacing things of the PGN into this empty Notepad. I post here the first ten lines with the only purpose of verification:

Code: Select all

X-gb1gQ1a-6ga16gRXYa2XYa
X-gb1gQ1a-6ga1OKXYa2XYa
X-gb1gQ1a-6ga1jQXYa2XYa
X-gb1gQ1a-6ga17iQXYa2XYa
X-gb1gQ1a-Oc1OKXYa2XYa
X-gb1gQ1a-Oc17iQXYa2XYa
X-gb1gQ1a-Oc1jQXYa2XYa
X-gb1gQ1a-Oc1BQXYa2XYa
X-gb1gQ1a-jb1OKXYa2XYa
X-gb1gQ1a-jb1jQXYa2XYa
I am not sure if it will work. Just save a file with .cgb extension: if these ten lines match the first ten lines of the PGN, then I will be more sure about my work. Sorry but I do not have time right now to explain my conversion method and do a proper release. I just ask if this preview works. If that, there will be a release tomorrow.

Regards from Spain.

Ajedrecista.

User avatar
hgm
Posts: 23718
Joined: Fri Mar 10, 2006 9:06 am
Location: Amsterdam
Full name: H G Muller
Contact:

Re: 2 Moves Engine Book

Post by hgm » Sun Dec 08, 2013 9:23 pm

michiguel wrote:Doing these things in Ruby is way much faster to write it, particularly with string manipulation (not necessarily fewer lines). Anyway, I am sure it could be written more compact in both, C and Ruby than I have done it.
Well, I don't know Ruby at all, so it is hard to judge for me how easy that makes it. But the task at hand seems quite trivial in C. It is basically just a matter of copying the indented lines over the end of the previous non-indented lines, starting at various positions. That is a single strcpy call, and counting some spaces.

Rein Halbersma
Posts: 685
Joined: Tue May 22, 2007 9:13 am

Re: 2 Moves Engine Book

Post by Rein Halbersma » Sun Dec 08, 2013 10:42 pm

hgm wrote:
michiguel wrote:Doing these things in Ruby is way much faster to write it, particularly with string manipulation (not necessarily fewer lines). Anyway, I am sure it could be written more compact in both, C and Ruby than I have done it.
Well, I don't know Ruby at all, so it is hard to judge for me how easy that makes it. But the task at hand seems quite trivial in C. It is basically just a matter of copying the indented lines over the end of the previous non-indented lines, starting at various positions. That is a single strcpy call, and counting some spaces.
It's not my intention to mock you in any way, but you do have to appreciate the irony of using the words "trivial in C" and a "single strcpy call" so close together, given the enormous thread going on :D

User avatar
michiguel
Posts: 6388
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: 2 Moves Engine Book

Post by michiguel » Sun Dec 08, 2013 10:52 pm

hgm wrote:
michiguel wrote:Doing these things in Ruby is way much faster to write it, particularly with string manipulation (not necessarily fewer lines). Anyway, I am sure it could be written more compact in both, C and Ruby than I have done it.
Well, I don't know Ruby at all, so it is hard to judge for me how easy that makes it. But the task at hand seems quite trivial in C. It is basically just a matter of copying the indented lines over the end of the previous non-indented lines, starting at various positions. That is a single strcpy call, and counting some spaces.
It is a different way of thinking. That is because of the many high level tools available. You can think in terms of tokens, rather than chars. When you do

a = s.chomp.trim.split(' ')

You take the string s and you get an array of tokens in a. Carriage returns, tabs, spaces etc. have been removed and tokens were separated by spaces. It does not matter how long the string is, and how long the array will be. Everything is handled dynamically under the hood.

In this particular case, words were not evenly spaced and to make matters less "C" friendly, there were tabs mixed with spaces. Of course I could have done with C, but then I would have had to pay much more attention to the input format.

Miguel

User avatar
Ajedrecista
Posts: 1398
Joined: Wed Jul 13, 2011 7:04 pm
Location: Madrid, Spain.
Contact:

Re: 2-move engine book.

Post by Ajedrecista » Mon Dec 09, 2013 10:16 am

Hello again:

It seems that nobody tried my small sample of 2moves_LT.cgb book, so anyway I release it today WITHOUT ANY WARRANTY. If it works, perfect; if not, sorry, I tried it and failed. I will not spent more time in this task.

Firstly, I want to describe my strange conversion method from *.pgn to *.cgb standard: I used Grob.cgb (the second post of the thread) and Nimzowitsch-Larsen.cgb (the first post of the thread) as guidelines. I opened them with Notepad (not Notepad++, which will be used later) and I spent some minutes without result until things began to match in my slow mind. I induced a chart of conversion of characters between the two formats. Here are my findings:

Code: Select all

.pgn   .cgb
-----------
  a      .
  b      g
  c      h
  d      O
  e      j
  f      i
  g      B
  h      N
  x      x
  1      Unknown to me.
  2      a
  3      b
  4      c
  5      K
  6      Q
  7      R
  8      d
  K      Unknown to me.
  Q      4
  R      Unknown to me.
  B      6
  N      7
  +
That is, I think that a check (+) is not indicated with any sign in CGB books. A checkmate (#) is also unknown to me. More things: the initial FEN of a chess game is denoted with X-, the space between white and black move is denoted with 1, the space between (n-1)-th black move and n-th white move (moves, not plies) is indicated with 1&-, where & is n in CGB notation (for example, between the first black move and the second white move, the separator is 1a-). Castlings and promotions are unknown to me and the finish of an opening book line is marked with XYa2XYa (Matthias Gemuh could verify or disprove my statements).

I deleted all the PGN tags and 1/2-1/2 results at the end of each line (it was easy with Notepad) but ECO codes were more time consuming to delete because of their variety. Once I got the PGN as I wanted, I copied it in Notepad++ and did the replacements (Notepad++ is much faster than Notepad for this task) with extreme care: I had luck because there were not moves in the rank 1 (I mean, Nb1 for example)... please remember that rank 1 conversion is unknown to me, as well as castlings and other things explained above. I began with initial FENs (X-) and separators between moves. Then I did the replacements by squares (a2 |---> .a and many more) and finally I took care of pieces (I had good luck again because there were neither king moves nor rook moves) in the following way: Bf4, Bf5, etc. were handled with the piece type and their file (Bf) but not their rank, so Bf was converted to 6i and so on. The last task was assign each opening book line to a physical line, therefore I had to force newlines (or line breaks or whatever they are called). I had no clue but Google is too much Google and \r\n did the trick in Notepad++. That is all!

I uploaded the file 2moves_LT.cgb with its duplicate in .txt extension, just to make easy corrections if needed. This link will die 30 days after the last download.

2moves_LT.cgb.rar (3 KB)

2moves_LT.cgb file size is 22.35 KB more less. I will appreciate feedback (basically 'it works!' or 'it does not work') and in the case of positive feedback, mirror links are accepted as expected.

Regards from Spain.

Ajedrecista.

Adam Hair
Posts: 3201
Joined: Wed May 06, 2009 8:31 pm
Location: Fuquay-Varina, North Carolina

Re: 2-move engine book.

Post by Adam Hair » Mon Dec 09, 2013 11:42 am

Hi Jesús,

In NotePad++:

Find
Replace
Check Regular expression

Find:\[ECO\s.+
Replace:

Study regular expressions. I do not know a lot, but regular expressions are quite helpful when trying to modify a text file or extract data.

Also, add the TextFX plugins to NotePad++. Then you can remove all blank lines at one time.

Post Reply