jja: convert CTG books to PolyGlot format (and more!)

[chesskobra]

Hi,

Thanks for your work. I am now testing the new binary on linux.

I created two books, one with polyglot with -min-game 256 and another with jja with -m 256 with the same pgn. I did polyglot info-book on both of them, and I got

• polyglot info-book -bin jja256.bin
  PolyGlot 2.0.4 by Fabien Letouzey.
  Lines for white : 135
  Lines for black : 107
  Positions on lines for white : 199
  Positions on lines for black : 186
  Isolated positions : 1485
  
  polyglot info-book -bin pg256.bin
  PolyGlot 2.0.4 by Fabien Letouzey.
  Lines for white : 1273
  Lines for black : 1324
  Positions on lines for white : 1817
  Positions on lines for black : 1812
  Isolated positions : 153

Maybe the difference is in the representation of the data or (very likely) I don't understand what lines and isolated positions mean exactly. The jja256.bin has many isolated positions and very few lines, while pg256 has many lines and very few isolated positions. It is very likely there is no bug, and this is expected, but if you could clarify, it will help me understand what goes into these books. I will try to dump pgn on the weekend and compare.

[alpltl]

Hi,

Thanks for your work. I am now testing the new binary on linux.

You're welcome and thanks a lot for testing.

I created two books, one with polyglot with -min-game 256 and another with jja with -m 256 with the same pgn. I did polyglot info-book on both of them, and I got

</snip>

Maybe the difference is in the representation of the data or (very likely) I don't understand what lines and isolated positions mean exactly. The jja256.bin has many isolated positions and very few lines, while pg256 has many lines and very few isolated positions. It is very likely there is no bug, and this is expected, but if you could clarify, it will help me understand what goes into these books. I will try to dump pgn on the weekend and compare.

I am also in the process of investigating this. I also don't quite get yet what isolated positions mean exactly. I have written a small script to test the discrepancies. I'm sharing it with the hope that it may help you too. I'll let you know if/when I have updates on this. Note also, I'm using -p2, ie --min-pieces=2, as the default value 8, will understandably create shorter lines than polyglot. There may be other options creating a similar effect.

Code: Select all

#!/usr/bin/env bash

data=caissabase.pgn
count=$(grep -c '^\[Event ' "$data")
echo "$count"

for i in `seq 1 ${count}`; do
	pgn-extract -s --selectonly "$i" -w32000 "$data" > tmp.pgn
	rm -f jja.{bin,pgn} poly.{bin,pgn}
	jja make -p2 -m1 -o jja.bin tmp.pgn
	polyglot make-book -min-game 1 -pgn tmp.pgn -bin poly.bin
	jja edit jja.bin -o jja.pgn
	jja edit poly.bin -o poly.pgn
	d=$(diff -u <(grep '^1' jja.pgn | sort) <(grep '^1' poly.pgn | sort))
	if [[ -n "$d" ]]; then
		echo >&2 "Discrepancy in game $i"
		break
	fi
done

[Michel]

From the manual

PolyGlot can also extract some useful information from PolyGlot books. The utility "dump-book" dumps the "lines" in a book for a given color. By definition a line is a sequence of moves (from the starting position) in which the given color makes only book moves and the other color makes arbitrary moves (i.e. not necessarily book moves).

Since a PolyGlot book is built up from positions and not lines there may be (and there usually are) many positions in the book that are not on a "line" as defined in the previous paragraph. It is convenient to call such positions "isolated" positions. The utility "info-book" counts such isolated positions.

Some of the isolated positions are provably unreachable and they could in principle be deleted from the book. For example if a book contains only the move "e4" in the starting position but also the position after "d4 d5" then this last position is provably unreachable since it requires white to make a non-book move when a book move is available. Such situations arise frequently from the priority rules in merging books.

Unfortunately not all isolated positions are provably unreachable and it is difficult to identify the latter. If invoked with "-exact" the utility info-book will attempt to count the isolated positions which require a player to make a non-book move when a book move is available. Due to the possibility of transpositions this is not a fool proof method.

[Jonathan003]

I tried to convert a small ctg book to bin format. The ctg book is made from the games, where Magnus Carlsen plays with the white pieces. About 5000 games I could find online. I don't get an error message but the created bin book has no moves (0 kb) I use a Windows 11 pc. I wonder what would be the problem?
I have made a video capture of it:



If you want you can watch the video from minute 3 onwards where the conversion begins.

[Jonathan003]

I tried to convert a small ctg book to bin format. The ctg book is made from the games, where Magnus Carlsen plays with the white pieces. About 5000 games I could find online. I don't get an error message but the created bin book has no moves (0 kb) I use a Windows 11 pc. I wonder what would be the problem?
I have made a video capture of it:



If you want you can watch the video from minute 3 onwards where the conversion begins.

I also tried by typing

Code: Select all

jja-0.5.0 edit -o Carlen_white.bin Carlsen_white.ct
g

This gives the same result, the created bin book has no moves (0 KB).

[alpltl]

I tried to convert a small ctg book to bin format. The ctg book is made from the games, where Magnus Carlsen plays with the white pieces. About 5000 games I could find online. I don't get an error message but the created bin book has no moves (0 kb) I use a Windows 11 pc. I wonder what would be the problem?
I have made a video capture of it:



If you want you can watch the video from minute 3 onwards where the conversion begins.

I also tried by typing

Code: Select all

jja-0.5.0 edit -o Carlen_white.bin Carlsen_white.ct
g

This gives the same result, the created bin book has no moves (0 KB).

Thank you for the detailed report. It is quite interesting that the book ends up as zero bytes because the progressbar seems to show everything is fine and the entries are written in the book. I have some questions:
1. (wild guess) Could it be some antivirus or some other software intervening?
2. Is it possible for you to share the ctg files with me so I can debug it further?
3. If (2) is not possible can you share the outputs of "jja info <file>.ctg" and "jja find <file>.ctg"?

[alpltl]

From the manual

Thank you very much. This explains in detail what an isolated position is, and so far I am in the progress of debugging why books generated by jja have more isolated positions than those generated by the original polyglot tool. What I can tell so far is this seems to be happening in games where repetitions happen. I first suspected en-passant encoding was to blame (EnPassantMode::Legal versus EnPassantMode::Always) but so far I have not noticed such a case. I am still not quite sure how repetitions are linked with this, I'll tell more when I know more.

[alpltl]

This gives the same result, the created bin book has no moves (0 KB).

Thank you for the detailed report. It is quite interesting that the book ends up as zero bytes because the progressbar seems to show everything is fine and the entries are written in the book. I have some questions:

I also want to let you know that we have generated a book of Carlsen's lichess games, and it is available for download here. You can read the technical details of the book generation here.

[Jonathan003]

Thank you for the detailed report. It is quite interesting that the book ends up as zero bytes because the progressbar seems to show everything is fine and the entries are written in the book. I have some questions:
1. (wild guess) Could it be some antivirus or some other software intervening?
2. Is it possible for you to share the ctg files with me so I can debug it further?
3. If (2) is not possible can you share the outputs of "jja info <file>.ctg" and "jja find <file>.ctg"?

Sure here is link to the little ctg book I used


I tried it with my Malwarebytes antivirus turned of. I also turned of Real-time protection, and other protections in Microsoft Defender on my Windows 11 64 bit PC
It didn't work, the output bin books stil has no moves (0 Kb)

[sarona]

Just an observation: maybe it is the size of the CTG book being used that is creating issues?

I have had success in converting CTG to Polyglot when the CTG book size is under 10-20 MB. However, anything larger seems not to succeed. The polyglot book is created, but no data is written (0 bytes).

I have tried using your Windows executable as well as compiling my own Linux build (Ubuntu 22.04) with your source. But the problem persists with larger CTG books.