jja: convert CTG books to PolyGlot format (and more!)

[chesskobra]

Just an observation: maybe it is the size of the CTG book being used that is creating issues?

In fact I have observed something similar when I try to write out pgn from a polyglot bin book. When I tested in on a book called Cerebellum3Merge.bin, which is quite large (nearly 178 MB), the pgn output is surprisingly small. (I had got the book from https://zipproth.de/Brainfish/download/, but that link doesn't load for me anymore.)

[Ajedrecista]

Hello:

Just an observation: maybe it is the size of the CTG book being used that is creating issues?

In fact I have observed something similar when I try to write out pgn from a polyglot bin book. When I tested in on a book called Cerebellum3Merge.bin, which is quite large (nearly 178 MB), the pgn output is surprisingly small. (I had got the book from https://zipproth.de/Brainfish/download/, but that link doesn't load for me anymore.)

It works for me. Direct link:

https://zipproth.de/Brainfish/Cerebellu ... _200916.7z

Regards from Spain.

Ajedrecista.

[alpltl]

Just an observation: maybe it is the size of the CTG book being used that is creating issues?

I have had success in converting CTG to Polyglot when the CTG book size is under 10-20 MB. However, anything larger seems not to succeed. The polyglot book is created, but no data is written (0 bytes).

I have tried using your Windows executable as well as compiling my own Linux build (Ubuntu 22.04) with your source. But the problem persists with larger CTG books.

I am happy to inform you that I have figured out the reason of the problem. The weight calculation code assigns zero weights to all entries in the book so the book ends up with zero entries because they lack performance rating/score information. You can confirm this with passing "--null" as an argument to "jja edit" in which case you should indeed have a book with non-zero size (--null tells edit to allow entries with zero weights in the book).

I am still working on a fix, and I'll release a new minor version soon after.

[alpltl]

Just an observation: maybe it is the size of the CTG book being used that is creating issues?

In fact I have observed something similar when I try to write out pgn from a polyglot bin book. When I tested in on a book called Cerebellum3Merge.bin, which is quite large (nearly 178 MB), the pgn output is surprisingly small. (I had got the book from https://zipproth.de/Brainfish/download/, but that link doesn't load for me anymore.)

This is interesting. Thanks for sharing. I'll need to delve deeper into this to figure out why the pgn ends up smaller than expected. As a rule, however, you should know that the output pgn does not include unreachable (or isolated in polyglot terms) positions/variations with regards to the initial position so it's expected that some entries will be missing from pgn.

[sarona]

I am happy to inform you that I have figured out the reason of the problem. The weight calculation code assigns zero weights to all entries in the book so the book ends up with zero entries because they lack performance rating/score information. You can confirm this with passing "--null" as an argument to "jja edit" in which case you should indeed have a book with non-zero size (--null tells edit to allow entries with zero weights in the book).

I am still working on a fix, and I'll release a new minor version soon after.

Thank you for looking into this issue.

You have created a wonderful tool in jja and I look forward to using it again.

Regards from Canada.

[alpltl]

I am happy to inform you that I have figured out the reason of the problem. The weight calculation code assigns zero weights to all entries in the book so the book ends up with zero entries because they lack performance rating/score information. You can confirm this with passing "--null" as an argument to "jja edit" in which case you should indeed have a book with non-zero size (--null tells edit to allow entries with zero weights in the book).

I am still working on a fix, and I'll release a new minor version soon after.

Thank you for looking into this issue.

You have created a wonderful tool in jja and I look forward to using it again.

Regards from Canada.

Thank you for the bug report, and the ctg book which helped me reproduce the issue. The tl;dr is your ctg book is missing performance ratings, which jja relies on to calculate weight. I have browsed through many CTG books in my archive too. This was a wrong idea appearently. What I figured out is performance and average rating information is hardly reliable because they're not always present. I have implemented a more reliable weight calculation algorithm, where we avoid zero weight at all costs. If, however, you would like to exclude, blunders (ie "red" moves), _and_ moves with no performance, average rating or win/draw/loss information from the book, you may use "--nag-weight-blunder=0" as an argument to "jja edit".

That said, here's what I have:
1. The bug report: https://todo.sr.ht/~alip/jja/12
2. The commit that fixes the issue: https://git.sr.ht/~alip/jja/commit/e806b7a
3. The git builds for the commit: windows, linux-glibc, and linux-musl.

Would you please be so kind to download a git build of your choice and check if the issue is fixed for you? If you can also take the time to check with CTG books created with different options on the ChessBase book configure window, or with different versions of ChessBase, or with different types of databases, whichever is possible, that'd be wonderful. What's important here is to check if the output PolyGlot book is generated correctly, and if the weights in the PolyGlot book look similarly sorted, compared to what you see in the ChessBase CTG window for the relevant chess position. To quickly check for a position in a PolyGlot book use "jja find -f 'FEN' book.bin" or "jja find -p 'e4 e5 Nf3' book.bin". Once again, thank you for everything and I am glad you find jja useful.

Regards from Berlin.

[sarona]

Thank you for the bug report, and the ctg book which helped me reproduce the issue. The tl;dr is your ctg book is missing performance ratings, which jja relies on to calculate weight. I have browsed through many CTG books in my archive too. This was a wrong idea appearently. What I figured out is performance and average rating information is hardly reliable because they're not always present. I have implemented a more reliable weight calculation algorithm, where we avoid zero weight at all costs. If, however, you would like to exclude, blunders (ie "red" moves), _and_ moves with no performance, average rating or win/draw/loss information from the book, you may use "--nag-weight-blunder=0" as an argument to "jja edit".

That said, here's what I have:
1. The bug report: https://todo.sr.ht/~alip/jja/12
2. The commit that fixes the issue: https://git.sr.ht/~alip/jja/commit/e806b7a
3. The git builds for the commit: windows, linux-glibc, and linux-musl.

Would you please be so kind to download a git build of your choice and check if the issue is fixed for you? If you can also take the time to check with CTG books created with different options on the ChessBase book configure window, or with different versions of ChessBase, or with different types of databases, whichever is possible, that'd be wonderful. What's important here is to check if the output PolyGlot book is generated correctly, and if the weights in the PolyGlot book look similarly sorted, compared to what you see in the ChessBase CTG window for the relevant chess position. To quickly check for a position in a PolyGlot book use "jja find -f 'FEN' book.bin" or "jja find -p 'e4 e5 Nf3' book.bin". Once again, thank you for everything and I am glad you find jja useful.

Regards from Berlin.

I have downloaded the Windows and the Linux glibc builds and will report back my results over the weekend.

Thanks again.

[Jonathan003]

I have downloaded the windows executable. I have tried it to convert the little CTG book off Magnus Carlsen's white games, to bin format, (the CTG book I have shared here).
It worked, it took about 2,5 minutes to convert the book.
I opened the bin book with Scid to compare it whit the original CTG book in Chessbase.
It looks like al moves are in the bin book, except if there is only one game with a certain move, in the original CTG book. Than this move is not always included in the bin book, (at least I can not see the move when I browse the bin book in Scid).
Sometimes the move is included and sometimes the move is not included.
It also looks like the weight of the moves in the bin book are conform to the weight of the moves in the original CTG book, at first glance.
I think that's already a good result, but I would prefer it if all moves would be in the bin book. Also the moves with only one game in the original CTG book.

[Dann Corbit]

The jja utility is really useful.
Now I can convert most book formats into something that I want to use.
For years, people have been yearning for this.
Thanks, jja team.

[alpltl]

I have downloaded the windows executable. I have tried it to convert the little CTG book off Magnus Carlsen's white games, to bin format, (the CTG book I have shared here).
It worked, it took about 2,5 minutes to convert the book.
I opened the bin book with Scid to compare it whit the original CTG book in Chessbase.
It looks like al moves are in the bin book, except if there is only one game with a certain move, in the original CTG book. Than this move is not always included in the bin book, (at least I can not see the move when I browse the bin book in Scid).
Sometimes the move is included and sometimes the move is not included.
It also looks like the weight of the moves in the bin book are conform to the weight of the moves in the original CTG book, at first glance.
I think that's already a good result, but I would prefer it if all moves would be in the bin book. Also the moves with only one game in the original CTG book.

Thanks for testing. It's good to know the main issue with the conversion is fixed. As to the problem you described, I need to delve deeper into this to understand what's going on. My first instinct is the moves are filtered out somehow because they're assigned zero weight. Can you try to run the conversion giving "--null" as an argument to "jja edit" and see if the moves appear in the bin book? It'd also help me greatly if you can provide me a sample ctg book, and list the positions/moves that you know are missing in the bin book. This way I can quickly reproduce the problem and see why this happens and hopefully fix it.