Polyglot - merge-book oddity

[Rebel]

There are numerous Polyglot versions available, the ones I tried all give the same oddity.

Merging 2 books - book1 with only 1.e4 openings and book2 with 1.d4, 1.c4, 1.Nf3 etc. openings will result that the merged book will only show (play) 1.e4

Is there a known fix?

[amanjpro]

There are numerous Polyglot versions available, the ones I tried all give the same oddity.

Merging 2 books - book1 with only 1.e4 openings and book2 with 1.d4, 1.c4, 1.Nf3 etc. openings will result that the merged book will only show (play) 1.e4

Is there a known fix?

That is interesting, I thought simply concatenating two polyglot books should properly merge them (after all the format is a bunch of records without headers or anything)

[amanjpro]

There are numerous Polyglot versions available, the ones I tried all give the same oddity.

Merging 2 books - book1 with only 1.e4 openings and book2 with 1.d4, 1.c4, 1.Nf3 etc. openings will result that the merged book will only show (play) 1.e4

Is there a known fix?

That is interesting, I thought simply concatenating two polyglot books should properly merge them (after all the format is a bunch of records without headers or anything)

In Windows you may want to try this:

Code: Select all

copy file1 + file2 targetfile /B 

Or if you are using *nix, then a simple cat should do:

Code: Select all

cat file1 file2 > target_file

[Norm Pollock]

This is an excellent Windows tool for polyglot books

https://chess.massimilianogoi.com/downl ... ttolerant/

[MikeB]

There are numerous Polyglot versions available, the ones I tried all give the same oddity.

Merging 2 books - book1 with only 1.e4 openings and book2 with 1.d4, 1.c4, 1.Nf3 etc. openings will result that the merged book will only show (play) 1.e4

Is there a known fix?

Ed,
The first file "-in1" has priority over the "in2" file.

polyglot merge-book -in1 w1.bin -in2 w2.bin -out w.bin

Apparently the main purpose of the first book is to drive the desired play and book 2 is to cover more obscure lines.
So you should have all lines that you want priority in book 1 either weighted or uniform depending on your needs.

The source of the above information is from the link below about 5/8 of the way down
http://www.wbec-ridderkerk.nl/html/deta ... yGlot.html

[Rebel]

This is an excellent Windows tool for polyglot books

https://chess.massimilianogoi.com/downl ... ttolerant/

Same trouble.

[Rebel]

There are numerous Polyglot versions available, the ones I tried all give the same oddity.

Merging 2 books - book1 with only 1.e4 openings and book2 with 1.d4, 1.c4, 1.Nf3 etc. openings will result that the merged book will only show (play) 1.e4

Is there a known fix?

Ed,
The first file "-in1" has priority over the "in2" file.

polyglot merge-book -in1 w1.bin -in2 w2.bin -out w.bin

Apparently the main purpose of the first book is to drive the desired play and book 2 is to cover more obscure lines.
So you should have all lines that you want priority in book 1 either weighted or uniform depending on your needs.

The source of the above information is from the link below about 5/8 of the way down
http://www.wbec-ridderkerk.nl/html/deta ... yGlot.html

I know all that. I also understand why it has been done the way it has been done, the bottleneck are the weights and the lack of WDL info to recalculate the weights in the above (blue) case I mentioned. The move 1.e4 has a weight of 100%, the moves 1.d4, 1.c4, 1.Nf3 in the second book have their own weights. In order to merge 1.d4, 1.c4, 1.Nf3 into the new book the weights of 1.e4, 1.d4, 1.c4, 1.Nf3 needs recalculation and that's a problem, hence the moves 1.d4, 1.c4, 1.Nf3 are skipped, creating another problem. I am hoping that someone once has tackled the problem else I will have to do it myself.

[hgm]

There is a difference between "only show", and "only play". So what is the case here? If it only plays 1.d4 in a finite number of cases it could just be that the weight of e4 is so much smaller than that of d4 in the initial position that you would have had to be very lucky to see e4 played. It doesn't prove there is anything wrong with the book or the merge.

The expected behavior of a merge operation would produce the same book as would be produced when first concatenating the PGN sets from which the individual books were created, and convert that to a book. This would be achieved by sorting all the entries in the combined books by hash key, and for equal key in mve order, and then combine entries with equal moves by adding their weights. (Note that concatenating the books would not work, as book probing is done by bisection, which requires the entries to be in order of ascending hash key.) The remark about 'priority' suggest Polyglot doesn't do the latter, but would simply delete the entry from the second book. (Which is highly illogical, btw: you would have to sort by move to recognize the duplicats, and when you do adding the weights of the entries you are going to delete to the one you are going to keep is next to zero work. If you want to be lazy you would ignore the problem of duplicats altogether, and leave them in. But then the book would still behave as if you had kept one with the total weight, it would only be unneccessarily big.)

If one book has only move e4 for the initial position, and the other d4, none of this should matter, and the merged book should always containg both moves for the hash key of the initial position. No matter how sloppily it was implemented. If not, the merge function is completely buggy.

So just show the list of moves it gives for the initial position in the merged book and the component books, then we can see wheter the merge was executed properly.

[Edit] I had a peek in the code, and it is indeed badly flawed. If the two books contain the same position, all info on this position from the second book will be completely ignored. Even if it are all completely different moves. I now made a quick (but dirty) fix of that in my on-line repository (learn branch), just adding the entries for that position from both books. (Without bothering to remove or collapse duplicats.)

[jefk]

a simple 'fix'/trick would be to start with the 1.d4 etc book (-in1), and
add the 1.e4 book as secondary (-in2) , but apparently there's already
a sourcecode fix, thx to hgm, who wrote

made a quick (but dirty) fix of that in my on-line repository (learn branch), just adding the entries for that position from both books

found it:
http://hgm.nubati.net/cgi-bin/gitweb.cg ... d6ec680cf6

is there also a new binary/compile somewhere ?

NB on github i found two polyglot repositories:
https://github.com/sshivaji/polyglot
https://github.com/flok99/polyglot

a bit confusing, isn't it

[Guenther]

NB on github i found two polyglot repositories:
https://github.com/sshivaji/polyglot
https://github.com/flok99/polyglot

a bit confusing, isn't it

Why is that confusing for you? You can find numerous forks of anything at github...