How to properly test new/changed opening books?

[phhnguyen]

Please share some your experiences/methodologies to test new/twisted opening books.

I have just created a new opening book from carefully-selected games but not sure if it’s better than old ones.

Thanks

[Ferdy]

Please share some your experiences/methodologies to test new/twisted opening books.

I have just created a new opening book from carefully-selected games but not sure if it’s better than old ones.

Thanks

You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.

Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more

if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.

[phhnguyen]

Thanks for your answer.

You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.

AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.

How do you organise the test for two books?

I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.

Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more

I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?

if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.

Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
- If I always select the best answers from openings, look like I can test only one opening line. If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion

[Ferdy]

Thanks for your answer.

You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.

AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.

How do you organise the test for two books?

SF can use polyglot book, most if not all uci engines can use polyglot book. In my reply I assume that you use polyglot book sorry. Most GUI's support 2 modes of opening book. (1) GUI opening book and (2) Engine opening book. You can disable (1) and use book via (2).

You can setup polyglot to act as an engine and use a book of your own choice. You need polyglot.ini, polyglot.exe and uci engine say SF9

Sample polyglot.ini

Code: Select all

[PolyGlot]
EngineCommand=SF9.exe
EngineDir=.
Book=true
BookFile=myoldbook.bin
[Engine]
Hash=128

In that setting, SF9 will use myoldbook.bin book.

1. Create a dir and name it engine1 and copy polyglot.exe, polyglot.ini and SF9.exe
2. Create another dir and name it engine2 and copy polyglot.exe, polyglot.ini and SF9.exe
In this case use a different polyglot.ini entry.
BookFile=mynewbook.bin
You can then install the engine by selecting polyglot.exe.

I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.

Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more

I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?

You can run test games and run ordo or bayeselo program to see if the lead of new book is statistically significant.

if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.

Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.

Go for the basic thing, If newbook won in a given specific line then it is good because it won that line. If you dig deeper, check the game and examine each move score, analyze each position, does its opponent blunders? For long TC game, the book line is more reliable, because each engine has more time to think on the move.

- If I always select the best answers from openings, look like I can test only one opening line.

Correct, this is why it is recommended to test your best lines against other books, not just your oldbook. Those other books may have different line continuation that might equalize or worst defeat your best line.

If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion

Running more test games is better here to see performance of your book when run on random mode.

[fantasmadel50]

Would this be correct, so that my engine would take the book .bin?

The folder is located in c: / desktop / New folder


Thanks for your help.

[fantasmadel50]

This is the .INI file

[hgm]

When an engine is UCI, you can run it through Polyglot, and then let Polyglot handle the book. That way the book is only used for a single engine.

If you use WinBoard/XBoard you can set the Polyglot book for the individual engines through their Engine #N Settings dialog. You would only have a problem if both engines are WB, and both engines do not support an own book. (Traditionally WB engines hanle their books themselves.)