TalkChess.com

Posted: **Wed Jun 13, 2018 1:07 pm**

Please share some your experiences/methodologies to test new/twisted opening books.

I have just created a new opening book from carefully-selected games but not sure if it’s better than old ones.

Thanks

Posted: **Wed Jun 13, 2018 4:28 pm**

phhnguyen wrote: ↑Wed Jun 13, 2018 1:07 pm Please share some your experiences/methodologies to test new/twisted opening books.

I have just created a new opening book from carefully-selected games but not sure if it’s better than old ones.

Thanks

You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.

Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more

if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.

Posted: **Thu Jun 14, 2018 2:04 am**

Thanks for your answer.

Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.

AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.

How do you organise the test for two books?

I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.

Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more

I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?

Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.

Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
- If I always select the best answers from openings, look like I can test only one opening line. If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion

Posted: **Thu Jun 14, 2018 6:30 am**

phhnguyen wrote: ↑Thu Jun 14, 2018 2:04 am Thanks for your answer.

Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.
AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.

How do you organise the test for two books?

SF can use polyglot book, most if not all uci engines can use polyglot book. In my reply I assume that you use polyglot book sorry. Most GUI's support 2 modes of opening book. (1) GUI opening book and (2) Engine opening book. You can disable (1) and use book via (2).

You can setup polyglot to act as an engine and use a book of your own choice. You need polyglot.ini, polyglot.exe and uci engine say SF9

Sample polyglot.ini

Code: Select all

[PolyGlot]
EngineCommand=SF9.exe
EngineDir=.
Book=true
BookFile=myoldbook.bin
[Engine]
Hash=128

In that setting, SF9 will use myoldbook.bin book.

1. Create a dir and name it engine1 and copy polyglot.exe, polyglot.ini and SF9.exe
2. Create another dir and name it engine2 and copy polyglot.exe, polyglot.ini and SF9.exe
In this case use a different polyglot.ini entry.
BookFile=mynewbook.bin
You can then install the engine by selecting polyglot.exe.

phhnguyen wrote: I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.

Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more
I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?

You can run test games and run ordo or bayeselo program to see if the lead of new book is statistically significant.

phhnguyen wrote:
Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.

You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.
Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.

Go for the basic thing, If newbook won in a given specific line then it is good because it won that line. If you dig deeper, check the game and examine each move score, analyze each position, does its opponent blunders? For long TC game, the book line is more reliable, because each engine has more time to think on the move.

phhnguyen wrote:- If I always select the best answers from openings, look like I can test only one opening line.

Correct, this is why it is recommended to test your best lines against other books, not just your oldbook. Those other books may have different line continuation that might equalize or worst defeat your best line.

phhnguyen wrote: If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion

Running more test games is better here to see performance of your book when run on random mode.

Posted: **Wed Jun 27, 2018 4:00 pm**

Would this be correct, so that my engine would take the book .bin?

The folder is located in c: / desktop / New folder

Thanks for your help.

Posted: **Wed Jun 27, 2018 4:03 pm**

This is the .INI file

Posted: **Wed Jun 27, 2018 4:05 pm**

When an engine is UCI, you can run it through Polyglot, and then let Polyglot handle the book. That way the book is only used for a single engine.

If you use WinBoard/XBoard you can set the Polyglot book for the individual engines through their Engine #N Settings dialog. You would only have a problem if both engines are WB, and both engines do not support an own book. (Traditionally WB engines hanle their books themselves.)

TalkChess.com

How to properly test new/changed opening books?

How to properly test new/changed opening books?

Re: How to properly test new/changed opening books?

Re: How to properly test new/changed opening books?

Re: How to properly test new/changed opening books?

Re: How to properly test new/changed opening books?

Re: How to properly test new/changed opening books?

Re: How to properly test new/changed opening books?