phhnguyen wrote: ↑Thu Jun 14, 2018 2:04 am
Thanks for your answer.
Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm
You can try to play some games say 100 games or more. Get 2 same engines say sf9, one using the old book and the other using the new book. If new book performs above 50% then it is better.
AFAIK, many engines such as sf (and mine) don't use their own books. In other hand, many popular GUI such as XBoard, Cutechess... don't allow to use more than one opening books.
How do you organise the test for two books?
SF can use polyglot book, most if not all uci engines can use polyglot book. In my reply I assume that you use polyglot book sorry. Most GUI's support 2 modes of opening book. (1) GUI opening book and (2) Engine opening book. You can disable (1) and use book via (2).
You can setup polyglot to act as an engine and use a book of your own choice. You need polyglot.ini, polyglot.exe and uci engine say SF9
Sample polyglot.ini
Code: Select all
[PolyGlot]
EngineCommand=SF9.exe
EngineDir=.
Book=true
BookFile=myoldbook.bin
[Engine]
Hash=128
In that setting, SF9 will use myoldbook.bin book.
1. Create a dir and name it engine1 and copy polyglot.exe, polyglot.ini and SF9.exe
2. Create another dir and name it engine2 and copy polyglot.exe, polyglot.ini and SF9.exe
In this case use a different polyglot.ini entry.
BookFile=mynewbook.bin
You can then install the engine by selecting polyglot.exe.
phhnguyen wrote:
I may add more code to my engines to use books, but it takes time, labor and I don't think my engine is good enough to confirm a given book is good or not.
Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm
Verify it by conductiong additional game tests using other book or books.
sf9_newbook vs sf9_otherbook, 100 games or more
sf9_oldbook vs sf9_otherbook, 100 games or more
I don't have experience of book testing, but I am confused about the number 100 since it is too small to for a test. Are you sure it is enough?
You can run test games and run ordo or bayeselo program to see if the lead of new book is statistically significant.
phhnguyen wrote:
Ferdy wrote: ↑Wed Jun 13, 2018 4:28 pm
if sf9_newbook performs better than sf9_oldbook against common different book then it is really better.
You may test it on two diffeent TC (Time Control), say 1m+1s and 5m+1s, there can be book lines that are good only on fast TC.
Some issues / questions still confuse me, need some suggestions:
- Results of games for a given opening line are enough to confirm that opening line is good? I am confused since for very long games it is hard to say how their opening lines affect to results.
Go for the basic thing, If newbook won in a given specific line then it is good because it won that line. If you dig deeper, check the game and examine each move score, analyze each position, does its opponent blunders? For long TC game, the book line is more reliable, because each engine has more time to think on the move.
phhnguyen wrote:- If I always select the best answers from openings, look like I can test only one opening line.
Correct, this is why it is recommended to test your best lines against other books, not just your oldbook. Those other books may have different line continuation that might equalize or worst defeat your best line.
phhnguyen wrote: If I select randomly (to test many opening lines), the results may have too much noise to make any conclusion
Running more test games is better here to see performance of your book when run on random mode.