Do you test engines with or without using a book?

[JohnW]

Just curious how most people test engines when running a match or tournament. I usually run them without.

[Terje]

With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.

[JohnW]

With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.

But if you use a book aren't you testing from a situation that could give one side an advantage without ever putting any thought into it?

[Terje]

With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.

But if you use a book aren't you testing from a situation that could give one side an advantage without ever putting any thought into it?

Both engines (should) get to play each side of each position. Also startpos already gives white an advantage without the engines doing anything

[Dann Corbit]

With no book you're just testing a handful of lines from startpos. If your engine picks moves somewhat randomly in the opening then you avoid this problem, but this is the job of an opening book, not the engine really.

But if you use a book aren't you testing from a situation that could give one side an advantage without ever putting any thought into it?

That depends on whether you put any thought into the book or not.

[Ovyron]

Both engines (should) get to play each side of each position. Also startpos already gives white an advantage without the engines doing anything

This is a myth believed by most testers and rating lists. In reality humans aren't forced to play the same openings with both colors, they play the openings they know well, and the same should happen with engines, building for them a book that makes them go into positions they understand better, to show their true strength.

This was critical on the days where Stockfish was still weaker than Rybka, Rybka was only stronger on these "repeated openings" set up, but Stockfish was clearly stronger if you guided both engines to the positions they played best (you could always force the game into positions that Stockfish played better than Rybka - even with black, but you couldn't force the game into positions where Rybka was better than Stockfish, even though on a random sample of generic openings Rybka would score better.)

Nowadays Stockfish 12+ will come up on top no matter what method you use, so the point is moot (but there could theoretically be a strongest chess entity that would be engine +book that would beat Stockfish 12+, and it'd appear weaker than that in the rating lists because of the generic openings.)

[Gabor Szots]

We test with books (short ones) to inroduce some variety to the openings.

To play the same opening with both colours is not important, when you play hundreds of games that will even out.

[supersharp77]

Just curious how most people test engines when running a match or tournament. I usually run them without.

Well my opinion is you should use a book...you must anyway because many of these engines have their own books anyway (ex. Deep Junior, Arasan Pedone Crafty) so you need a book sometimes...Chessmaster (The King) plays quite well without a book some engines (Deep Junior, Open Tal Crafty) don't play their proper style with out the proper book...attacking engines need a sharp book...positional engines do better with a quiet book...LC0 plays ok without a book but plays a bit better with a appropriate book (D-4 lines) my opinion is that these books line should not go too deep (8-9 moves) plus playing the same lines over and over is quite boring to watch...The sharper and more complex the opening lines are the better it is and the more we can learn....AR


ps You can mix and match and create opening books in the Fritz, Chess Ok and Arena GUI without any problems..

[hgm]

In Fairy-Max I randomize the first few moves quite heavily. (Every move that isn't more than 50cP below the best is eligible.) So I can test it without book. That was important for me, as in most variants it plays no theory exist. And I often start with meterial odds, so that even when theory would exist, it would not apply.

If you play many opening lines, it isn't really important to play them with reversed colors. As long as you randomly assign the lines, and they are not extremely unbalanced. You don't want the latter anyway; playing games that are a certain win is just a waste of time. And if you play them with reversed colors a double waste of time.

[Michel]

Replaying games with reversed colors reduces the variance of the test outcome (one should use the pentanomial model to correctly estimate this variance). So you need fewer games to reach a decision. This effect is quite substantial. Fishtest (which is the gold standard in engine testing) uses a very balanced book and there is still a 5% saving. With their previous slightly less balanced book it was 10%. With very unbalanced books it is much more.

The math is discussed here.

http://hardy.uhasselt.be/Fishtest/accou ... entity.pdf

This document is actually about comparing the trinomial and the pentanomial model, but this is the same problem.