Dear Adam,
First of all,
This thread is not about: what is the ideal way of testing the chess engines!))
But however,
Your opinion is greatly appreciated and very interesting...so let's discuss)
Btw, here is the ideal rating list (in my opinion):
-Games should be available with annotations
-Engines should be tested as far as possible with strongest openings (up to 10-12 moves)
-Live tools should not be allowed on same Tournament PC
-Engines should be tested with Ponder ON
-Engines should be tested as MP
-Multiple matches should not be allowed on same Tournament PC
-Clone and Derivative engines should not be allowed...only in case of 100 Elo stronger!)
-Adapted time controls and many different hardwares should not be allowed
etc...
And is there a such available rating?, of course no !!
But however,I think some ratings are more near to ideal way of testing ...)!
And I am not going to count which ones, because I am afraid to miss some of them...))!!
And now let's back again to the current issue,
If you check my 1st posting...you will notice: why the testers prefer a lot of weaker opening lines?!
And still no any answer...)))
Adam Hair wrote:
In general, the problem is with using the book format for openings when conducting engine testing.
Books are appropriate for contests. PGNs and EPDs are much better for engine testing.
Bias is introduced, which leads to error in the Elo estimation.
Are sure about this ?
The chess engines do not have hearts, but they have brains!
A little note: the Top Grandmaster Elo points are not are based on PGNs and EPDs positions!
And why the Top Chess Engines Elo points should be based especially on PGNs and EPDs positions ?
Furthermore,
And I don't see nothing wrong in case of using opening books
Plus,
It's much more important to use stronger opening lines than those handicapped opening positions
Actually there is one true,
if we use many weak positions, only then it leads to error in the Elo estimation!
As far as I know,
The below rating lists are created mainly by using opening books, correct me if I am wrong ?
CCRL
CEGT
FCT
Jurek
Marijan
OWL
SCCT
SSDF
----------------
The below rating list are created by using PGNs and EPDs positions:
IPON
FGRL
Note: in FGRL rating is leader Stockfish, but in IPON is leading Komodo !
Even on ratings based on PGNs and EPDs positions, we see different leaders!
We see different standings, probably due to both ratings are used different opening lines,time control etc....
But you have right,
That there is nothing wrong in case of using PGNs and EPDs positions, but be sure those positions to be the strongest!
Adam Hair wrote:
I would like to point out that this is not an attack against you, Sedat. And this is not a critique of your books. My purpose is to inform that a book is not the best openings format if someone is conducting engine testing with the purpose of computing ratings.
Btw, sometimes I am a little bit in emotion...please don't look at this...
I respect you as good chess friend of mine...I still remember your help during my Scratch testing!
Plus...you was one of the 1st friends who supported my Stockfish bemnchmark project, I will never forget this!
Best,
Sedat
