ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

building a good test suite to predict the rating of engines
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Don Dailey



Joined: 29 Apr 2008
Posts: 4322
PostPost subject: Re: building a good test suite to predict the rating of engi    Posted: Mon Jun 11, 2012 5:06 pm Reply to topic Reply with quote
Uri Blass wrote:
The idea is that the test suite can be used by programmers to improve their engine when they simply tune for better results in the test suite and do not tune for better result in games because testing for games take more time and in 99% of the cases better results in the test suite means also better result in games(assuming that we do not talk about time management changes but about evaluation or search changes).

I wonder if there are programmers who work about building this type of test suite including the scoring system.

Note that I do not think that existing tactical test suites are good for this purpose but it does not mean that it is impossible to build a test suite that is good for this purpose and after you build it you can save testing time by not playing games against other programs.


We have such a set. I think it is quite good at measuring whether a given program has improved and it correlates highly with general strength too, but not perfectly.

A good set to do this is still subject to the same error margins you see in playing matches so there are no real shortcuts to take and unlike real games certain aspects of a chess program cannot be tested like this such as time control algorithms. Each program has it's own sense of good and bad and these tests don't measure how well a program is able to push it's own agenda, a key factor in being strong. Nevertheless I think it's possible for such a set to be pretty useful. Our set has about 150,000 positions and we don't make much use of it, but when we make a small improvement it almost always checks out on the set (when we use it) as being slightly better too.
_________________
"Your superior intellect is no match for our puny weapons." -Kang and Kodos
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Subject Author Date/Time
building a good test suite to predict the rating of engines Uri Blass Sun Jun 10, 2012 12:00 pm
      Re: building a good test suite to predict the rating of engi Adam Hair Sun Jun 10, 2012 12:44 pm
      Re: building a good test suite to predict the rating of engi Dan Honeycutt Sun Jun 10, 2012 3:08 pm
            Re: building a good test suite to predict the rating of engi Uri Blass Mon Jun 11, 2012 5:17 am
                  Re: building a good test suite to predict the rating of engi Uri Blass Mon Jun 11, 2012 5:23 am
      Re: building a good test suite to predict the rating of engi Kevin Hearn Sun Jun 10, 2012 7:53 pm
      Re: building a good test suite to predict the rating of engi Don Dailey Mon Jun 11, 2012 5:06 pm
      Re: building a good test suite to predict the rating of engi John Merlino Mon Jun 11, 2012 6:50 pm
            Re: building a good test suite to predict the rating of engi Richard Allbert Sun Aug 05, 2012 6:43 am
      Re: building a good test suite to predict the rating of engi Jens Bęk Nielsen Sat Aug 04, 2012 10:56 pm
            Re: building a good test suite to predict the rating of engi Jens Bęk Nielsen Mon Aug 06, 2012 4:16 pm
            Re: building a good test suite to predict the rating of engi Yves Lejeail Thu Aug 09, 2012 2:36 pm
                  Re: building a good test suite to predict the rating of engi Jens Bęk Nielsen Thu Aug 09, 2012 5:59 pm
                        Re: building a good test suite to predict the rating of engi Jens Bęk Nielsen Thu Aug 16, 2012 3:21 pm
                              Re: building a good test suite to predict the rating of engi Yves Lejeail Thu Aug 16, 2012 5:22 pm
                                    Re: building a good test suite to predict the rating of engi Jens Bęk Nielsen Thu Aug 16, 2012 9:34 pm
                                          Re: building a good test suite to predict the rating of engi Yves Lejeail Tue Aug 21, 2012 2:35 pm
                                                Re: building a good test suite to predict the rating of engi Jens Bęk Nielsen Wed Aug 22, 2012 9:06 pm
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads