I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.
http://www.top-5000.nl/tuning.htm
A poor man's testing environment
Moderators: hgm, Rebel, chrisw
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
-
- Posts: 2949
- Joined: Mon May 05, 2008 12:16 pm
- Location: Bordeaux (France)
- Full name: Julien Marcel
Re: A poor man's testing environment
Thanks Ed! Very nice starters. I have started to work on my engine again, and I was planning to use cutechess-cli for the first time. Your guide will prove useful!
"The only good bug is a dead bug." (Don Dailey)
[Blog: http://tinyurl.com/predateur ] [Facebook: http://tinyurl.com/fbpredateur ] [MacEngines: http://tinyurl.com/macengines ]
[Blog: http://tinyurl.com/predateur ] [Facebook: http://tinyurl.com/fbpredateur ] [MacEngines: http://tinyurl.com/macengines ]
-
- Posts: 1471
- Joined: Tue Mar 16, 2010 12:00 am
Re: A poor man's testing environment
- I don't understand the need for making 8 copies of the engine in different folders.Rebel wrote:I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.
http://www.top-5000.nl/tuning.htm
- Why don't you use the "-concurrency" option of cutechess-cli?
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: A poor man's testing environment
Are the 2 questions related?Houdini wrote:- I don't understand the need for making 8 copies of the engine in different folders.Rebel wrote:I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.
http://www.top-5000.nl/tuning.htm
- Why don't you use the "-concurrency" option of cutechess-cli?
Code: Select all
-concurrency <n>::
Set the maximum number of concurrent games to <n>.
The need to make 8 separate copies is related to file usage. I keep statistics and 8 engines writing in the same file is a receipt for trouble, a crash eventually.
-
- Posts: 1971
- Joined: Wed Jul 13, 2011 9:04 pm
- Location: Madrid, Spain.
Re: A poor man's testing environment.
Hello Ed:
I am sure that more people will thank you for this 'tuning guide'.
Regards from Spain.
Ajedrecista.
AFAIK, -concurrency means the number of parallel games played at the same time. I briefly used cutechess-cli 0.5.1 at the end of last summer... I have an old dual core, so I could only choose between -concurrency 1 and -concurrency 2 with single core engines, such as Quazar 0.4 w32 (I choosed -concurrency 2). If you have a quad, then you can go up to -concurrency 4 with single core engines.Rebel wrote:I never understood the meaning of this option.Code: Select all
-concurrency <n>:: Set the maximum number of concurrent games to <n>.
I am sure that more people will thank you for this 'tuning guide'.
Regards from Spain.
Ajedrecista.
-
- Posts: 1471
- Joined: Tue Mar 16, 2010 12:00 am
Re: A poor man's testing environment
Possibly, that depends on you...Rebel wrote:Are the 2 questions related?
The concurrency argument will tell cutechess-cli to play multiple simultaneous games. If you want to use 8 cores, set "-concurrency 8" and cutechess-cli will play 8 simultaneous games.
If I understand correctly, you create 8 engine files in 8 folders and run 8 separate testing processes (clicking 8 times on the batch file) creating 8 PGN output files that need to be combined.
I have 1 engine file and run a single testing process using "-concurrency 8" creating a single PGN output file. KISS!
-
- Posts: 2559
- Joined: Fri Nov 26, 2010 2:00 pm
- Location: Czech Republic
- Full name: Martin Sedlak
Re: A poor man's testing environment
Well you can run one batch/script to handle this, Ed said he needs to log debug output for each engine separately.Houdini wrote:If I understand correctly, you create 8 engine files in 8 folders and run 8 separate testing processes (clicking 8 times on the batch file) creating 8 PGN output files that need to be combined.
So true, nothing can beat cut'n'pasteHoudini wrote: KISS!
-
- Posts: 3232
- Joined: Mon May 31, 2010 1:29 pm
- Full name: lucasart
Re: A poor man's testing environment
Cutechess-cli is perfectly capable of handling concurrent writing into the resulting PGN if that's your point.Rebel wrote:Are the 2 questions related?Houdini wrote:- I don't understand the need for making 8 copies of the engine in different folders.Rebel wrote:I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.
http://www.top-5000.nl/tuning.htm
- Why don't you use the "-concurrency" option of cutechess-cli?
I never understood the meaning of this option.Code: Select all
-concurrency <n>:: Set the maximum number of concurrent games to <n>.
The need to make 8 separate copies is related to file usage. I keep statistics and 8 engines writing in the same file is a receipt for trouble, a crash eventually.
An engine is spawn from a single file (the executable), and cutechess-cli will spawn 8 processes from it (all using their own ring fenced adress space). So you can perfectly have 8 instances running of the same engine, all coming form the same executable file. Just like when you open notepad 8 times by clicking on it 8 times, if you prefer a concrete example.
Using the concurrency option is the most efficient way. It means you only have one instance of the cutechess-cli.
Theory and practice sometimes clash. And when that happens, theory loses. Every single time.
-
- Posts: 6997
- Joined: Thu Aug 18, 2011 12:04 pm
- Full name: Ed Schröder
Re: A poor man's testing environment
I will investigate if -concurrency also works for me, thanks for pointing out.
-
- Posts: 2684
- Joined: Sat Jun 14, 2008 9:17 pm
Re: A poor man's testing environment
Thanks!Rebel wrote:I am interested in some feedback for further improvement.
The interesting part for me is the test from not out-of the book starting positions.
In your start-positions pool you have different kind of late-end game positions. Normally I always test from opening books, eventually truncated at say 10 moves.
But your approach is interesting; one one side you probably enhance the test sensitivity toward the patch you are interested to verify, for instance if it is an evaluation's endgame tweak I guess starting just from late-midgame positions enhances the signal to noise ratio and so would require less games to have reliable result (although current ELO estimators still do not take in account noise level, but this is a different topic).
On the other hand it is easier to introduce artifacts and unwanted bias if you apply it blindly, without understanding what you are doing.
As a general approach I'd still prefer to start out-of-the-book, seems safer to me and, most important, better approximates the "real world" conditions you'll find in public list tests.