A poor man's testing environment

[Rebel]

I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.

http://www.top-5000.nl/tuning.htm

[JuLieN]

Thanks Ed! Very nice starters. I have started to work on my engine again, and I was planning to use cutechess-cli for the first time. Your guide will prove useful!

[Houdini]

I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.

http://www.top-5000.nl/tuning.htm

- I don't understand the need for making 8 copies of the engine in different folders.
- Why don't you use the "-concurrency" option of cutechess-cli?

[Rebel]

I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.

http://www.top-5000.nl/tuning.htm

- I don't understand the need for making 8 copies of the engine in different folders.
- Why don't you use the "-concurrency" option of cutechess-cli?

Are the 2 questions related?

Code: Select all

-concurrency <n>&#58;&#58;
	Set the maximum number of concurrent games to <n>.

I never understood the meaning of this option.

The need to make 8 separate copies is related to file usage. I keep statistics and 8 engines writing in the same file is a receipt for trouble, a crash eventually.

[Ajedrecista]

Hello Ed:

Code: Select all

-concurrency <n>&#58;&#58;
	Set the maximum number of concurrent games to <n>.

I never understood the meaning of this option.

AFAIK, -concurrency means the number of parallel games played at the same time. I briefly used cutechess-cli 0.5.1 at the end of last summer... I have an old dual core, so I could only choose between -concurrency 1 and -concurrency 2 with single core engines, such as Quazar 0.4 w32 (I choosed -concurrency 2). If you have a quad, then you can go up to -concurrency 4 with single core engines.

I am sure that more people will thank you for this 'tuning guide'.

Regards from Spain.

Ajedrecista.

[Houdini]

Are the 2 questions related?

Possibly, that depends on you...

The concurrency argument will tell cutechess-cli to play multiple simultaneous games. If you want to use 8 cores, set "-concurrency 8" and cutechess-cli will play 8 simultaneous games.

If I understand correctly, you create 8 engine files in 8 folders and run 8 separate testing processes (clicking 8 times on the batch file) creating 8 PGN output files that need to be combined.

I have 1 engine file and run a single testing process using "-concurrency 8" creating a single PGN output file. KISS!

[mar]

If I understand correctly, you create 8 engine files in 8 folders and run 8 separate testing processes (clicking 8 times on the batch file) creating 8 PGN output files that need to be combined.

Well you can run one batch/script to handle this, Ed said he needs to log debug output for each engine separately.

KISS!

So true, nothing can beat cut'n'paste

[lucasart]

I like to present a page for starters how to test a chess engine with limited hardware. I am interested in some feedback for further improvement.

http://www.top-5000.nl/tuning.htm

- I don't understand the need for making 8 copies of the engine in different folders.
- Why don't you use the "-concurrency" option of cutechess-cli?

Are the 2 questions related?

Code: Select all

-concurrency <n>&#58;&#58;
	Set the maximum number of concurrent games to <n>.

I never understood the meaning of this option.

The need to make 8 separate copies is related to file usage. I keep statistics and 8 engines writing in the same file is a receipt for trouble, a crash eventually.

Cutechess-cli is perfectly capable of handling concurrent writing into the resulting PGN if that's your point.

An engine is spawn from a single file (the executable), and cutechess-cli will spawn 8 processes from it (all using their own ring fenced adress space). So you can perfectly have 8 instances running of the same engine, all coming form the same executable file. Just like when you open notepad 8 times by clicking on it 8 times, if you prefer a concrete example.

Using the concurrency option is the most efficient way. It means you only have one instance of the cutechess-cli.

[Rebel]

I will investigate if -concurrency also works for me, thanks for pointing out.

[mcostalba]

I am interested in some feedback for further improvement.

Thanks!

The interesting part for me is the test from not out-of the book starting positions.

In your start-positions pool you have different kind of late-end game positions. Normally I always test from opening books, eventually truncated at say 10 moves.

But your approach is interesting; one one side you probably enhance the test sensitivity toward the patch you are interested to verify, for instance if it is an evaluation's endgame tweak I guess starting just from late-midgame positions enhances the signal to noise ratio and so would require less games to have reliable result (although current ELO estimators still do not take in account noise level, but this is a different topic).

On the other hand it is easier to introduce artifacts and unwanted bias if you apply it blindly, without understanding what you are doing.

As a general approach I'd still prefer to start out-of-the-book, seems safer to me and, most important, better approximates the "real world" conditions you'll find in public list tests.