How do you organize a self-play?

[jaroslav.tavgen]

I want to improve my engine by making it play itself. How should I organize it? What should be the time limit or searching-depth limit? How can I evaluate if a stronger version of engine won or was it just a pure luck?

Thank you!

[syzygy]

I want to improve my engine by making it play itself. How should I organize it? What should be the time limit or searching-depth limit? How can I evaluate if a stronger version of engine won or was it just a pure luck?

The more games the better. Let two versions play each other at very fast time controls (seconds per full game, not minutes) and play as many games in parallel as your computer has cores (or even threads).

Do not limit search depth. Limit total time for the game.

You can use cutechess-cli to run the games and keep track of the results.

Then learn about statistics to gain some understanding of when a result is significant or not.

[flok]

I want to improve my engine by making it play itself. How should I organize it? What should be the time limit or searching-depth limit? How can I evaluate if a stronger version of engine won or was it just a pure luck?

The more games the better. Let two versions play each other at very fast time controls (seconds per full game, not minutes) and play as many games in parallel as your computer has cores (or even threads).

Do not limit search depth. Limit total time for the game.

You can use cutechess-cli to run the games and keep track of the results.

Then learn about statistics to gain some understanding of when a result is significant or not.

...or use 'sprt' in fastchess and let fastchess tell you what the result is

[syzygy]

...or use 'sprt' in fastchess and let fastchess tell you what the result is

I did not know fastchess. It looks interesting!
https://github.com/Disservin/fastchess

[benvining]

Here's an example of integrating fastchess SPRT into a CMake workflow: https://github.com/benthevining/BenBot/ ... eLists.txt

I create 2 custom targets, sprt_set_baseline and sprt. sprt_set_baseline builds the engine and copies the binary to a eng_baseline file, sprt always tests the latest build against the last baseline. So then the workflow is, when I'm at a stable point, run sprt_set_baseline, then make some changes and run sprt, and that should tell you if your changes have helped or hurt.

[jaroslav.tavgen]

Thank you for your answers!

Here is what I do. I use "VICE for JavaScript" which I've written on my own following these series:

I make this engine play itself 150 games. Two versions play agains each other: one where I tweaked some of the parameters and another is an original version. They play from starting position. First game: tweaked version plays as white, second game: tweaked version plays as black, third game: tweaked version plays as white etc.

Each win brings 1 point to the winning side and a draw gives 0.5 for both.

It takes 480 seconds to play 150 games.

And the variance is huge. For example, I've changed the array "pawnTable" which rewards the engine for putting pieces on certain squares. I changed the reward for putting the pawn on e4 from 20 to 19.

And the first time I made them play 150 the "tweaked" version lost miserably being 21 points behind. But the second time I made them play it won by 11 points!

I don't know what to do with variance. What am I doing wrong (probably everything but that's not the point)?