How do you organize a self-play?

[jaroslav.tavgen]

I want to improve my engine by making it play itself. How should I organize it? What should be the time limit or searching-depth limit? How can I evaluate if a stronger version of engine won or was it just a pure luck?

Thank you!

[syzygy]

I want to improve my engine by making it play itself. How should I organize it? What should be the time limit or searching-depth limit? How can I evaluate if a stronger version of engine won or was it just a pure luck?

The more games the better. Let two versions play each other at very fast time controls (seconds per full game, not minutes) and play as many games in parallel as your computer has cores (or even threads).

Do not limit search depth. Limit total time for the game.

You can use cutechess-cli to run the games and keep track of the results.

Then learn about statistics to gain some understanding of when a result is significant or not.

[flok]

I want to improve my engine by making it play itself. How should I organize it? What should be the time limit or searching-depth limit? How can I evaluate if a stronger version of engine won or was it just a pure luck?

The more games the better. Let two versions play each other at very fast time controls (seconds per full game, not minutes) and play as many games in parallel as your computer has cores (or even threads).

Do not limit search depth. Limit total time for the game.

You can use cutechess-cli to run the games and keep track of the results.

Then learn about statistics to gain some understanding of when a result is significant or not.

...or use 'sprt' in fastchess and let fastchess tell you what the result is

[syzygy]

...or use 'sprt' in fastchess and let fastchess tell you what the result is

I did not know fastchess. It looks interesting!
https://github.com/Disservin/fastchess

[benvining]

Here's an example of integrating fastchess SPRT into a CMake workflow: https://github.com/benthevining/BenBot/ ... eLists.txt

I create 2 custom targets, sprt_set_baseline and sprt. sprt_set_baseline builds the engine and copies the binary to a eng_baseline file, sprt always tests the latest build against the last baseline. So then the workflow is, when I'm at a stable point, run sprt_set_baseline, then make some changes and run sprt, and that should tell you if your changes have helped or hurt.

[jaroslav.tavgen]

Thank you for your answers!

Here is what I do. I use "VICE for JavaScript" which I've written on my own following these series:

I make this engine play itself 150 games. Two versions play agains each other: one where I tweaked some of the parameters and another is an original version. They play from starting position. First game: tweaked version plays as white, second game: tweaked version plays as black, third game: tweaked version plays as white etc.

Each win brings 1 point to the winning side and a draw gives 0.5 for both.

It takes 480 seconds to play 150 games.

And the variance is huge. For example, I've changed the array "pawnTable" which rewards the engine for putting pieces on certain squares. I changed the reward for putting the pawn on e4 from 20 to 19.

And the first time I made them play 150 the "tweaked" version lost miserably being 21 points behind. But the second time I made them play it won by 11 points!

I don't know what to do with variance. What am I doing wrong (probably everything but that's not the point)?

[hgm]

The standard deviation in the score of an N-game match of independent games should be about 40%/sqrt(N). Which for 150 games means about 3%, or 4.5 points. So a deviation of 21 or 11 should be very unlikely.

The explanation is probably that the games are not independent. Bij starting all games from the same position, many games could be identical until after the point where a decisive error is made. To avoid this you could start all games from a different position.

[Ajedrecista]

Hello Müller:

The standard deviation in the score of an N-game match of independent games should be about 40%/sqrt(N). Which for 150 games means about 3%, or 4.5 points. So a deviation of 21 or 11 should be very unlikely.

The explanation is probably that the games are not independent. Bij starting all games from the same position, many games could be identical until after the point where a decisive error is made. To avoid this you could start all games from a different position.

Those deviations are not that unlikely. Assuming your standard deviation of 40%/sqrt(N), which happens with a draw ratio of 36% in a trinomial distribution in an even match, which is a fair estimate for not super strong engines, a standard deviation for 150 games is 40%*sqrt(150) ~ 4.9 points.

Then, one engine scores 50% + z*sigma and the other, 50% - z*sigma, so the difference is 2z*sigma. A 2-sigma deviation for 150 games is 75 + 2*4.9 = 84.8 for one engine and 75 - 2*4.9 = 65.2 for the other. The difference is 84.8 - 65.2 = 19.6 points, so 11 points are well within those 19 or 20 points, and 21 not being very far.

Regards from Spain.

Ajedrecista.

[jaroslav.tavgen]

The standard deviation in the score of an N-game match of independent games should be about 40%/sqrt(N). Which for 150 games means about 3%, or 4.5 points. So a deviation of 21 or 11 should be very unlikely.

The explanation is probably that the games are not independent. Bij starting all games from the same position, many games could be identical until after the point where a decisive error is made. To avoid this you could start all games from a different position.

Thank you for clarifying!

Now I do the following: I play 4 games from different positions (starting position, after 1. e4 e5, after 1. d4 d5 and after 1. c4 e5) also alternating between "tweaked" version being white or black.

40%/sqrt(N) = 0.4 / 2 = 0.2. 4 x 0.2 = 0.8 or 1 point difference.

So I assume that if the "tweaked" version has +3 points after playing those 4 games then I assume that it is stronger and now this "tweaked" version is the new "original" version.

Is this reasonable?

[abulmo2]

The standard deviation in the score of an N-game match of independent games should be about 40%/sqrt(N). Which for 150 games means about 3%, or 4.5 points. So a deviation of 21 or 11 should be very unlikely.

The explanation is probably that the games are not independent. Bij starting all games from the same position, many games could be identical until after the point where a decisive error is made. To avoid this you could start all games from a different position.

Thank you for clarifying!

Now I do the following: I play 4 games from different positions (starting position, after 1. e4 e5, after 1. d4 d5 and after 1. c4 e5) also alternating between "tweaked" version being white or black.

40%/sqrt(N) = 0.4 / 2 = 0.2. 4 x 0.2 = 0.8 or 1 point difference.

So I assume that if the "tweaked" version has +3 points after playing those 4 games then I assume that it is stronger and now this "tweaked" version is the new "original" vers

Is this reasonable?

No, for small number you have to compute directly the trinomial probabilities. The formula 40%/sqrt(N) holds for big numbers (N>30).
Assuming for 2 engines of equal strength a probabilty of winning Pw = 30%, of drawing Pd = 40% and of losing Pl = 30%; with 4 games, you can compute the following probabilities of having x points:
P(4) = 0.0081
P(3.5)=0.0432
P(3)=0.1188
P(2.5)=0.2064
P(2)=0.247
...(the probabilities are symmetrical).
Thus, the probability of having P(x>=3)=17,01% is too important to conclude anything else that the engines are of equal strength. Only a result of +4 can lead to a conclusion of a superior engine.