Experiment: influence of colours at fixed depth.

[Ajedrecista]

Hello:

I did a tiny experiment: I used EXchess_v6.50b_win32 by Dan Homan (thanks Dan! ) because it runs fast into depth. The ideal approach will be the following: play lots of games at certain time controls. But I do not have much time and big CPU resources, so fixed depth is fast enough for me. I choosed depths 5, 6, 7, 8 and 9. Here is my summary of results:

Code: Select all

Engine: EXchess_v6.50b_win32 by Dan Homan.
PC: Intel Pentium D930 (3 GHz) of year 2006.
Test: 2500 games without opening books for each fixed depth.

W(d=5) ~ 0.4139; D(d=5) ~ 0.2858; B(d=5) ~ 0.3002; µ(d=5) ~ 0.5568
W(d=6) = 0.3956; D(d=6) = 0.3612; B(d=6) = 0.2432; µ(d=6) = 0.5762
W(d=7) = 0.4388; D(d=7) = 0.3220; B(d=7) = 0.2392; µ(d=7) = 0.5998
W(d=8) = 0.3396; D(d=8) = 0.3968; B(d=8) = 0.2636; µ(d=8) = 0.5380
W(d=9) = 0.3708; D(d=9) = 0.3812; B(d=9) = 0.2480; µ(d=9) = 0.5614

W means 'white wins', D means 'draw' and B means 'black wins'. I did a self test (EXchess versus EXchess) and there are probably tons of duplicated games so my experiment means nothing. By the way, I can not reach any conclusions.

I encourage everybody to do this sort of test (with fixed depths and/or different time controls). Everybody can expect an increasing draw ratio with higher depths and longer time controls but I can say nothing about the score for white.

I have uploaded the PGN files of the experiment, which was made with cutechess-cli 0.5.1; the command line and engines.json file are also included:

Influence_of_colours_at_fixed_depth.rar (5.35 MB)

Here is the command line that I used:

Code: Select all

cutechess-cli -engine conf="Depth_9a" depth=9 -engine conf="Depth_9b" depth=9 -each tc=inf -concurrency 2 -draw 100 50 -resign 5 900 -games 2 -rounds 1250 -ratinginterval 1 -repeat -wait 5 -pgnout Depth_9.pgn

Regards from Spain.

Ajedrecista.

[Ajedrecista]

Hello:

There is a new rating list by a tester of SF distributed testing framework:

FastGM's Rating Lists (FGRL)

There is a comparison with doubling time controls that is interesting and worth to be shared in TalkChess.

Taking a look to white wins, draws and black wins, it looks like the general thoughts and existing data are confirmed once again: longer TC (of course I am not talking about correspondence chess TC) bring a better white performance and an increasing draw ratio. I guess that the raise of the draw ratio with increasing TC is due to the fact that engines are right in their evals more times due to the extra time, so less errors are made and the alledged perfect result of a draw is reached more frequently... but I do not easily understand the fact of the increasing white performance: is the so called initiative of the first move so important with longer TC?

By the way, here are the statistics computed with EloSTAT:

Code: Select all

Time Control: 7.50 Sec. + 0.05 Sec. per Game

Games        :  50000 (finished)

White Wins   :  16164 (32.3 %)
Black Wins   :  11554 (23.1 %)
Draws        :  22282 (44.6 %)

White Perf.  : 54.6 %
Black Perf.  : 45.4 %

Code: Select all

Time Control: 15.00 Sec. + 0.05 Sec. per Game

Games        :  50000 (finished)

White Wins   :  15723 (31.4 %)
Black Wins   :  10623 (21.2 %)
Draws        :  23654 (47.3 %)

White Perf.  : 55.1 %
Black Perf.  : 44.9 %

Code: Select all

Time Control: 30.00 Sec. + 0.05 Sec. per Game

Games        :  50000 (finished)

White Wins   :  15329 (30.7 %)
Black Wins   :  10055 (20.1 %)
Draws        :  24616 (49.2 %)

White Perf.  : 55.3 %
Black Perf.  : 44.7 %

As you can see, the evolutions in the number of white wins, draws and black wins between consecutive time controls (from 7.50 Sec. + 0.05 Sec. to 15.00 Sec. + 0.05 Sec. and from 15.00 Sec. + 0.05 Sec. to 30.00 Sec. + 0.05 Sec.) are:

Code: Select all

White wins:  -441 --> -394
Draws:      +1372 --> +962
Black wins:  -931 --> -568

Anyone knows why white performance raises with increasing time controls? I just ask for a reasonable explanation and not for a theoretical maximum of white performance. Thanks in advance.

Regards from Spain.

Ajedrecista.

[Daniel Shawul]

Hi Jesus
You know what would help, plots! I did something similar tests sometime ago, but not with colors, to see how ELO and draw ratio change with depth. Would you prefer the data or the plots?

Code: Select all

Rank	Name	Elo	+	-		games	score	oppo.	draws
15	1	scorpio15	409	9	9	4227	76%	141	36%
14	2	scorpio14	338	8	7	6813	69%	149	38%
13	3	scorpio13	268	6	6	9754	63%	146	39%
12	4	scorpio12	184	6	6	11826	60%	87	37%
11	5	scorpio11	91	5	5	14023	58%	16	34%
10	6	scorpio10	-9	6	5	15488	55%	-52	32%
9	7	scorpio9	-123	6	5	16814	50%	-129	29%
8	8	scorpio8	-245	6	5	16286	45%	-195	27%
7	9	scorpio7	-380	6	6	13366	32%	-209	25%
6	10	scorpio6	-534	7	7	10271	15%	-195	20%

[Laskos]

Hi Jesus
You know what would help, plots! I did something similar tests sometime ago, but not with colors, to see how ELO and draw ratio change with depth. Would you prefer the data or the plots?

Code: Select all

Rank	Name	Elo	+	-		games	score	oppo.	draws
15	1	scorpio15	409	9	9	4227	76%	141	36%
14	2	scorpio14	338	8	7	6813	69%	149	38%
13	3	scorpio13	268	6	6	9754	63%	146	39%
12	4	scorpio12	184	6	6	11826	60%	87	37%
11	5	scorpio11	91	5	5	14023	58%	16	34%
10	6	scorpio10	-9	6	5	15488	55%	-52	32%
9	7	scorpio9	-123	6	5	16814	50%	-129	29%
8	8	scorpio8	-245	6	5	16286	45%	-195	27%
7	9	scorpio7	-380	6	6	13366	32%	-209	25%
6	10	scorpio6	-534	7	7	10271	15%	-195	20%

I believe the draw ratio usually is a monotonously increasing function of depth or time.

[Daniel Shawul]

I think so too. That bent you see at the end is probably due to not enough games for scorpio 15 and 14, that take too much time to finish their iterations.

[Uri Blass]

I have a possible explanation.

When the players are weak they do many mistakes so the color is less important.

When the players get stronger the color becomes more important so white scores better.

At very high level I guess that white score only slightly more than 50% because all games are draws except some games that are book wins for white(and smaller number of games that are book win for black) and I expect the book win for white to be less than 10% so I guess that white is going to score less than 55% if the time control is slow enough.

[lucasart]

I have a possible explanation.

When the players are weak they do many mistakes so the color is less important.

When the players get stronger the color becomes more important so white scores better.

At very high level I guess that white score only slightly more than 50% because all games are draws except some games that are book wins for white(and smaller number of games that are book win for black) and I expect the book win for white to be less than 10% so I guess that white is going to score less than 55% if the time control is slow enough.

I agree, and I can say that it is conformed by my experiments. I recently setup ELO levels in DiscoCheck, and did some measuring to calibrate them (self-play).

The 'advantage' (computed by BayesELO) was very low for the weak levels. It eventually increases to more reasonable values. It is indeed not surprising, given that weak players always win or lose because of a stupid mistake.

I have not been able to clearly measure that 'advantage' becomes a decreasing function of ELO for large values of ELO, but it must be true, since when ELO->infinity we only have draws (assuming no nonsense in the opening book).

PS: here's the script I used to verify my ELO levels

Code: Select all

for elo0 in $(seq 1500 100 2600)
do
	elo1=$((${elo0} + 100))
	echo "Matching ${elo1} vs. ${elo0}"

	./cutechess/projects/cli/cutechess-cli \
		-each proto=uci tc=inf option.UCI_LimitStrength=true \
		-openings file=./8moves_GM.pgn format=pgn \
		-engine cmd=./chess/master name=${elo1} option.UCI_Elo=${elo1} \
		-engine cmd=./chess/master name=${elo0} option.UCI_Elo=${elo0} \
		-tournament gauntlet -rounds 5000 -concurrency 8 \
		-draw movenumber=60 movecount=5 score=50 \
		-resign movecount=3 score=700 \
		-pgnout ./uci_elo.pgn
done

[Ajedrecista]

Hello:

Thank you very much to everybody for your answers.

Hi Jesus
You know what would help, plots! I did something similar tests sometime ago, but not with colors, to see how ELO and draw ratio change with depth. Would you prefer the data or the plots?

Code: Select all

Rank	Name	Elo	+	-		games	score	oppo.	draws
15	1	scorpio15	409	9	9	4227	76%	141	36%
14	2	scorpio14	338	8	7	6813	69%	149	38%
13	3	scorpio13	268	6	6	9754	63%	146	39%
12	4	scorpio12	184	6	6	11826	60%	87	37%
11	5	scorpio11	91	5	5	14023	58%	16	34%
10	6	scorpio10	-9	6	5	15488	55%	-52	32%
9	7	scorpio9	-123	6	5	16814	50%	-129	29%
8	8	scorpio8	-245	6	5	16286	45%	-195	27%
7	9	scorpio7	-380	6	6	13366	32%	-209	25%
6	10	scorpio6	-534	7	7	10271	15%	-195	20%

Diminishing returns again. CPW is an excellent source in such tests with fixed depth. Please read this article for more info.

I recall that I did a short study about that using my limited resources (link here)... fortunately it is included in CPW. But there is other topic here in TalkChess that I have not found in Forum Post section of that CPW article. I hope that Gerd will read this post and realizes about its existance:

Diminishing return

Daniel, you will find there the data that I quote in this post, although in an earlier stage. I find interesting that thread... it should be on CPW.

Regards from Spain.

Ajedrecista.