Page 1 of 1
Experiment: influence of colours at fixed depth.
Posted: Sun Mar 10, 2013 4:43 pm
by Ajedrecista
Hello:
I did a tiny experiment: I used EXchess_v6.50b_win32 by Dan Homan (thanks Dan!
) because it runs fast into depth. The ideal approach will be the following: play lots of games at certain time controls. But I do not have much time and big CPU resources, so fixed depth is fast enough for me. I choosed depths 5, 6, 7, 8 and 9. Here is my summary of results:
Code: Select all
Engine: EXchess_v6.50b_win32 by Dan Homan.
PC: Intel Pentium D930 (3 GHz) of year 2006.
Test: 2500 games without opening books for each fixed depth.
W(d=5) ~ 0.4139; D(d=5) ~ 0.2858; B(d=5) ~ 0.3002; µ(d=5) ~ 0.5568
W(d=6) = 0.3956; D(d=6) = 0.3612; B(d=6) = 0.2432; µ(d=6) = 0.5762
W(d=7) = 0.4388; D(d=7) = 0.3220; B(d=7) = 0.2392; µ(d=7) = 0.5998
W(d=8) = 0.3396; D(d=8) = 0.3968; B(d=8) = 0.2636; µ(d=8) = 0.5380
W(d=9) = 0.3708; D(d=9) = 0.3812; B(d=9) = 0.2480; µ(d=9) = 0.5614
W means 'white wins', D means 'draw' and B means 'black wins'. I did a self test (EXchess versus EXchess) and there are probably tons of duplicated games so my experiment means nothing. By the way, I can not reach any conclusions.
I encourage everybody to do this sort of test (with fixed depths and/or different time controls). Everybody can expect an increasing draw ratio with higher depths and longer time controls but I can say nothing about the score for white.
I have uploaded the PGN files of the experiment, which was made with cutechess-cli 0.5.1; the command line and engines.json file are also included:
Influence_of_colours_at_fixed_depth.rar (5.35 MB)
Here is the command line that I used:
Code: Select all
cutechess-cli -engine conf="Depth_9a" depth=9 -engine conf="Depth_9b" depth=9 -each tc=inf -concurrency 2 -draw 100 50 -resign 5 900 -games 2 -rounds 1250 -ratinginterval 1 -repeat -wait 5 -pgnout Depth_9.pgn
Regards from Spain.
Ajedrecista.
Influence of colours with increasing time controls.
Posted: Sat Jun 08, 2013 2:06 pm
by Ajedrecista
Hello:
There is a new rating list by a tester of SF distributed testing framework:
FastGM's Rating Lists (FGRL)
There is a
comparison with doubling time controls that is interesting and worth to be shared in TalkChess.
Taking a look to white wins, draws and black wins, it looks like the general thoughts and existing data are confirmed once again: longer TC (of course I am not talking about correspondence chess TC) bring a better white performance and an increasing draw ratio. I guess that the raise of the draw ratio with increasing TC is due to the fact that engines are right in their evals more times due to the extra time, so less errors are made and the alledged perfect result of a draw is reached more frequently... but I do not easily understand the fact of the increasing white performance: is the so called initiative of the first move so important with longer TC?
By the way, here are the statistics computed with EloSTAT:
Code: Select all
Time Control: 7.50 Sec. + 0.05 Sec. per Game
Games : 50000 (finished)
White Wins : 16164 (32.3 %)
Black Wins : 11554 (23.1 %)
Draws : 22282 (44.6 %)
White Perf. : 54.6 %
Black Perf. : 45.4 %
Code: Select all
Time Control: 15.00 Sec. + 0.05 Sec. per Game
Games : 50000 (finished)
White Wins : 15723 (31.4 %)
Black Wins : 10623 (21.2 %)
Draws : 23654 (47.3 %)
White Perf. : 55.1 %
Black Perf. : 44.9 %
Code: Select all
Time Control: 30.00 Sec. + 0.05 Sec. per Game
Games : 50000 (finished)
White Wins : 15329 (30.7 %)
Black Wins : 10055 (20.1 %)
Draws : 24616 (49.2 %)
White Perf. : 55.3 %
Black Perf. : 44.7 %
As you can see, the evolutions in the number of white wins, draws and black wins between consecutive time controls (from 7.50 Sec. + 0.05 Sec. to 15.00 Sec. + 0.05 Sec. and from 15.00 Sec. + 0.05 Sec. to 30.00 Sec. + 0.05 Sec.) are:
Code: Select all
White wins: -441 --> -394
Draws: +1372 --> +962
Black wins: -931 --> -568
Anyone knows why white performance raises with increasing time controls? I just ask for a reasonable explanation and not for a theoretical maximum of white performance. Thanks in advance.
Regards from Spain.
Ajedrecista.
Re: Experiment: influence of colours at fixed depth.
Posted: Sat Jun 08, 2013 2:37 pm
by Daniel Shawul
Hi Jesus
You know what would help, plots!
I did something similar tests sometime ago, but not with colors, to see how ELO and draw ratio change with depth. Would you prefer the data or the plots?
Code: Select all
Rank Name Elo + - games score oppo. draws
15 1 scorpio15 409 9 9 4227 76% 141 36%
14 2 scorpio14 338 8 7 6813 69% 149 38%
13 3 scorpio13 268 6 6 9754 63% 146 39%
12 4 scorpio12 184 6 6 11826 60% 87 37%
11 5 scorpio11 91 5 5 14023 58% 16 34%
10 6 scorpio10 -9 6 5 15488 55% -52 32%
9 7 scorpio9 -123 6 5 16814 50% -129 29%
8 8 scorpio8 -245 6 5 16286 45% -195 27%
7 9 scorpio7 -380 6 6 13366 32% -209 25%
6 10 scorpio6 -534 7 7 10271 15% -195 20%
Re: Experiment: influence of colours at fixed depth.
Posted: Sat Jun 08, 2013 2:52 pm
by Laskos
Daniel Shawul wrote:Hi Jesus
You know what would help, plots!
I did something similar tests sometime ago, but not with colors, to see how ELO and draw ratio change with depth. Would you prefer the data or the plots?
Code: Select all
Rank Name Elo + - games score oppo. draws
15 1 scorpio15 409 9 9 4227 76% 141 36%
14 2 scorpio14 338 8 7 6813 69% 149 38%
13 3 scorpio13 268 6 6 9754 63% 146 39%
12 4 scorpio12 184 6 6 11826 60% 87 37%
11 5 scorpio11 91 5 5 14023 58% 16 34%
10 6 scorpio10 -9 6 5 15488 55% -52 32%
9 7 scorpio9 -123 6 5 16814 50% -129 29%
8 8 scorpio8 -245 6 5 16286 45% -195 27%
7 9 scorpio7 -380 6 6 13366 32% -209 25%
6 10 scorpio6 -534 7 7 10271 15% -195 20%
I believe the draw ratio usually is a monotonously increasing function of depth or time.
Re: Experiment: influence of colours at fixed depth.
Posted: Sat Jun 08, 2013 2:57 pm
by Daniel Shawul
I think so too. That bent you see at the end is probably due to not enough games for scorpio 15 and 14, that take too much time to finish their iterations.
Re: Influence of colours with increasing time controls.
Posted: Sat Jun 08, 2013 3:03 pm
by Uri Blass
I have a possible explanation.
When the players are weak they do many mistakes so the color is less important.
When the players get stronger the color becomes more important so white scores better.
At very high level I guess that white score only slightly more than 50% because all games are draws except some games that are book wins for white(and smaller number of games that are book win for black) and I expect the book win for white to be less than 10% so I guess that white is going to score less than 55% if the time control is slow enough.
Re: Influence of colours with increasing time controls.
Posted: Sat Jun 08, 2013 3:24 pm
by lucasart
Uri Blass wrote:I have a possible explanation.
When the players are weak they do many mistakes so the color is less important.
When the players get stronger the color becomes more important so white scores better.
At very high level I guess that white score only slightly more than 50% because all games are draws except some games that are book wins for white(and smaller number of games that are book win for black) and I expect the book win for white to be less than 10% so I guess that white is going to score less than 55% if the time control is slow enough.
I agree, and I can say that it is conformed by my experiments. I recently setup ELO levels in DiscoCheck, and did some measuring to calibrate them (self-play).
The 'advantage' (computed by BayesELO) was very low for the weak levels. It eventually increases to more reasonable values. It is indeed not surprising, given that weak players always win or lose because of a stupid mistake.
I have not been able to clearly measure that 'advantage' becomes a decreasing function of ELO for large values of ELO, but it must be true, since when ELO->infinity we only have draws (assuming no nonsense in the opening book).
PS: here's the script I used to verify my ELO levels
Code: Select all
for elo0 in $(seq 1500 100 2600)
do
elo1=$((${elo0} + 100))
echo "Matching ${elo1} vs. ${elo0}"
./cutechess/projects/cli/cutechess-cli \
-each proto=uci tc=inf option.UCI_LimitStrength=true \
-openings file=./8moves_GM.pgn format=pgn \
-engine cmd=./chess/master name=${elo1} option.UCI_Elo=${elo1} \
-engine cmd=./chess/master name=${elo0} option.UCI_Elo=${elo0} \
-tournament gauntlet -rounds 5000 -concurrency 8 \
-draw movenumber=60 movecount=5 score=50 \
-resign movecount=3 score=700 \
-pgnout ./uci_elo.pgn
done
Re: Experiment: influence of colours at fixed depth.
Posted: Sat Jun 08, 2013 5:12 pm
by Ajedrecista
Hello:
Thank you very much to everybody for your answers.
Daniel Shawul wrote:Hi Jesus
You know what would help, plots!
I did something similar tests sometime ago, but not with colors, to see how ELO and draw ratio change with depth. Would you prefer the data or the plots?
Code: Select all
Rank Name Elo + - games score oppo. draws
15 1 scorpio15 409 9 9 4227 76% 141 36%
14 2 scorpio14 338 8 7 6813 69% 149 38%
13 3 scorpio13 268 6 6 9754 63% 146 39%
12 4 scorpio12 184 6 6 11826 60% 87 37%
11 5 scorpio11 91 5 5 14023 58% 16 34%
10 6 scorpio10 -9 6 5 15488 55% -52 32%
9 7 scorpio9 -123 6 5 16814 50% -129 29%
8 8 scorpio8 -245 6 5 16286 45% -195 27%
7 9 scorpio7 -380 6 6 13366 32% -209 25%
6 10 scorpio6 -534 7 7 10271 15% -195 20%
Diminishing returns again.
CPW is an excellent source in such tests with fixed depth. Please read
this article for more info.
I recall that I did a short study about that using my limited resources (link
here)... fortunately it is included in CPW. But there is other topic here in TalkChess that I have not found in Forum Post section of that CPW article. I hope that Gerd will read this post and realizes about its existance:
Diminishing return
Daniel, you will find there the data that I quote in this post, although in an earlier stage. I find interesting that thread... it should be on CPW.
Regards from Spain.
Ajedrecista.