testing the random mover agianst small depths

[Uri Blass]

I tested the random mover against the following engines:
1)Caissa1.17(won 81 games and drew 19 games)
2)Seer2.8(won 97 games and drew 3 games)
3)Alexandria6(won 99 games and drew 1 game)

other engines that I tested scored 100% and I will continue the tests.

[Uri Blass]

second test. opponents that did less than 100% in 100 games because of draws

1)Rubi depth 1 (1 draw)
2)Clover depth 1 (2 draws)
3)Caissa depth 2(5 draws)
4)Seer depth 1(6 draws)
5)Caissa depth 1(23 draws)

Alexandria6 depth 1 this time won all the games

[Uri Blass]

third test against depth 3.

Only Caissa depth 3 drew 2 games when the rest lost 100 games.

[Uri Blass]

4th test against all the engines that score points:

1-2)Clover ,RubiChess 499 wins and 1 draw.(repetition by the opponent move)
3)Alexandria 498 wins and 2 draws(repetition by the opponent,stalemate)
4)Seer 485 wins and 15 draws(repetitions)
5)Caissa depth 3 480 wins and 20 draws(16 stalemate,3 repetition 1 fifty move rule)
6)Caissa depth 2 467 wins and 33 draws
7)Caissa depth 1 425 wins and 75 draws(more than 50 of them by repetition when stalemate is the second common option).

[Uri Blass]

5th test

obsidian depth 1:one stale mate out of 500 games
Stockfish16.1:2 stalemate out of 500 games.

Berserk and Igel won all 500 games at depth 1.

[Jouni]

Uri sorry, but I think all fixed depth/fixed nodes tests are 100% useless.

[Uri Blass]

Uri sorry, but I think all fixed depth/fixed nodes tests are 100% useless.

I think that they may be useful to get some minimal rating.
The main problem is that programs allow repetition or stalemate at small depths.

They do not do it with fixed number of nodes that is high enough and I think that I will use only fixed nodes for my rating list.

I would like to know what is the difference in rating between the best engine and the worst engines that can win the random player in more than 99.9% of the games.

[Witek]

Caissa has really bad "depth 1" performance because it performs pruning in root node. Other engines seem not to do that. I will change it in next release, which will be like +200 Elo in that testing conditions. It will improve my selfplay games quality too. But it has no real impact in regular games

[Carlos777]

Uri sorry, but I think all fixed depth/fixed nodes tests are 100% useless.

I think that they may be useful to get some minimal rating.
The main problem is that programs allow repetition or stalemate at small depths.

They do not do it with fixed number of nodes that is high enough and I think that I will use only fixed nodes for my rating list.

I would like to know what is the difference in rating between the best engine and the worst engines that can win the random player in more than 99.9% of the games.

Hi Uri,

Could you share your "small depth" rating list and if you have merged it with the "low nodes count", could you share that too?

Thanks in advance,
Carlos

[Uri Blass]

I did not conitnue in these games for rating list lately but
here is what I found based on league tournament everybody play everybody white and black.

I think that I will probably use later minimal hash that I can after finding that with this number of nodes engines probably do not need hash so as a rule I will not use hash that is bigger than the number of nodes unless I have to do it because the engine does not allow 1 mbyte hash(minimal option for berserk is 2 mbyte hash)


# PLAYER : RATING POINTS PLAYED (%)
1 Stockfish 16.1 4096nodes : 1176.0 113.5 118 96
2 Stockfish 16.1 2048nodes : 928.8 103.0 118 87
3 Caissa 1.17 AVX2 4096nodes : 912.9 102.0 118 86
4 Berserk 12 4096 nodes : 869.0 99.0 118 84
5 Alexandria-6.0 4096 nodes : 823.0 95.5 118 81
6 Seer 2.8.0 4096 nodes : 816.8 95.0 118 81
7 Viridithas 12.0.0 4096 nodes : 816.8 95.0 118 81
8 Black Marlin 9.0 4096 nodes : 810.6 94.5 118 80
9 Koivisto 9.0 4096 nodes : 792.6 93.0 118 79
10 PlentyChess 4096 nodes : 781.0 92.0 118 78
11 Obsidian 11.0 4096nodes : 781.0 92.0 118 78
12 Altair 7.0.0 4096 nodes : 747.3 89.0 118 75
13 Igel 3.5.0 4096 nodes : 720.5 86.5 118 73
14 Velvet v7.1.0 4096 nodes : 715.3 86.0 118 73
15 Viridithas 12.0.0 2048 nodes : 710.1 85.5 118 72
16 Caissa 1.17 AVX2 2048nodes : 679.7 82.5 118 70
17 RubiChess 20240112 4096nodes : 674.7 82.0 118 69
18 Clover 6.1 4096 nodes : 650.3 79.5 118 67
19 Alexandria-6.0 2048 nodes : 640.7 78.5 118 67
20 Stockfish 16.1 1024nodes : 603.0 74.5 118 63
21 Velvet v7.1.0 2048 nodes : 575.5 71.5 118 61
22 Seer 2.8.0 2048 nodes : 552.9 69.0 118 58
23 PlentyChess 2048 nodes : 543.9 68.0 118 58
24 Altair 7.0.0 2048 nodes : 521.7 65.5 118 56
25 Viridithas 12.0.0 1024 nodes : 521.7 65.5 118 56
26 Berserk 12 2048 nodes : 499.5 63.0 118 53
27 Velvet v7.1.0 512 nodes : 499.5 63.0 118 53
28 Velvet v7.1.0 1024 nodes : 486.3 61.5 118 52
29 Obsidian 11.0 2048nodes : 486.3 61.5 118 52
30 Caissa 1.17 AVX2 1024nodes : 473.1 60.0 118 51
31 Koivisto 9.0 2048 nodes : 473.1 60.0 118 51
32 Altair 7.0.0 1024 nodes : 451.2 57.5 118 49
33 Black Marlin 9.0 2048 nodes : 446.8 57.0 118 48
34 PlentyChess 1024 nodes : 420.3 54.0 118 46
35 Alexandria-6.0 1024 nodes : 415.9 53.5 118 45
36 Rubi20240112 2048nodes : 411.5 53.0 118 45
37 Altair 7.0.0 512 nodes : 380.4 49.5 118 42
38 Igel 3.5.0 2048 nodes : 380.4 49.5 118 42
39 Viridithas 12.0.0 512 nodes : 367.0 48.0 118 41
40 Stockfish 16.1 512 nodes : 330.7 44.0 118 37
41 Clover 6.1 2048 nodes : 330.7 44.0 118 37
42 Alexandria-6.0 512 nodes : 326.1 43.5 118 37
43 Seer 2.8.0 1024 nodes : 274.5 38.0 118 32
44 Obsidian 11.0 1024nodes : 235.4 34.0 118 29
45 PlentyChess 512 nodes : 230.4 33.5 118 28
46 Caissa 1.17 512 nodes : 220.3 32.5 118 28
47 RubiChess 20240112 1024nodes : 220.3 32.5 118 28
48 Koivisto 9.0 1024 nodes : 215.2 32.0 118 27
49 Berserk 12 1024 nodes : 194.4 30.0 118 25
50 Koivisto 9.0 512 nodes : 162.0 27.0 118 23
51 Black Marlin 9.0 1024 nodes : 139.4 25.0 118 21
52 Clover 6.1 1024 nodes : 139.4 25.0 118 21
53 Rubi20240112 512nodes : 121.9 23.5 118 20
54 Igel 3.5.0 1024 nodes : 97.5 21.5 118 18
55 Seer 2.8.0 512 nodes : 71.9 19.5 118 17
56 Black Marlin 9.0 512 nodes : 71.9 19.5 118 17
57 Igel 3.5.0 512 nodes : 58.5 18.5 118 16
58 Berserk 12 512 nodes : 51.6 18.0 118 15
59 Obsidian 11.0 512 nodes : 7.8 15.0 118 13
60 Clover 6.1 512 nodes : 0.0 14.5 118 12

White advantage = 34.02
Draw rate (equal opponents) = 50.00 %



# PLAYER : RATING POINTS PLAYED (%)
1 Stockfish 16.1 2048nodes : 1218.1 186.0 198 94
2 Caissa 1.17 depth 7 : 1066.2 174.0 198 88
3 Alexandria-6.0 depth7 : 1051.3 172.5 198 87
4 Wasp 6.50 depth 7 : 1046.5 172.0 198 87
5 Alexandria-6.0 2048 nodes : 1032.3 170.5 198 86
6 Stockfish 16.1 1024nodes : 1027.7 170.0 198 86
7 Wasp 6.50 depth 4 : 1027.7 170.0 198 86
8 Caissa 1.17 AVX2 2048nodes : 1023.2 169.5 198 86
9 Seer 2.8.0 depth 7 : 1005.5 167.5 198 85
10 Wasp 6.50 depth 6 : 992.7 166.0 198 84
11 Wasp 6.50 depth 5 : 968.1 163.0 198 82
12 Alexandria-6.0 depth6 : 964.1 162.5 198 82
13 Obsidian 11.0 depth 7 : 944.7 160.0 198 81
14 Wasp 6.50 depth 3 : 933.3 158.5 198 80
15 Stockfish 16.1 depth 7 : 925.9 157.5 198 80
16 Caissa 1.17 depth 6 : 904.3 154.5 198 78
17 Igel 3.5.0 depth 7 : 900.8 154.0 198 78
18 Seer 2.8.0 2048 nodes : 883.4 151.5 198 77
19 Caissa 1.17 AVX2 1024nodes : 880.0 151.0 198 76
20 Alexandria-6.0 depth5 : 869.9 149.5 198 76
21 Wasp 6.50 2048 nodes : 866.5 149.0 198 75
22 Berserk 12 2048 nodes : 856.6 147.5 198 74
23 Wasp 6.50 depth 2 : 837.2 144.5 198 73
24 Obsidian 11.0 2048nodes : 837.2 144.5 198 73
25 Alexandria-6.0 1024 nodes : 793.7 137.5 198 69
26 RubiChess 20240112 2048nodes : 790.7 137.0 198 69
27 Caissa 1.17 depth 5 : 784.7 136.0 198 69
28 Igel 3.5.0 2048 nodes : 761.1 132.0 198 67
29 Clover 6.1 depth 7 : 761.1 132.0 198 67
30 Stockfish 16.1 512 nodes : 752.4 130.5 198 66
31 Berserk 12 depth7 : 740.9 128.5 198 65
32 Wasp 6.50 1024 nodes : 732.4 127.0 198 64
33 Alexandria-6.0 depth4 : 729.5 126.5 198 64
34 Stockfish 16.1 depth 6 : 721.1 125.0 198 63
35 Wasp 6.50 512 nodes : 712.7 123.5 198 62
36 Alexandria-6.0 512 nodes : 704.4 122.0 198 62
37 Rubi20240112 depth7 : 685.2 118.5 198 60
38 Seer 2.8.0 1024 nodes : 685.2 118.5 198 60
39 Clover 6.1 2048 nodes : 679.8 117.5 198 59
40 Igel 3.5.0 depth 6 : 645.0 111.0 198 56
41 Alexandria-6.0 depth 3 : 618.7 106.0 198 54
42 Obsidian 11.0 depth 6 : 613.5 105.0 198 53
43 Berserk 12 1024 nodes : 603.1 103.0 198 52
44 Caissa 1.17 512 nodes : 600.5 102.5 198 52
45 Rubi20240112 depth6 : 600.5 102.5 198 52
46 RubiChess 20240112 1024nodes : 595.3 101.5 198 51
47 Wasp 6.50 depth 1 : 592.7 101.0 198 51
48 Caissa 1.17 depth 4 : 590.1 100.5 198 51
49 Obsidian 11.0 1024nodes : 585.0 99.5 198 50
50 Seer 2.8.0 depth 6 : 579.8 98.5 198 50
51 Stockfish 16.1 depth 5 : 543.9 91.5 198 46
52 Igel 3.5.0 1024 nodes : 543.9 91.5 198 46
53 Berserk 12 depth6 : 508.2 84.5 198 43
54 Seer 2.8.0 depth 5 : 500.5 83.0 198 42
55 Rubi20240112 512nodes : 498.0 82.5 198 42
56 Alexandria-6.0 depth 2 : 495.4 82.0 198 41
57 Seer 2.8.0 512 nodes : 487.7 80.5 198 41
58 Obsidian 11.0 depth 5 : 472.3 77.5 198 39
59 Obsidian 11.0 depth 1 : 469.7 77.0 198 39
60 Rubi20240112 depth5 : 456.8 74.5 198 38
61 Clover 6.1 1024 nodes : 441.2 71.5 198 36
62 Clover 6.1 depth 6 : 438.5 71.0 198 36
63 Rubi20240112 depth2 : 438.5 71.0 198 36
64 Igel 3.5.0 depth 5 : 435.9 70.5 198 36
65 Obsidian 11.0 512 nodes : 433.3 70.0 198 35
66 Igel 3.5.0 depth 1 : 425.4 68.5 198 35
67 Rubi20240112 depth4 : 412.1 66.0 198 33
68 Obsidian 11.0 depth 4 : 412.1 66.0 198 33
69 Stockfish 16.1 depth 1 : 412.1 66.0 198 33
70 Rubi20240112 depth3 : 406.8 65.0 198 33
71 Stockfish 16.1 depth 4 : 398.7 63.5 198 32
72 Igel 3.5.0 512 nodes : 385.1 61.0 198 31
73 Berserk 12 512 nodes : 385.1 61.0 198 31
74 Igel 3.5.0 depth 3 : 385.1 61.0 198 31
75 Seer 2.8.0 depth 1 : 382.4 60.5 198 31
76 Igel 3.5.0 depth 4 : 382.4 60.5 198 31
77 Seer 2.8.0 depth 4 : 374.1 59.0 198 30
78 Clover 6.1 depth 1 : 371.3 58.5 198 30
79 Berserk 12 depth5 : 368.6 58.0 198 29
80 Igel 3.5.0 depth 2 : 365.8 57.5 198 29
81 Rubi20240112 depth1 : 363.0 57.0 198 29
82 Alexandria-6.0 depth1 : 363.0 57.0 198 29
83 Clover 6.1 512 nodes : 357.3 56.0 198 28
84 Stockfish 16.1 depth 3 : 351.7 55.0 198 28
85 Obsidian 11.0 depth 2 : 351.7 55.0 198 28
86 Clover 6.1 depth 2 : 351.7 55.0 198 28
87 Clover 6.1 depth 5 : 351.7 55.0 198 28
88 Caissa 1.17 depth 3 : 337.3 52.5 198 27
89 Clover 6.1 depth 3 : 325.6 50.5 198 26
90 Berserk 12 depth2 : 313.6 48.5 198 24
91 Clover 6.1 depth 4 : 298.4 46.0 198 23
92 Berserk 12 depth4 : 273.0 42.0 198 21
93 Berserk 12 depth1 : 266.4 41.0 198 21
94 Obsidian 11.0 depth 3 : 249.7 38.5 198 19
95 Seer 2.8.0 depth 2 : 239.3 37.0 198 19
96 Berserk 12 depth3 : 217.7 34.0 198 17
97 Seer 2.8.0 depth 3 : 210.3 33.0 198 17
98 Stockfish 16.1 depth 2 : 195.0 31.0 198 16
99 Caissa 1.17 depth 2 : 139.9 24.5 198 12
100 Caissa 1.17 depth 1 : 0.0 12.5 198 6

White advantage = 22.29
Draw rate (equal opponents) = 50.00 %