Strategic Test Suite 1...15 in one file

[Mike S.]

Swaminathan has collected 1,500 strategical (or positional as I assume) test positions. For your convenience in one EPD file:

http://computerschach.twoday.net/files/STS1-15 (61 KB zip)

Source: https://sites.google.com/site/strategictestsuite/

Results welcome A test run with 10 seconds/pos. only will take about four hours.

[Nordlandia]

Swaminathan has collected 1,500 strategical (or positional as I assume) test positions. For your convenience in one EPD file:

http://computerschach.twoday.net/files/STS1-15 (61 KB zip)

Source: https://sites.google.com/site/strategictestsuite/

Results welcome A test run with 10 seconds/pos. only will take about four hours.

Is it possible to run the test positions in Chessbase GUI.

It need to be (.CBH) file.

(Edit) i figured it out

http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=21126

[Nordlandia]

i7-5960X 3.9GHz | 1024 MB | Latest Stockfish Dev

Code: Select all

Maximum time = 10s.

1. Undermine.001,  STS_1-15   Solved in 0.25s/12; Solved: 1
2. Undermine.002,  STS_1-15   Solved in 3.78s/23; Solved: 2
3. Undermine.003,  STS_1-15   Solved in 0.03s/7; Solved: 3
4. Undermine.004,  STS_1-15   Solved in 0.05s/9; Solved: 4
5. Undermine.005,  STS_1-15   Solved in 0.03s/10; Solved: 5
6. Undermine.006,  STS_1-15   Solved in 0.06s/7; Solved: 6
7. Undermine.007,  STS_1-15   Solved in 0.02s/7; Solved: 7
8. Undermine.008,  STS_1-15   Solved in 0.13s/12; Solved: 8
9. Undermine.009,  STS_1-15   Solved in 0.02s/7; Solved: 9
10. Undermine.010,  STS_1-15   Solved in 0.03s/7; Solved: 10
11. Undermine.011,  STS_1-15   Solved in 0.03s/7; Solved: 11
12. Undermine.012,  STS_1-15   Solved in 0.05s/7; Solved: 12
13. Undermine.013,  STS_1-15   Solved in 0.09s/13; Solved: 13
14. Undermine.014,  STS_1-15   Solved in 0.05s/7; Solved: 14
15. Undermine.015,  STS_1-15   > 10s.
16. Undermine.016,  STS_1-15   Solved in 0.05s/7; Solved: 15
17. Undermine.017,  STS_1-15   Solved in 0.03s/7; Solved: 16
18. Undermine.018,  STS_1-15   Solved in 0.03s/7; Solved: 17
19. Undermine.019,  STS_1-15   Solved in 0.05s/7; Solved: 18
20. Undermine.020,  STS_1-15   Solved in 0.03s/7; Solved: 19
21. Undermine.021,  STS_1-15   Solved in 0.05s/9; Solved: 20
22. Undermine.022,  STS_1-15   Solved in 0.05s/11; Solved: 21
23. Undermine.023,  STS_1-15   Solved in 0.05s/12; Solved: 22
24. Undermine.024,  STS_1-15   Solved in 3.09s/22; Solved: 23
25. Undermine.025,  STS_1-15   Solved in 0.03s/7; Solved: 24
26. Undermine.026,  STS_1-15   Solved in 0.05s/7; Solved: 25
27. Undermine.027,  STS_1-15   Solved in 0.03s/11; Solved: 26
28. Undermine.28,  STS_1-15   Solved in 0.03s/7; Solved: 27
29. Undermine.029,  STS_1-15   Solved in 0.03s/11; Solved: 28
30. Undermine.030,  STS_1-15   Solved in 0.05s/7; Solved: 29
31. Undermine.031,  STS_1-15   Solved in 0.02s/7; Solved: 30
32. Undermine.032,  STS_1-15   Solved in 0.05s/7; Solved: 31
33. Undermine.033,  STS_1-15   Solved in 0.02s/7; Solved: 32
34. Undermine.034,  STS_1-15   Solved in 2.45s/22; Solved: 33
35. Undermine.035,  STS_1-15   Solved in 0.05s/8; Solved: 34
36. Undermine.036,  STS_1-15   Solved in 8.23s/26; Solved: 35
37. Undermine.037,  STS_1-15   > 10s.
38. Undermine.038,  STS_1-15   Solved in 0.05s/7; Solved: 36
39. Undermine.039,  STS_1-15   Solved in 0.03s/7; Solved: 37
40. Undermine.040,  STS_1-15   Solved in 0.03s/7; Solved: 38
41. Undermine.041,  STS_1-15   Solved in 0.05s/11; Solved: 39
42. Undermine.42,  STS_1-15   Solved in 0.03s/7; Solved: 40
43. Undermine.043,  STS_1-15   Solved in 0.06s/11; Solved: 41
44. Undermine.044,  STS_1-15   Solved in 0.05s/9; Solved: 42
45. Undermine.045,  STS_1-15   Solved in 1.41s/19; Solved: 43
46. Undermine.046,  STS_1-15   Solved in 0.03s/8; Solved: 44
47. Undermine.047,  STS_1-15   Solved in 0.06s/7; Solved: 45
48. Undermine.048,  STS_1-15   Solved in 0.05s/9; Solved: 46
49. Undermine.049,  STS_1-15   Solved in 0.05s/10; Solved: 47
50. Undermine.050,  STS_1-15   Solved in 0.09s/12; Solved: 48
51. Undermine.051,  STS_1-15   Solved in 1.50s/22; Solved: 49
52. Undermine.052,  STS_1-15   Solved in 0.20s/9; Solved: 50
53. Undermine.053,  STS_1-15   Solved in 0.05s/9; Solved: 51
54. Undermine.054,  STS_1-15   Solved in 0.05s/7; Solved: 52
55. Undermine.055,  STS_1-15   Solved in 0.03s/7; Solved: 53
56. Undermine.056,  STS_1-15   Solved in 0.20s/8; Solved: 54
57. Undermine.057,  STS_1-15   Solved in 0.05s/13; Solved: 55
58. Undermine.058,  STS_1-15   Solved in 0.03s/7; Solved: 56
59. Undermine.059,  STS_1-15   Solved in 0.06s/7; Solved: 57
60. Undermine.060,  STS_1-15   Solved in 0.02s/7; Solved: 58
61. Undermine.061,  STS_1-15   Solved in 3.92s/22; Solved: 59
62. Undermine.062,  STS_1-15   Solved in 0.03s/7; Solved: 60
63. Undermine.063,  STS_1-15   Solved in 0.05s/7; Solved: 61
64. Undermine.064,  STS_1-15   Solved in 0.03s/7; Solved: 62
65. Undermine.065,  STS_1-15   Solved in 0.05s/7; Solved: 63
66. Undermine.066,  STS_1-15   Solved in 0.03s/7; Solved: 64
67. Undermine.067,  STS_1-15   Solved in 0.02s/8; Solved: 65
68. Undermine.068,  STS_1-15   Solved in 0.06s/10; Solved: 66
69. Undermine.069,  STS_1-15   Solved in 0.05s/7; Solved: 67
70. Undermine.070,  STS_1-15   Solved in 0.05s/7; Solved: 68
71. Undermine.071,  STS_1-15   > 10s.
72. Undermine.072,  STS_1-15   Solved in 0.03s/7; Solved: 69
73. Undermine.073,  STS_1-15   Solved in 0.17s/15; Solved: 70
74. Undermine.074,  STS_1-15   Solved in 0.25s/15; Solved: 71
75. Undermine.075,  STS_1-15   > 10s.
76. Undermine.076,  STS_1-15   Solved in 0.23s/13; Solved: 72
77. Undermine.077,  STS_1-15   Solved in 0.05s/7; Solved: 73
78. Undermine.078,  STS_1-15   Solved in 0.05s/7; Solved: 74
79. Undermine.079,  STS_1-15   Solved in 0.53s/17; Solved: 75
80. Undermine.080,  STS_1-15   Solved in 0.05s/7; Solved: 76
81. Undermine.081,  STS_1-15   Solved in 0.20s/10; Solved: 77
82. Undermine.082,  STS_1-15   Solved in 0.03s/7; Solved: 78
83. Undermine.083,  STS_1-15   Solved in 0.05s/7; Solved: 79
84. Undermine.084,  STS_1-15   Solved in 2.30s/19; Solved: 80
85. Undermine.085,  STS_1-15   Solved in 1.22s/20; Solved: 81
86. Undermine.086,  STS_1-15   Solved in 0.19s/7; Solved: 82
87. Undermine.087,  STS_1-15   Solved in 0.05s/11; Solved: 83
88. Undermine.088,  STS_1-15   Solved in 0.05s/7; Solved: 84
89. Undermine.089,  STS_1-15   Solved in 0.06s/7; Solved: 85
90. Undermine.090,  STS_1-15   Solved in 0.11s/13; Solved: 86
91. Undermine.091,  STS_1-15   Solved in 0.03s/9; Solved: 87
92. Undermine.092,  STS_1-15   Solved in 0.03s/7; Solved: 88
93. Undermine.093,  STS_1-15   Solved in 0.13s/14; Solved: 89
94. Undermine.094,  STS_1-15   Solved in 0.03s/10; Solved: 90
95. Undermine.095,  STS_1-15   Solved in 0.08s/12; Solved: 91
96. Undermine.096,  STS_1-15   Solved in 0.14s/12; Solved: 92
97. Undermine.097,  STS_1-15   Solved in 0.05s/7; Solved: 93
98. Undermine.098,  STS_1-15   Solved in 0.03s/7; Solved: 94
99. Undermine.099,  STS_1-15   Solved in 0.05s/10; Solved: 95
100. Undermine.100,  STS_1-15   Solved in 0.20s/7; Solved: 96
101. STS(v10.0) Simplification.001,  STS_1-15   Solved in 0.02s/7; Solved: 97
102. STS(v10.0) Simplification.002,  STS_1-15   > 10s.
103. STS(v10.0) Simplification.003,  STS_1-15   Solved in 0.03s/9; Solved: 98
104. STS(v10.0) Simplification.004,  STS_1-15   Solved in 0.03s/10; Solved: 99
105. STS(v10.0) Simplification.005,  STS_1-15   Solved in 0.06s/7; Solved: 100
106. STS(v10.0) Simplification.006,  STS_1-15   Solved in 0.05s/7; Solved: 101
107. STS(v10.0) Simplification.007,  STS_1-15

Result: 101 out of 107 = 94.3%. Average time = 0.33s / 9.81

[Jouni]

I wasted some 40 hours last week in testing. Fritz GUI because Arena crashes after around 150 positions with multicore . Tested with corei5 and 4 cores. 20s and 1s runs:

Code: Select all

Houdini 3     1385/1302 (+83)
SF200116      1362/1264 (+98)
Komodo 9.2    1360/1288 (+72)
SF7           1348/1263 (+85)

[Norm Pollock]

Actually 1497 due to 3 duplicates.

1q2r1k1/1b2bpp1/p2ppn1p/2p5/P3PP1B/2PB1RP1/2P1Q2P/2KR4 b - - c0 2; c1 line(s): 5 1405;
1r1r1bk1/1bq2p1p/pn2p1p1/2p1P3/5P2/P1NBB3/1P3QPP/R2R2K1 b - - c0 2; c1 line(s): 308 804;
r1r3k1/pb3p1p/1pqBp1p1/4P3/3b4/2P2P2/PR1N2PP/2RQ3K w - - c0 2; c1 line(s): 377 873;

[bob]

Swaminathan has collected 1,500 strategical (or positional as I assume) test positions. For your convenience in one EPD file:

http://computerschach.twoday.net/files/STS1-15 (61 KB zip)

Source: https://sites.google.com/site/strategictestsuite/

Results welcome A test run with 10 seconds/pos. only will take about four hours.

Here's current (25.1) crafty, 10 seconds per move, 20 cores, with one addition: If the correct move is chosen for 10 consecutive plies, the search is stopped even if 10 seconds has not been used...

total positions searched.......... 1501
number right...................... 1313
number wrong...................... 188
percentage right.................. 87
percentage wrong.................. 12
total nodes searched..............281658977336
average search depth.............. 16.5
nodes per second.................. 74650606
total time........................ 62:53

That is only matching the correct move. I have not looked to see if there is any unusual scoring involved (fractional point for second best move and such).

This is far from being the optimal time limit for 20 cores. I will run it at 60 seconds per position overnight to see what happens... That lets the SMP search get pretty well cranked up.

[bob]

Here's the 60 second results, again only counting right and wrong, not the various different scores for different moves...

total positions searched.......... 1501
number right...................... 1358
number wrong...................... 143
percentage right.................. 90
percentage wrong.................. 9
total nodes searched..............1295238136842
average search depth.............. 17.4
nodes per second.................. 85047695
total time........................ 253:49

[Mike S.]

Thanks to all for the replies.

It seems the test could be compressed to 500 or less positions, by (simply speaking) removing the meaningless ones. Those would be (a) pos which every engine solves under medium conditions, and which (b) no engine solves even under good conditions. But that is probably a titans task...