Hard-Talkchess-2020 set, final release

[Alayan]

I think this shows that the ability to solve these trick positions has little relevance when it comes to the overall playing strength of an engine. Xiphos and Ethereal are much stronger than Houdini 1.5 and Critter 1.6.

At the same time, these positions show us how complex chess is and that there is an immense amount of winning sequences that are completely missed during search. The improvement potential is enormous.

[Jouni]

2 things: 1. Clearly Houdini 1.5 and Critter 1.6 are same engine . 2. And Crystal best in test, when it uses least time. Why? Imagine situation where engine A solves all 213 positions in 15 minutes each, but engine B "only" 212 positions in 15 seconds each. Who is by far better?

[Ovyron]

Imagine situation where engine A solves all 213 positions in 15 minutes each, but engine B "only" 212 positions in 15 seconds each. Who is by far better?

Who knows? Let them play a match and you'll figure it out under some error bars. If solving a test set could tell us what engine was best, TCEC and the CCRL would fire all their volunteers and dedicate their time to solve test sets instead of playing games.

[Vinvin]

can anyone please post a few epds of the toughest ones with a brief comment about the right plan to follow?

...
About the plan to follow, I will import all the positions in "Lichess Studies" (packs of 20 positions) with mainlines in following days.

I imported all the positions in Lichess Studies (but not all the mainlines yet) :
#1 to #60 : https://lichess.org/study/y6sM5eAf
#61 to #120 : https://lichess.org/study/Vfe59GOH
#121 to #180 : https://lichess.org/study/0FQa6riN
#181 to #213 : https://lichess.org/study/s2mTErEZ

Mainlines added for : #1 to #60 : https://lichess.org/study/y6sM5eAf

[Vinvin]

Restarted the live test at
with Black-Diamond-XI, McCain-v2 and Sting-sf-16

Current summary of results (30 minutes, 1 core per engine) :

Stockfish 11 : 141/213
Stockfish 10 : 137/213
Black-Diamond-XR7 : 136/213
Crystal 1.1 : 135/213
Houdini 1.5 x64 : 97/213
Critter 1.6a 64bit : 96/213
Xiphos 0.6 : 81/213
Ethereal 11.75 : 77/213

Replays of solving are here :

[MOBMAT]

What is your criteria for entry into the list?

[Vinvin]

New results :
Sting-SF-16 : 122/213
McCain-v2 : 130/213
Black-Diamond-XI : 136/213
Honey-XI : 139/213

Now with SF 7, SF 8 and Bluefish-XI :

Restarted the live test at
with Black-Diamond-XI, McCain-v2 and Sting-sf-16

Current summary of results (30 minutes, 1 core per engine) :

Stockfish 11 : 141/213
Stockfish 10 : 137/213
Black-Diamond-XR7 : 136/213
Crystal 1.1 : 135/213
Houdini 1.5 x64 : 97/213
Critter 1.6a 64bit : 96/213
Xiphos 0.6 : 81/213
Ethereal 11.75 : 77/213

Replays of solving are here :

[Vinvin]

2 more results :
Bluefish-XI : 159/213 NEW LEADER !
Stockfish 7 : 76/213

Current standing :
Bluefish-XI : 159/213
Stockfish 11 : 141/213
Honey-XI : 139/213
Stockfish 10 : 137/213
Black-Diamond-XI : 136/213
Black-Diamond-XR7 : 136/213
Crystal 1.1 : 135/213
McCain-v2 : 130/213
Sting-SF-16 : 122/213
Houdini 1.5 x64 : 97/213
Critter 1.6a 64bit : 96/213
Xiphos 0.6 : 81/213
Ethereal 11.75 : 77/213
Stockfish 7 : 76/213

Still running : Stockfish 8 ... 93/210

New results :
Sting-SF-16 : 122/213
McCain-v2 : 130/213
Black-Diamond-XI : 136/213
Honey-XI : 139/213

Now with SF 7, SF 8 and Bluefish-XI :

Restarted the live test at
with Black-Diamond-XI, McCain-v2 and Sting-sf-16

Current summary of results (30 minutes, 1 core per engine) :

Stockfish 11 : 141/213
Stockfish 10 : 137/213
Black-Diamond-XR7 : 136/213
Crystal 1.1 : 135/213
Houdini 1.5 x64 : 97/213
Critter 1.6a 64bit : 96/213
Xiphos 0.6 : 81/213
Ethereal 11.75 : 77/213

Replays of solving are here :

[xr_a_y]

2 more results :
Bluefish-XI : 159/213 NEW LEADER !
Stockfish 7 : 76/213

Current standing :
Bluefish-XI : 159/213
Stockfish 11 : 141/213
Honey-XI : 139/213
Stockfish 10 : 137/213
Black-Diamond-XI : 136/213
Black-Diamond-XR7 : 136/213
Crystal 1.1 : 135/213
McCain-v2 : 130/213
Sting-SF-16 : 122/213
Houdini 1.5 x64 : 97/213
Critter 1.6a 64bit : 96/213
Xiphos 0.6 : 81/213
Ethereal 11.75 : 77/213
Stockfish 7 : 76/213

Still running : Stockfish 8 ... 93/210

Please can you recall how much time for each position ? how many threads ? size of TT ?

[Vinvin]

2 more results :
Bluefish-XI : 159/213 NEW LEADER !
Stockfish 7 : 76/213

Current standing :
Bluefish-XI : 159/213
Stockfish 11 : 141/213
Honey-XI : 139/213
Stockfish 10 : 137/213
Black-Diamond-XI : 136/213
Black-Diamond-XR7 : 136/213
Crystal 1.1 : 135/213
McCain-v2 : 130/213
Sting-SF-16 : 122/213
Houdini 1.5 x64 : 97/213
Critter 1.6a 64bit : 96/213
Xiphos 0.6 : 81/213
Ethereal 11.75 : 77/213
Stockfish 7 : 76/213

Still running : Stockfish 8 ... 93/210

Please can you recall how much time for each position ? how many threads ? size of TT ?

30 minutes with 1 core for each position. 2 GB of HT (I set to 2200 MB in Arena).
I'll publish a sheet with all timing in next days.

Vincent