In a thrilling battle that came down to the final games, Stockfish won the Chess.com Computer Chess Championship this week, edging out the machine-learning engine Lc0.
Stockfish scored 82.5/150 to reclaim its championship title and top Lc0, which came in at 81 points. Lc0 had won the last Computer Chess Championship in a historic performance.
The neural-network engine Leelenstein was third with 76/150, and Houdini trailed far behind in fourth at 60.5 points in the CCC 8: Deep Dive finals.
Lc0 defeated Stockfish in their head-to-head match, four wins to three. Lc0 also won its head-to-head matches with the other two finalists. But Stockfish performed better in the four-engine round-robin to take back the title by beating up on the two weaker engines.
In a thrilling battle that came down to the final games, Stockfish won the Chess.com Computer Chess Championship this week, edging out the machine-learning engine Lc0.
Stockfish scored 82.5/150 to reclaim its championship title and top Lc0, which came in at 81 points. Lc0 had won the last Computer Chess Championship in a historic performance.
The neural-network engine Leelenstein was third with 76/150, and Houdini trailed far behind in fourth at 60.5 points in the CCC 8: Deep Dive finals.
Lc0 defeated Stockfish in their head-to-head match, four wins to three. Lc0 also won its head-to-head matches with the other two finalists. But Stockfish performed better in the four-engine round-robin to take back the title by beating up on the two weaker engines.
Head to head matters. Lc0 always underperforms against weaker engines. It will always show deflated rating in rating lists. Lc0 doesn't obey the Elo model in a pool of regular engines. In TCEC Premiere Division, Lc0 came second too, after SF, but in head-to-head Superfinal it is clobbering SF.
Laskos wrote: ↑Sat May 25, 2019 9:38 amLc0 doesn't obey the Elo model in a pool of regular engines. In TCEC Premiere Division, Lc0 came second too, after SF, but in head-to-head Superfinal it is clobbering SF.
It could be that the SF team has worked intensively on contempt since before version 9, while Lc0 doesn't play that game, just chess.
Laskos wrote: ↑Sat May 25, 2019 9:38 amLc0 doesn't obey the Elo model in a pool of regular engines. In TCEC Premiere Division, Lc0 came second too, after SF, but in head-to-head Superfinal it is clobbering SF.
It could be that the SF team has worked intensively on contempt since before version 9, while Lc0 doesn't play that game, just chess.
Both SF contempt and the misses of Leela against weaker opponents. Leela can easily fail to convert a win against a much weaker engine.
Ozymandias wrote: ↑Sat May 25, 2019 10:00 am
It could be that the SF team has worked intensively on contempt since before version 9, while Lc0 doesn't play that game, just chess.
Is there really intensive work to be done there, though? You can set it more aggressively (no real work). You can make it adjust dynamically (some work). What else is there to do?
Laskos wrote: ↑Sat May 25, 2019 10:15 am
Both SF contempt and the misses of Leela against weaker opponents. Leela can easily fail to convert a win against a much weaker engine.
There is still a lot of easily accessible low lying fruit with regards to Lc0. Lot of areas that can be improved and it is just a matter of time before that happens. Time management is something that needs a serious look with regards to the version that is playing at TCEC and that probably cost Lc0 several unnecessary losses. And although it has improved a lot since last TCEC, there is still some very obvious tactical blindness that causes some blunders and some weak endgame play. I think once there is less ELO to be gained by simply choosing a newer network, these things will start to be worked on. At the moment there still seems a good ways to go with regards to networks. I also think that in TCEC the +10 rule should be changed as +10 for Lc0 does not mean the same as +10 for Komodo or SF or Houdini and on many occasions Lc0 will not find the win even if it is +10 ahead.
M ANSARI wrote: ↑Sat May 25, 2019 5:35 pm I also think that in TCEC the +10 rule should be changed as +10 for Lc0 does not mean the same as +10 for Komodo or SF or Houdini and on many occasions Lc0 will not find the win even if it is +10 ahead.
Yes, I saw the danger with adjudication and Lc0, so all the matches I've played so far have been with adjudication turned off completely. Makes for long tortuous games sometimes that is for sure. I'll find a compromise perhaps.
M ANSARI wrote: ↑Sat May 25, 2019 5:35 pm I also think that in TCEC the +10 rule should be changed as +10 for Lc0 does not mean the same as +10 for Komodo or SF or Houdini and on many occasions Lc0 will not find the win even if it is +10 ahead.
Yes, I saw the danger with adjudication and Lc0, so all the matches I've played so far have been with adjudication turned off completely. Makes for long tortuous games sometimes that is for sure. I'll find a compromise perhaps.
Yes, this is a big problem with TCEC. A compromise is to have draw adjudication rules but play out wins to checkmate or a TB win.
M ANSARI wrote: ↑Sat May 25, 2019 5:35 pm I also think that in TCEC the +10 rule should be changed as +10 for Lc0 does not mean the same as +10 for Komodo or SF or Houdini and on many occasions Lc0 will not find the win even if it is +10 ahead.
Yes, I saw the danger with adjudication and Lc0, so all the matches I've played so far have been with adjudication turned off completely. Makes for long tortuous games sometimes that is for sure. I'll find a compromise perhaps.
Yes, this is a big problem with TCEC. A compromise is to have draw adjudication rules but play out wins to checkmate or a TB win.
I think this is a problem with Leela, not TCEC. The fact that her wpct -> cp formula results in such absurd numbers which don't fit the rather rough values that other engines output are the issue here, not the +10 for 10 rule. If you are -10 for 10 ply you should be hopelessly lost and if not then there is an issue (eg #fortress) with your eval.
The rule used to be -6 for a couple of moves iirc. That worked completely fine and nobody really complained with the very rare exception when the engines both clearly saw the win, but your average chess joe did not. I don't think I myself ever encountered a winning result under those rules that I had to really disagree with. Now can you imagine Leela under those rules?
Imo there is no justification whatsoever to not adjust the formula. You can complain about CP not meaning anything, you don't have to like CP, you don't have to use CP when you run the program at home, but it is a standard that people are used to and you would like to be on a similar scale to all the other engines that added this output.
