Official: Lc0 is the strongest engine :)

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 8450
Joined: Wed Jul 26, 2006 8:21 pm

Re: Official: Lc0 is the strongest engine :)

Post by Laskos » Sat Oct 13, 2018 1:27 pm

It's even more Official: Lc0 is the strongest engine

In the Openings and Middlegames

After correcting for the methodological problem Uri pointed out, I got again that Lc0 is the strongest engine in openings and middlegames. To have some statistical significance, first at very fast time control, 0.2s/move.
At this short time control, Lc0 performs badly, and I got from 4-mover PGN opening suite the following, playing the full games without any adjudication:

Code: Select all

Score of lc0_v18.1 11261 vs SF8 (4 cores): 
+20 -117 =63 [0.258]
Elo difference: -183.97 +/- 42.52

200 of 200 games finished. 
Now, at the same 0.2s/move time control, I played the first 25 moves Lc0 vs SF8, and the rest SF8 vs SF8. The result is dramatically different and advantageous for the side using Lc0 for the first 25 moves:

+55 -34 =111 55.25%
+37 Elo points
LOS = 98.7%

Lc0 is stronger than SF8 (4 cores) in the first 25 moves, adding value to the final result even at this short TC, after the rest of the games (endgames) were finished as SF8 vs SF8. And the difference of the two performances is a whopping 200+ Elo points.


To see how well Lc0 fares in the first 25 moves against SF-dev, I used longer 2'+2'' time control (similar to CCRL 40/4' conditions), and again, first 25 moves were played Lc0 vs SF-dev, the rest (endgames) were SF-dev vs SF-dev (all engines Syzygy enabled). The result in 100 games is:

+20 -15 =65 52.5%
LOS = 80.0%

Not very conclusive, but it seems Lc0 is stronger than SF-dev (4 cores) in these conditions in the first 25 moves.

Another aspect: analyzing the 100 games at 2'+2'', 4 were lost by Lc0 by game-changing tactical blunders (one of them was from fairly balanced position to SF-dev showing Mate score) during those initial 25 moves. So, a combination of Lc0 + Houdini Tactical (for tactical blunders above 100cp) in the openings and middlegames, and SF-dev for the late middlegames and endgames would score:

+20 -11 =69

against SF-dev. Houdini Tactical saw the tactical blunders almost instantly, so the time allotted to it shouldn't be too high. Also, these crass tactical blunders occur only in some 5% of the games in the openings and middlegames. This "combination" of engines, which is clearly stronger than SF-dev, might interest, for example, correspondence chess players. I tried to automate this kind of playing using ChessCombi or Nucleus, without success.

megamau
Posts: 34
Joined: Wed Feb 10, 2016 5:20 am
Location: Singapore

Re: Official: Lc0 is the strongest engine :)

Post by megamau » Sat Oct 13, 2018 1:50 pm

It is actually one of the path to improvement.
Dedicated Alpha / Beta "blunder checker", perhaps running on CPU which usually have spare cycles for LC0.

peter
Posts: 1628
Joined: Sat Feb 16, 2008 6:38 am

Re: Official: Lc0 is the strongest engine :)

Post by peter » Sat Oct 13, 2018 1:55 pm

Hi Kai!
Thanks for you testing.
Laskos wrote:
Sat Oct 13, 2018 1:27 pm
This "combination" of engines, which is clearly stronger than SF-dev, might interest, for example, correspondence chess players. I tried to automate this kind of playing using ChessCombi or Nucleus, without success.
Nucleus I do know, but what's ChessCombi?
Peter Martan

User avatar
Laskos
Posts: 8450
Joined: Wed Jul 26, 2006 8:21 pm

Re: Official: Lc0 is the strongest engine :)

Post by Laskos » Sat Oct 13, 2018 4:17 pm

peter wrote:
Sat Oct 13, 2018 1:55 pm
Hi Kai!
Thanks for you testing.
Laskos wrote:
Sat Oct 13, 2018 1:27 pm
This "combination" of engines, which is clearly stronger than SF-dev, might interest, for example, correspondence chess players. I tried to automate this kind of playing using ChessCombi or Nucleus, without success.
Nucleus I do know, but what's ChessCombi?
It's an older utility, maybe from year 2011 or so, but it is not working as I expected.

From the Readme.txt:

CHESSCOMBI V1
created by Mark Alba

Release: ChessCombi is distributed free of charge.
ChessCombi may not be distributed as part of any software package,
service or website without prior written permission from Mark Alba.

Support: If you encountered any problems, email me at detective_mark7@yahoo.com.

ChessCombi is a UCI chess engine that combines two UCI chess engines into 1.
It can be used in a typical GUI program that supports UCI.
The combination of 2 chess engine can increase playing style but not necessarily produce better results.
The input from ChessCombi is directly fed to the input of the 2 engines.
But the output of the 2 engines however, is screened so that switching occurs.
The ChessCombi V1 is only available in Windows.

peter
Posts: 1628
Joined: Sat Feb 16, 2008 6:38 am

Re: Official: Lc0 is the strongest engine :)

Post by peter » Sat Oct 13, 2018 5:40 pm

Laskos wrote:
Sat Oct 13, 2018 4:17 pm
It's an older utility, maybe from year 2011 or so, but it is not working as I expected.
Thanks for the infos.
Peter Martan

lkaufman
Posts: 3190
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Official: Lc0 is the strongest engine :)

Post by lkaufman » Sat Oct 13, 2018 11:50 pm

You inspired me to run a test to see if Komodo MCTS also plays middlegames relatively stronger than endgames. Current Komodo MCTS (dev) on one thread now can beat normal Komodo 9.1 (by 31 elo in my test). I then reran the test starting with positions with the queens already exchanged. The result was almost the same (+27 elo). So the endgame dropoff is unique to Lc0, which suggests that the cause is the neural network, rather than the MCTS search. It seems to me that the strength of Lc0 is not because the neural network evals are all that good, but is because they are much faster than a similar quality short normal search if they are run on a good GPU. It's as if the fast GPU makes the NN act like a rather poor evaluator with a normal search running at super-speed. Maybe I'm wrong, but that's the way it looks to me. If someone finds a way to use a good GPU effectively with a more normal eval, that might be the "holy grail" of computer chess.
Komodo rules!

dkappe
Posts: 12
Joined: Tue Aug 21, 2018 5:52 pm
Full name: Dietrich Kappe

Re: Official: Lc0 is the strongest engine :)

Post by dkappe » Sun Oct 14, 2018 3:08 am

megamau wrote:
Sat Oct 13, 2018 1:50 pm
It is actually one of the path to improvement.
Dedicated Alpha / Beta "blunder checker", perhaps running on CPU which usually have spare cycles for LC0.
I made one of these for leela in the days of the “blunder prone’ 192x15 networks. You can see the background and results: https://github.com/dkappe/leela-chess-w ... r-Checking

smatovic
Posts: 493
Joined: Wed Mar 10, 2010 9:18 pm
Location: Germany
Full name: Srdja Matovic
Contact:

Re: Official: Lc0 is the strongest engine :)

Post by smatovic » Sun Oct 14, 2018 8:19 am

lkaufman wrote:
Sat Oct 13, 2018 11:50 pm
It seems to me that the strength of Lc0 is not because the neural network evals are all that good, but is because they are much faster than a similar quality short normal search if they are run on a good GPU. It's as if the fast GPU makes the NN act like a rather poor evaluator with a normal search running at super-speed. Maybe I'm wrong, but that's the way it looks to me.
Hm, afaik LC0 makes only about 10 Knps to 40 Knps on top GPU with 2 threads on Host.

So it should be the other way around, cos of the good NN evaluation LC0 needs less nodes to search.
lkaufman wrote:
Sat Oct 13, 2018 11:50 pm
If someone finds a way to use a good GPU effectively with a more normal eval, that might be the "holy grail" of computer chess.
I agree, there is a lot of horse power present in GPUs,
unfortunately there are some limitations like SIMT architecture and memory hierarchy.

--
Srdja

duncan
Posts: 8116
Joined: Mon Jul 07, 2008 8:50 pm

Re: Official: Lc0 is the strongest engine :)

Post by duncan » Sun Oct 14, 2018 10:45 am

lkaufman wrote:
Sat Oct 13, 2018 11:50 pm
You inspired me to run a test to see if Komodo MCTS also plays middlegames relatively stronger than endgames. Current Komodo MCTS (dev) on one thread now can beat normal Komodo 9.1 (by 31 elo in my test). I then reran the test starting with positions with the queens already exchanged. The result was almost the same (+27 elo). So the endgame dropoff is unique to Lc0, which suggests that the cause is the neural network, rather than the MCTS search. It seems to me that the strength of Lc0 is not because the neural network evals are all that good, but is because they are much faster than a similar quality short normal search if they are run on a good GPU. It's as if the fast GPU makes the NN act like a rather poor evaluator with a normal search running at super-speed. Maybe I'm wrong, but that's the way it looks to me. If someone finds a way to use a good GPU effectively with a more normal eval, that might be the "holy grail" of computer chess.
A bit off topic, but do you think there is any risk that lco will overtake Komodo, in the next year. I get the impression most people here say yes. But you are in the best position to know.

jp
Posts: 232
Joined: Mon Apr 23, 2018 5:54 am

Re: Official: Lc0 is the strongest engine :)

Post by jp » Sun Oct 14, 2018 11:18 am

duncan wrote:
Sun Oct 14, 2018 10:45 am
lkaufman wrote:
Sat Oct 13, 2018 11:50 pm
A bit off topic, but do you think there is any risk that lco will overtake Komodo, in the next year.
But you are in the best position to know.
On the same off topic, why is Komodo playing so badly on chess.com's CCCC right now? Is it a buggy version?

Post Reply