Lc0 51010

[Raphexon]

In tests with Arena 3.5.1, Stockfish 10 bit the dust vs Herman 2.8 when nodes per move was limited to 2500 nodes (or less.)
Stockfish started being slightly superior to Herman around the 2700 node per move mark.

SF isn't optimized to deal with such low node counts.
I wouldn't be surprised if SF gets completely demolished vs Mephisto if limited to less than 1000 nodes per move.

[lkaufman]

LC0 engine does have a search, but we are discussing here the outputs of the neural network itself without search (you can get LC0 to do this with the command nodes=0).

Some people are claiming that the network output without search is 2400 or whatever, or they know somebody who knows somebody who says it is very strong (at blitz). Well, I posted some games from a batch of 100 test games in which I took a look at the moves that caused LC0 to lose, in another thread that shows LC0 policy alone plays some completely idiotic moves, so at least my disagreement with 2400 has some data to back it. Of course, LC0 policy plays some good moves, it's just statistics and sometimes it screws up.

FWIW. adding another layer to this, I was actually very surprised by the 2700's comment, and chose not to say that my one experiment with this last year had been different. There had been claims that Test10, I do not recall which build, was a good 2400 at 1 node, based on some spreadsheet. I found that absurd, but decided to test it myself. I played against it from the official lczero.org website with some Sicilian from White's side. I figured tactics would be the Achilles Heel and was correct as it fell for a two-mover that dropped a piece, then a few moves a pawn. I concluded that while positionally it was no doubt stronger than I was, even at 1 node (never was my forte anyhow), I could not get behind the 2400 Elo statement.I have not bothered to revisit this test since then, but have trouble imagining that Test30 or 40, the latest Leelas, or even Alpha Zero, would somehow be vastly different. This was only one test, one game, so certainly not statistically significant, and perhaps I just got lucky that my one example fit my expectation. Still...

OK, I got some real data on the question. I determined that latest 40xxx Lc0 playing policy move only (setting one ply does this) is a close match with skill level 14 on Komodo 12.3; Lc0 won by 26.5 to 23.5 (21 elo). Skill level 14 has a rapid chess.com rating of 2038. So based on this we can expect that policy network move would earn a rating at rapid (not blitz) time controls of around 2060. This is probably reasonably close to the truth. My experience is that in 4' + 2" blitz I usually lose, but with a bit more time like 5' + 5" I become the favorite. It's a bit strange that a 2700 would lose at blitz though; perhaps they were bullet games?

[jp]

Exactly. Stockfish is artificial stupidity. It concentrates on search instead of knowledge. Take away the millions and millions of NPS and you see the artificial stupidity.

It is more a pocket calculator with enormous power then an entity that can play chess.

But all programs, including NN engines, rely on "millions and millions" of operations to play very strong chess. It sounds like you think number of nodes is the correct measure of how much calculation it's doing. That's not correct. So if you think doing lots of calculations is "artificial stupidity", then all engines are artificial stupidity. Why single out SF? We can agree that all chess engines are not AI.

[jp]

OK, I got some real data on the question. I determined that latest 40xxx Lc0 playing policy move only (setting one ply does this) is a close match with skill level 14 on Komodo 12.3; Lc0 won by 26.5 to 23.5 (21 elo). Skill level 14 has a rapid chess.com rating of 2038. So based on this we can expect that policy network move would earn a rating at rapid (not blitz) time controls of around 2060. This is probably reasonably close to the truth. My experience is that in 4' + 2" blitz I usually lose, but with a bit more time like 5' + 5" I become the favorite. It's a bit strange that a 2700 would lose at blitz though; perhaps they were bullet games?

This sounds more believable.

[nabildanial]

LC0 engine does have a search, but we are discussing here the outputs of the neural network itself without search (you can get LC0 to do this with the command nodes=0).

Some people are claiming that the network output without search is 2400 or whatever, or they know somebody who knows somebody who says it is very strong (at blitz). Well, I posted some games from a batch of 100 test games in which I took a look at the moves that caused LC0 to lose, in another thread that shows LC0 policy alone plays some completely idiotic moves, so at least my disagreement with 2400 has some data to back it. Of course, LC0 policy plays some good moves, it's just statistics and sometimes it screws up.

FWIW. adding another layer to this, I was actually very surprised by the 2700's comment, and chose not to say that my one experiment with this last year had been different. There had been claims that Test10, I do not recall which build, was a good 2400 at 1 node, based on some spreadsheet. I found that absurd, but decided to test it myself. I played against it from the official lczero.org website with some Sicilian from White's side. I figured tactics would be the Achilles Heel and was correct as it fell for a two-mover that dropped a piece, then a few moves a pawn. I concluded that while positionally it was no doubt stronger than I was, even at 1 node (never was my forte anyhow), I could not get behind the 2400 Elo statement.I have not bothered to revisit this test since then, but have trouble imagining that Test30 or 40, the latest Leelas, or even Alpha Zero, would somehow be vastly different. This was only one test, one game, so certainly not statistically significant, and perhaps I just got lucky that my one example fit my expectation. Still...

OK, I got some real data on the question. I determined that latest 40xxx Lc0 playing policy move only (setting one ply does this) is a close match with skill level 14 on Komodo 12.3; Lc0 won by 26.5 to 23.5 (21 elo). Skill level 14 has a rapid chess.com rating of 2038. So based on this we can expect that policy network move would earn a rating at rapid (not blitz) time controls of around 2060. This is probably reasonably close to the truth. My experience is that in 4' + 2" blitz I usually lose, but with a bit more time like 5' + 5" I become the favorite. It's a bit strange that a 2700 would lose at blitz though; perhaps they were bullet games?

I believe net 11258 is better at 1 node than any T30/T40 nets.

[lkaufman]

LC0 engine does have a search, but we are discussing here the outputs of the neural network itself without search (you can get LC0 to do this with the command nodes=0).

Some people are claiming that the network output without search is 2400 or whatever, or they know somebody who knows somebody who says it is very strong (at blitz). Well, I posted some games from a batch of 100 test games in which I took a look at the moves that caused LC0 to lose, in another thread that shows LC0 policy alone plays some completely idiotic moves, so at least my disagreement with 2400 has some data to back it. Of course, LC0 policy plays some good moves, it's just statistics and sometimes it screws up.

FWIW. adding another layer to this, I was actually very surprised by the 2700's comment, and chose not to say that my one experiment with this last year had been different. There had been claims that Test10, I do not recall which build, was a good 2400 at 1 node, based on some spreadsheet. I found that absurd, but decided to test it myself. I played against it from the official lczero.org website with some Sicilian from White's side. I figured tactics would be the Achilles Heel and was correct as it fell for a two-mover that dropped a piece, then a few moves a pawn. I concluded that while positionally it was no doubt stronger than I was, even at 1 node (never was my forte anyhow), I could not get behind the 2400 Elo statement.I have not bothered to revisit this test since then, but have trouble imagining that Test30 or 40, the latest Leelas, or even Alpha Zero, would somehow be vastly different. This was only one test, one game, so certainly not statistically significant, and perhaps I just got lucky that my one example fit my expectation. Still...

OK, I got some real data on the question. I determined that latest 40xxx Lc0 playing policy move only (setting one ply does this) is a close match with skill level 14 on Komodo 12.3; Lc0 won by 26.5 to 23.5 (21 elo). Skill level 14 has a rapid chess.com rating of 2038. So based on this we can expect that policy network move would earn a rating at rapid (not blitz) time controls of around 2060. This is probably reasonably close to the truth. My experience is that in 4' + 2" blitz I usually lose, but with a bit more time like 5' + 5" I become the favorite. It's a bit strange that a 2700 would lose at blitz though; perhaps they were bullet games?

I believe net 11258 is better at 1 node than any T30/T40 nets.

Yes, it only took a few minutes to test that; 11258 beat 41812 by 29 to 21 (56 elo). It beat Komodo level 14 by 32.5 to 17.5 and even beat level 15 by 28 to 22. So I suppose it would earn a rapid rating in the mid 2100s and a blitz rating in the mid 2300s.

[EroSennin]

I played T40 policy in 5 blitz games and managed two draws. One draw was due to it stalemating me in a winning position. I resigned two of the games, so there might have been more draws like that. Honestly don't feel like I have any chances to win it. My elo is 2400. If you exclude the poor endgame play, it definitely plays blitz at gm level.

[jp]

If you exclude the poor endgame play, it definitely plays blitz at gm level.

But you mean GM blitz level, don't you, not GM classical level? How many elo does a human GM lose (just based on the quality of moves) going from classical TC to blitz TC?

[EroSennin]

If you exclude the poor endgame play, it definitely plays blitz at gm level.

But you mean GM blitz level, don't you, not GM classical level? How many elo does a human GM lose (just based on the quality of moves) going from classical TC to blitz TC?

Yes I a mean blitz as I have no proof of longer time controls. I think Larry has said humans lose around 500 elos.

I just won one blitz game, but I forced it to start with 1.d4. It became an endgame, which I guess requires calculation the most. It seems success heavily depends what opening one plays.

[jp]

Yes I a mean blitz as I have no proof of longer time controls. I think Larry has said humans lose around 500 elos.

That would be consistent with the guess that at N=0 it plays at no more than roughly 2000 elo.
(Larry above estimates a rapid rating around 2060.)