lkaufman wrote: ↑Fri Feb 22, 2019 7:45 pm
By the way, I note that it is not feasible to play against Lc0 at knight odds, because it soon starts to blunder pieces almost randomly. At smaller handicaps like pawn f7 and move or two this doesn't happen, it plays very well. It's almost as if all win probabilities beyond 99% are the same.
Does it do this blundering from both sides (knight up and knight down) or only from knight up?
Both sides, but more noticeably when down a piece or more. Basically, if its winning prob. is below 2% or so, it just "resigns" by making random blunders. Even taking more time doesn't generally cure this. Although Lc0 can probably beat top GMs giving them pawn or maybe pawn and move handicap just like Komodo does, I don't think that Lc0 can even beat a 1400 rated human at knight odds (Komodo can give knight odds in blitz to GMs, but in slow games only to players aroun 2000). Even with somewhat higher win probs., say around 5% or so, Lc0 still blunders, though not as often. I've seen the same behavior from a couple different networks, I don't think the specific network is the problem. Perhaps it's a problem with rarely occurring material balances, such as being down two pieces or more.
Knight Odds of Lco were already played by 2900 blitz rated players like Daniel Nardoitsky and other GMs at blitz time control. Lc0 won them comfortably. Beat them with 70-80% score with knight odd.
please review the games on his account by yourself.
lkaufman wrote: ↑Fri Feb 22, 2019 7:45 pm
By the way, I note that it is not feasible to play against Lc0 at knight odds, because it soon starts to blunder pieces almost randomly. At smaller handicaps like pawn f7 and move or two this doesn't happen, it plays very well. It's almost as if all win probabilities beyond 99% are the same.
Does it do this blundering from both sides (knight up and knight down) or only from knight up?
Both sides, but more noticeably when down a piece or more. Basically, if its winning prob. is below 2% or so, it just "resigns" by making random blunders. Even taking more time doesn't generally cure this. Although Lc0 can probably beat top GMs giving them pawn or maybe pawn and move handicap just like Komodo does, I don't think that Lc0 can even beat a 1400 rated human at knight odds (Komodo can give knight odds in blitz to GMs, but in slow games only to players aroun 2000). Even with somewhat higher win probs., say around 5% or so, Lc0 still blunders, though not as often. I've seen the same behavior from a couple different networks, I don't think the specific network is the problem. Perhaps it's a problem with rarely occurring material balances, such as being down two pieces or more.
Knight Odds of Lco were already played by 2900 blitz rated players like Daniel Nardoitsky and other GMs at blitz time control. Lc0 won them comfortably. Beat them with 70-80% score with knight odd.
please review the games on his account by yourself.
Thanks, but I couldn't find either the games or the results of the games from that link, but I saw a reference to a "modified version of Leela" for the games. Can you give a link to the actual results or else summarize them? Something isn't right here; the two recent networks I downloaded (including the one with the highest estimated CCRL rating) both just gave away pieces when giving knight odds as if they were playing Losing chess; yet there is nothing wrong with my setup since they play well enough to beat me at f7 and two or even three move odds, about like Komodo does. My guess is that they found a way to make it play good chess despite the knight odds for this "modified version"; does anyone know anything about this?
lkaufman wrote: ↑Fri Feb 22, 2019 7:45 pm
By the way, I note that it is not feasible to play against Lc0 at knight odds, because it soon starts to blunder pieces almost randomly. At smaller handicaps like pawn f7 and move or two this doesn't happen, it plays very well. It's almost as if all win probabilities beyond 99% are the same.
Does it do this blundering from both sides (knight up and knight down) or only from knight up?
Both sides, but more noticeably when down a piece or more. Basically, if its winning prob. is below 2% or so, it just "resigns" by making random blunders. Even taking more time doesn't generally cure this. Although Lc0 can probably beat top GMs giving them pawn or maybe pawn and move handicap just like Komodo does, I don't think that Lc0 can even beat a 1400 rated human at knight odds (Komodo can give knight odds in blitz to GMs, but in slow games only to players aroun 2000). Even with somewhat higher win probs., say around 5% or so, Lc0 still blunders, though not as often. I've seen the same behavior from a couple different networks, I don't think the specific network is the problem. Perhaps it's a problem with rarely occurring material balances, such as being down two pieces or more.
Knight Odds of Lco were already played by 2900 blitz rated players like Daniel Nardoitsky and other GMs at blitz time control. Lc0 won them comfortably. Beat them with 70-80% score with knight odd.
please review the games on his account by yourself.
I just played the following game against the supposedly best net of Lc0, at 5' + 5", GPU 1060, knight (b1) odds. 1.Nf3 d5 2.e3 Nf6 3.d4 Bg4 4.Ne5??? and I took the queen. Using infinite analysis it still prefers 4.Ne5??? even after a full minute, so a faster GPU or longer TC wouldn't have helped. And after I capture the queen it doesn't even recapture but still thinks it has a 3.6% win prob!! But at two pawn and move odds (b7 and c7) I see no bad moves, it plays well enough to beat a top GM in blitz I think. Are you sure that these games were at full knight odds (there is a mention of two pawns or knight odds)? Something had to be different, Naroditsky would not lose to an engine that hangs its queen on move 4. Maybe some parameter needs special setting for knight odds?
Different nets react differently to unusual positions. The t30 nets used in this TCEC Season (like 32930) had a real allergy to double queen positions. The late t10 nets (11248 and 11258) were a little bit more robust in this regard. I believe most of the public odds games were played against t10 nets.
My recommendation would be to try 11248 or 11258 for these odds positions.
Fat Titz by Stockfish, the engine with the bodaciously big net. Remember: size matters. If you want to learn more about this engine just google for "Fat Titz".
You were quite correct, except for the "little bit more robust" comment; the difference is day and night. With the 11248 network the win probs are all much closer to 50%, so knight odds gets 13% instead of 1%! Of course 13% for knight odds is ridiculous, but the result is that it plays quite well giving knight odds, and I lost the first blitz game I tried with it. It seems that with this version the silly play and absurd relative win probs only show up when it is down about two knights or more or the equivalent, instead of just one knight, so for any games against strong human players this does solve the problem. I note that this version (well actually the other one you mentioned, 11258) is listed as having an estimated CCRL rating just around 25 elo below the top one, so for handicap play it is a clear choice, probably even for smaller handicaps like pawn and move.
This is a very good illustration of how bad it is to have the score by which an engine plays saturate for a forcible win, or put at zero for a theoretical draw. It works the other way as well: the engine will not only stop offering meaningful resistance in a lost position; it will also not try to convert wins. Training the network for expected distance to win would solve this problem.
Lc0 misevaluates the starting position of knight odds. Lc0 plays chess very well.
With a little more work, one could make a proof showing the evaluation of odds starting positions are poor indicators of a NN's ability to evaulate standard chess positions.
The poit is that if it cannot evaluate an odds start position, it will also not be able to evaluate other positions with a material imbalance. And this indeed shows: Leela type engines are already infamous for the excruciatingly slow way they convert wins.
