Leela Output

[Stephen Ham]

Hello All,

I don't have Leela but hope to get it after a hardware upgrade. So, I'm trying to get educated now.

In the display found under Nibbler commentary here: http://blog.lczero.org/2019/
what does the 75.4% next to e5 mean? Is that a probability of victory or a probability of being played? If it's the former, then what's the probability of loss and drawing? Since three different results are possible, showing the percentage for only one result variable seems silly and incomplete.

In that display, all other move options have progressively lower percentages. But I've seen some displays from a friend's Leela where the percentages are only generally lower, as some lower moves actually have higher percentage values. Why is that?

Thanks in advance.

All the best,
-Steve-

[zullil]

Hello All,

I don't have Leela but hope to get it after a hardware upgrade. So, I'm trying to get educated now.

In the display found under Nibbler commentary here: http://blog.lczero.org/2019/
what does the 75.4% next to e5 mean? Is that a probability of victory or a probability of being played? If it's the former, then what's the probability of loss and drawing? Since three different results are possible, showing the percentage for only one result variable seems silly and incomplete.

In that display, all other move options have progressively lower percentages. But I've seen some displays from a friend's Leela where the percentages are only generally lower, as some lower moves actually have higher percentage values. Why is that?

Thanks in advance.

All the best,
-Steve-

The 75.4% is the evaluation of the move, where 100% is interpreted as certainly winning and 0% as certainly losing (for the side that has the move). The value P = 3.28% in the list that follows the PV is the neural net's initial probability that this move is the "correct" move to play in this position, not altered by any searching of the position. So, in the Nibbler output you linked to, it seems that Lc0's initial "instinct" is that Nc3 is most likely the correct move in this position, with about a 27% chance of being best. After searching, that move is found to be weaker than e5.

[Stephen Ham]

Thank you Louis,

Still, using a percentile value is confusing. I suppose 50% means that a draw is forecast. Or does it mean that a 50% chance of victory is predicted? If it's the latter, then that's quite different from the former. What about the chance of loss...is that also 50%? If so, that leaves no chance for a draw. Again, the issue is that a percentile is given for only one of three variable options, leaving the percentile of the other two unstated.

So, in an ultra-sharp position where there's an equal chance of victory or loss, will the result be 50%? If so, then how does that differ from the exact opposite where there's a 100% chance of a draw, as that's also 50%. If I'm correct, then that's my gripe with only stating of the value for one of three possible outcomes.

I suppose I should use the perspective of what you wrote, Louis. But then, that means that 75.4% really is a misnomer as it's not a percentage at all. So perhaps the % sign should be removed as only the numeric value means anything. Instead, since the numeric value is higher than 50 but below 100, then it just means a better than average chance of victory.

I also suppose that once it's Black's move, the value flip-flops to 24.6%. Still, that's only providing a value for one of three outcomes, so this seems a very poor evaluation. Or am I missing something?

Since Leela is using MCTS, why not instead show the probability percentage for all three variables based upon the games played? Such a result might then be: 65.3 - 21.8 - 12.9, where 65.3 is victories, 21.8 represents losses, and 12.9 represents draws. This also has the benefit of telling the human how dynamic Leela thinks the position is. This fixes the problem of getting a result similar to 50%, as that doesn't tell us that it's 0 - 100 - 0 where it's a forced draw, or 50 - 50 - 0, where it's totally dynamic.

Sincerely,
-Steve-

[Hamster]

Stephen makes a very good point.

why not instead show the probability percentage for all three variables based upon the games played
Would that even be possible to display or would be whole structure of the NN need to be changed?

[zullil]

Stephen makes a very good point.

why not instead show the probability percentage for all three variables based upon the games played
Would that even be possible to display or would be whole structure of the NN need to be changed?

Lc0 doesn't play games (i.e., do complete rollouts to terminal nodes). It's not doing MCTS. And it seems that the raw NN output gives an expected value, V, for the position/state. So you get V = 1 p(win) + 0 p(draw) + (-1) p(loss) = p(win) - p(loss), in the interval [-1,1]. It seems that this single quantity serves as a proxy for the three individual probabilities.

I should add that I'm just beginning to learn about this stuff, so what I've just offered might be wrong. See https://github.com/LeelaChessZero/lc0/w ... Chess-Zero for more.

[Stephen Ham]

Hi Louis and Hamster,

Gentlemen, thanks so much for your valued input. I too have a lot to learn about Leela. For example, I thought it played MCTS in order to generate "learning" from the results of those games. I see from your link, Louis, that it instead uses PUCT. But MCTS is data from actual games (i.e. reality), while PUCT is just an estimate. So, how close to reality is that estimate? I've already seen Leela perform a 180-degree change in predicting a best move in one of my ICCF games. That gives me little confidence.

I agree that a score near 1 means a victory, 0 means a draw, while a negative score means a greater chance of losing. However, that doesn't jibe with the Nibbler GUI that I referenced. There, it's not centi-pawn based, but instead percentages. And those percentages don't match the 1, 0, -1 format outside of Nibbler. Instead, it only reports one of three variables. This is akin to the George Carlin sportscaster skit: "And now for a partial score, Minnesota 27!" In short, it doesn't provide enough information, like the other team's score. In this case, it's silent on the other two variables.

But, that's just my 2 cents and I have a lot to learn.

All the very best,
-Steve-

[zullil]

Steve,

You can have Nibbler show values for many variables. Analysis drop-down and then Infobox stats.

Louis

[jp]

I've already seen Leela perform a 180-degree change in predicting a best move in one of my ICCF games. That gives me little confidence.

What do you mean by 180-degree change? Is this an example you can show us?

[Ovyron]

What do you mean by 180-degree change?

I guess it's when Leela really loves a move, it gives a very high score greatly over the rest, say, best move 1.50 and second best 0.60. When you play the moves and investigate the variations, specially with help from another engine like Stockfish, it turns out the variation is really bad, and Leela now shows a score <0.60 for the move. If it's really, really bad it may even go into the negative and think it's losing.

Backward investigation would reveal that there's no saving of this variation to the root, the move is bad and Leela's output was nonsensical. Checking it was just a waste of time, so there's "little confidence" of Leela's eval if it gives great scores to bad moves.

The same happens with great moves that Leela scored around the rest, say, Stockfish announces a winning move over the rest, Leela at MultiPV has it at top 8, if you play the variations into Leela she bends down and accepts she was missing the greatness.

The certain thing is that you'd be out of your mind to analyze with Leela alone, it needs a tactically aware engine that checks its sanity. For comparison, analyzing with Stockfish alone is fine.

Disclaimer: I'm talking about snail CPU Leela and low depth, but I have yet to see someone claim that this stops happening at some point. Those blind spots of Leela don't gain sight with more depth.

[phhnguyen]

Allow me to add an extra question (perhaps it is answered somewhere but I cannot find out).

Can someone explain what N, P, Q, D, U, V are in lc0's move stats. What is the first number (e.g. 230 right after the move b2b3)?
What is the condition for a move to be the best one?

Thanks.

Code: Select all

info string b2b3  (230 ) N:      75 (+ 3) (P:  3.35%) (Q: -0.00201) (D:  0.193) (U: 0.11151) (Q+U:  0.10950) (V:  0.0143) 
info string h2h3  (400 ) N:      76 (+ 1) (P:  3.30%) (Q: -0.00108) (D:  0.206) (U: 0.11124) (Q+U:  0.11017) (V: -0.0038) 
info string b1c3  (36  ) N:      77 (+ 8) (P:  3.53%) (Q:  0.00258) (D:  0.194) (U: 0.10784) (Q+U:  0.11043) (V: -0.0135)