Why Lc0 eval (in cp) is asymmetric against AB engines?

[Laskos]

With the talk about supervised learning escalating, I would like to a point where the reinforcement learning seems to fail pretty badly -- converting endgames. I presented the middlegame graph, where the things often linger into endgames (aside issues with tactics in both middlegames and endgames), and the graph of Lc0 was shifted with regard to axis origins. From the shapes of these plots, one can suspect that the shift in the middlegames is partly due to poor endgame evaluation and performance. Here the supervised learning would help a lot, and learning from Stockfish or Komodo (enabled with 6 men Syzygy) games (without adjudications) from endgame starting positions would help greatly (in both eval and tactics). Simply, hand-crafted endgame eval terms and TBs are very helpful in traditional engines, and self-play with NN isn't that productive. Here are the plots of the performances versus eval for three cases (games of equaled in strength Leela against Komodo):

[Laskos]

New shape of performance versus eval with new hardware (RTX 2070 GPU), new test30 net, time control 15s + 0.25s. In these conditions, Lc0 on RTX 2070 and SF10 on 4 strong i7 cores are equally matched in strength:

Score of lc0_v201_32721 vs SF10: 49 - 49 - 102 [0.500] 200
Elo difference: 0.00 +/- 33.76
Finished match

And the performance versus eval in opening - early midgame looks like this:




Lc0, to smaller degree, still has difficulty converting even large evals it shows. It exhibits a sort of Contempt of some 60cp. SF10 has an in-built Contempt of about 20cp. This difficulty converting of Lc0 is probably again related to bad endgames and occasional blunders.
But the negative evals of Lc0 show almost exact performance as SF10, it behaves regularly when seeing itself in disadvantage.

[megamau]

Lc0, to smaller degree, still has difficulty converting even large evals it shows. It exhibits a sort of Contempt of some 60cp. SF10 has an in-built Contempt of about 20cp. This difficulty converting of Lc0 is probably again related to bad endgames and occasional blunders.
But the negative evals of Lc0 show almost exact performance as SF10, it behaves regularly when seeing itself in disadvantage.

What you see as "contempt" is actually "failure to convert". As you balance the overall elo, LC0 will in general be stronger and obtain better positions, but fail to convert or blunder them and lose. The overall result is a wash (by design of the experiment).
The matching in the negative eval is because the AB engine will not blunder the better positions.

[yanquis1972]

I think that’s the major factor, especially at this TC, but “comtempt” (relatively dramatic cp scores) also plays a part...when possible I exclusively use win% with leela now because after a short time you can understand her evaluation in traditional “+-, +/-, +/=“, etc terms. If the cp value is intended to mimic a/b evals it does so poorly....

What I don’t immediately understand is why leelas negative evals plot so well with stockfishes, but I’d bet that again precise conversion is the primary factor.

[Laskos]

I think that’s the major factor, especially at this TC, but “comtempt” (relatively dramatic cp scores) also plays a part...when possible I exclusively use win% with leela now because after a short time you can understand her evaluation in traditional “+-, +/-, +/=“, etc terms. If the cp value is intended to mimic a/b evals it does so poorly....

What I don’t immediately understand is why leelas negative evals plot so well with stockfishes, but I’d bet that again precise conversion is the primary factor.

Well, it is still contempt of some 60cp. Say, against an equal or stronger regular engine, if Leela's eval is +40cp, it should rather forcefully draw if it can, than hoping to get a better result, as +40cp has some 48% performance, or -14 Elo points average outcome (against an equal regular engine, even worse against a superior engine).
It's a good thing against significantly weaker regular engines. All in all, acts like regular Contempt.

I am not sure why for negative evals, Leela's performance shape is so similar to that of SF10. It seems that what Leela considers as an inferior position is seen by SF10 also as inferior, of similar (negative) cp value.

Sure, for those high (positive) Leela's eval values, the main reason is the failure to convert, including blunders.

I am not using win% output with Leela, what does it say for 300cp-400cp values in early midgame? Here it is around 80-85% performance, against an equal in strength SF10. Do they use some general logistic conversion or it's more complicated and dependent on each position?

[chrisw]

I think that’s the major factor, especially at this TC, but “comtempt” (relatively dramatic cp scores) also plays a part...when possible I exclusively use win% with leela now because after a short time you can understand her evaluation in traditional “+-, +/-, +/=“, etc terms. If the cp value is intended to mimic a/b evals it does so poorly....

What I don’t immediately understand is why leelas negative evals plot so well with stockfishes, but I’d bet that again precise conversion is the primary factor.

Well, it is still contempt of some 60cp. Say, against an equal or stronger regular engine, if Leela's eval is +40cp, it should rather forcefully draw if it can, than hoping to get a better result, as +40cp has some 48% performance, or -14 Elo points average outcome (against an equal regular engine, even worse against a superior engine).

I am not sure why for negative evals, Leela's performance shape is so similar to that of SF10. It seems that what Leela considers as an inferior position is seen by SF10 also as inferior, of similar (negative) cp value.

Sure, for those high (positive) Leela's eval values, the main reason is failure to convert, including blunders.

I am not using win% output with Leela, what does it say for 300cp-400cp values in early midgame? Here it is around 80-85% performance, against an equal in strength SF10. Do they use some general logistic conversion or it's more complicated and dependent on each position?

I think this “asymmetry” is just an effect of the way the search finds an evaluation. The explore exploit algorithm favours searching “good” moves. The search at any node doesn’t go full width. The backed up evaluation averages what has been looked at. So the score tends to be an averaging over the better lines in the tree. Hence positively asymmetric.

[grahamj]

I have found a quite different type of assymmetry.

I use board positions taken randomly from various sets of games, evaluate them with both SF and LC0 using python-chess, convert to prob(win), and graph. I gave them 150ms each. The graph uses 1000 positions from 1000 games from LC0's Test20 self-play games. These include a lot of positions with and obvious winner, so most points are clustered near (0,0) and (1,1).

For LC0 I use its prob(win) value [though I'm actually calculating it by reversing the prob(win) -> cp mapping] and for SF I use (math.atan(int(text)/100.0) / (math.pi/2) + 1) / 2. This makes most values roughly follow a straight line from (0,0) to (1,1) (though there is a characteristic waviness that you can see if you squint). However there are a few board postions where LC0 and SF strongly disagree, with LC0 always being pessimistic.