Why Lc0 eval (in cp) is asymmetric against AB engines?
Posted: Wed Jul 25, 2018 1:21 pm
I guess that they convert winning probabilities to evals in CPs, but do they convert them asymmetrically? I mean, if equal in strength opponent, the performance vs eval graph would better be symmetric with regard to eval=0 and performance=50% point (point of symmetry).
I managed to choose time controls such as Komodo 12.1.1 on one i7 3.8 GHz core is almost exactly equal in strength to Lc0 ID10166 on GTX 1060.
The result in 1000 games match at fast time control is:
Score of Komodo 12.1.1 vs lc0_v16 ID10166: 399 - 406 - 195 [0.496] 1000
Elo difference: -2.43 +/- 19.31
So, at these chosen time controls, they are practically equal in strength. But while Komodo shows an approximate symmetry about the point of symmetry, Lc0 does not. Also, high eval, say of 300cp to 400cp, does not mean more than 65-70% performance against Komodo (of equal strength), which is unusual. Moreover, eval of 0.00 of Lc0 in middlegames means a performance of only some 32% against an equal in strength AB engine like Komodo. 50% is achieved by Lc0 only at eval of +140cp.
Here is the plot of performance of engines versus eval shown by them in middlegames (moves 10 to 30):
Also, for curiosity, I plotted the length of the won games for Komodo (median of 46 moves) and Lc0 (median of 67 moves). No adjudication was used. Lc0 seems to linger more to mate the opponent.
I managed to choose time controls such as Komodo 12.1.1 on one i7 3.8 GHz core is almost exactly equal in strength to Lc0 ID10166 on GTX 1060.
The result in 1000 games match at fast time control is:
Score of Komodo 12.1.1 vs lc0_v16 ID10166: 399 - 406 - 195 [0.496] 1000
Elo difference: -2.43 +/- 19.31
So, at these chosen time controls, they are practically equal in strength. But while Komodo shows an approximate symmetry about the point of symmetry, Lc0 does not. Also, high eval, say of 300cp to 400cp, does not mean more than 65-70% performance against Komodo (of equal strength), which is unusual. Moreover, eval of 0.00 of Lc0 in middlegames means a performance of only some 32% against an equal in strength AB engine like Komodo. 50% is achieved by Lc0 only at eval of +140cp.
Here is the plot of performance of engines versus eval shown by them in middlegames (moves 10 to 30):
Also, for curiosity, I plotted the length of the won games for Komodo (median of 46 moves) and Lc0 (median of 67 moves). No adjudication was used. Lc0 seems to linger more to mate the opponent.