LCZero: Progress and Scaling. Relation to CCRL Elo

[CMCanavessi]

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323

Leela Chess 0.10 64-bit w323 #110‑111 (2651 +19 −19)

2100 elo engine!!!!

[Albert Silver]

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323

Leela Chess 0.10 64-bit w323 #110‑111 (2651 +19 −19)

2100 elo engine!!!!

It is. Around 2100. You are not really going to nitpick over a piddly 550 Elo are you? Mind you, I think that was just using the CPU version too.

[Werewolf]

Has Leela hit a plateau? There doesn't seem to be progress anymore.

Also any news on the bigger network?

[Milos]

Has Leela hit a plateau? There doesn't seem to be progress anymore.

Also any news on the bigger network?

It hit plateau already early with 15x192 network around 237. So far this was mostly regression and very small improvement (since network 320 basically).
20x256 won't bring anything since net gain is smaller than computation slowdown.

[yanquis1972]

Has Leela hit a plateau? There doesn't seem to be progress anymore.

Also any news on the bigger network?

don't know any details, but superficially it looks like self-play elo has plateaued but benchmarks against SF show progress continuing. the latest net looks like the strongest by that criteria & seems like it'd be a good one for thorough testing at longer time controls. think the CCRL rating on a GTX 1060 would surprise a lot of people one way or another if it was tested properly.

[Leto]

According to this chart it seems Leela is now almost as strong as Rodent 3, so it seems to be making progress:

[Werewolf]

To me it looks like stalling. I wonder if this can be resurrected.

[CMCanavessi]

To me it looks like stalling. I wonder if this can be resurrected.

How can you "resurrect" something that's not dead?

[yanquis1972]

yeah idk, maybe look at it harder. it's not gonna be 50 elo/day or whatever, that part of the curve has sailed. it's clearly not stalled. 321-351 is tiny a blip & there's measurable growth.

[Laskos]

To me it looks like stalling. I wonder if this can be resurrected.

From NN342 to NN352 it indeed seems to be stalling against an A/B engine Arasan 20.5. Here is the results of gauntlet:

Code: Select all

Games Completed = 1600 of 1600 (Avg game length = 13.001 sec)
Settings = Gauntlet/64MB/100ms per move/M 9000cp for 30 moves, D 150 moves/EPD:C:\LittleBlitzer\3moves_GM_04.epd(817)
Time = 29352 sec elapsed, 0 sec remaining
 1.  Arasan 20.5              	1062.5/1600	872-347-381  	(L: m=347 t=0 i=0 a=0)	(D: r=297 i=55 f=7 s=9 a=13)	(tpm=107.6 d=16.21 nps=1682392)
 2.  Lc0 NN342                	269.0/800	172-434-194  	(L: m=429 t=0 i=5 a=0)	(D: r=149 i=26 f=4 s=6 a=9)	(tpm=108.8 d=1.25 nps=1871)
 3.  Lc0 NN352                	268.5/800	175-438-187  	(L: m=436 t=0 i=2 a=0)	(D: r=148 i=29 f=3 s=3 a=4)	(tpm=108.6 d=1.22 nps=1700)

Lc0 is the cuDNN version with default settings, I am tired of fiddling with its parameters, with performances varying at time controls, and it seems anyway there are some serious bugs.

To see the scaling, I played matches of 800 games each at 0.1s/move, 0.4s/move, 1.6s/move, in total 4 doublings, of Lc0 NN352 on GTX 1060 6GB against Arasan 20.5 on one core (3061 CCRL 40/4' Elo points on one core). In fact the time used per move was taken from what LittleBlitzer reports, and it is not exactly what I set there. The CCRL performance of Lc0 cuDNN ID352 at three time controls are:

0.109 s/move --> 2943 Elo points
0.344 s/move --> 3015 Elo points
1.215 s/move --> 3091 Elo points

The fitted function as CCRL Elo performance as function of time control (scaling) of Lc0 with time per move is

Code: Select all

CCRL Elo of cuDNN Lc0 = 3079.51 + 61.3627 * ln(seconds per move) 
on GTX 1060. 

I also assumed that Arasan 20.5 scales as a standard A/B engine. The fitted intuitive function of scaling with two parameters correlates 0.99999 with the three data points, and it is not an overfit. Here is the plot:



So, at LTC, cuDNN Lc0 ID352 is above 3300 CCRL Elo points by this extrapolation, which confirms my earlier results, when playing games against Stockfish (but with very weak accumulated statistic). Here the weak point is the extrapolation itself, but I prefer that. For a top GTX 1080 Ti GPU, add some 70 Elo points to these results, maybe more.