STS rating v13.1 for Lc0 0.21.2 with nodes = 1

[chrisw]

My suite is rough and unpolished.

Question about your suite, how long does it take (an estimation is also ok) to analyze your 200 positions at (say) depth=5 with a 256x20 NN on your (GPU) hardware.

Not sure about depth, I am testing at fixed time, but depth=5 is really low for current Lc0 on my GPU. The times are from 1s per position (about 15,000 nodes) to 120s per position (about 2,500,000 nodes).

Thanks Kai. That's more than 100 x faster I have right now. I think I am going to buy some flowers for my wife tomorrow.

Ed, NB that LCZero “node” counter actually counts playouts, not nodes. To convert to normal nps, multiply by the average line depth.

[jp]

Ed, NB that LCZero “node” counter actually counts playouts, not nodes. To convert to normal nps, multiply by the average line depth.

I wonder if that's how Alpha counts too.

[chrisw]

Ed, NB that LCZero “node” counter actually counts playouts, not nodes. To convert to normal nps, multiply by the average line depth.

I wonder if that's how Alpha counts too.

Allegedly so, but in both cases I am secondary or tertiary source, only repeating what someone else wrote.

[Max]

Aren't the 1500 positions from the Strategic Test Suite look promising to compare several lc0 networks without any search involved? Lets find out.

STS rating v13.1 for Lc0 0.21.2 with nodes = 1 or search depth = 1|4 for ab-engines

Code: Select all

STS(elo) 	net (nodes=1) | eng (depth=1|4)
-----------------------------------------------
1405		Hakkapeliitta TCEC v2 (d=1)
1449		K2 v.0.95 (d=1)
1473		Arasan 21.3 (d=1)
1519		Arasan 21.3 (d=4)
1540		11258-16x2-se
1555		Texel 1.07 (d=1)
1602		Vajolet 2.6.2 (d=1)
1690		11258-24x3-se  
1705		Stockfish 10 (d=1)
1754		Komodo 10 (d=1)
1767		Stockfish 10 (d=4)
1780		11258-32x4-se
1954		Hakkapeliitta TCEC v2 (d=4)
1993		11258-48x5-se 
1999		net54023
2052		K2 v.0.95 (d=4)
2077	new	net54057
2094		Texel 1.07 (d=4)
2105		Komodo 10 (d=4)	
2116		11258-64x6-se	
2233		11258-80x7-se	
2239		32930-112x9-se 
2287		11258-96x8-se  
2292		11258-104x9-se
2333		11258-112x9-se
2334		net52340	
2353		11258-128x10-se
2367		net53316
2475		net40x256_130
2516		net33000
2517	new	net40x256_150
2529		11258-256x12-se
2536		net42610
2544	new	net42647
2567		11258-200x20-se
2571		netT40.T8.610
2625		net11258

It's obvious, that networks without search are in quiet positions superior to ab-engines with minimal search. Imo STS delivers this and ranks the different networks quite nice .

[Max]

Update with more networks like Bad Gyal, Little Demon, the 49xxx, jhorthos 320x24 and others.

The nps values are from my cpu, after about one minute search from the start position.

Code: Select all

STS   weights file      nps (CPU)
================================
1411  54010              390 nps
1540  11258-16x2-se     7200 nps
1581  bad-gyal-1        3300 nps
1690  11258-24x3-se     5200 nps
1780  11258-32x4-se     3500 nps
1909  37080              450 nps
1990  bad-gyal-2        3300 nps
1993  11258-48x5-se     2500 nps
1999  54023              390 nps
2077  54057              390 nps
2091  bad-gyal-3        2200 nps
2116  11258-64x6-se     1650 nps
2148  35229              560 nps
2154  bad-gyal-4         530 nps
2167  54089              390 nps
2233  11258-80x7-se     1100 nps
2238  40323               45 nps
2239  32930-112x9-se     550 nps
2255  54109              390 nps
2275  badgyal-128x10     530 nps
2275  54205              390 nps
2285  54176              390 nps
2287  11258-96x8-se      850 nps
2287  54252              390 nps
2291  LD2                530 nps
2292  11258-104x9-se     600 nps
2303  54215              390 nps
2312  53200              390 nps
2323  40500               45 nps
2324  53092              390 nps
2329  54147              390 nps
2333  11258-112x9-se     570 nps
2334  30933               45 nps
2334  52340              390 nps
2339  52377              390 nps
2346  20879               60 nps
2353  11258-128x10-se    400 nps
2353  53234              390 nps
2353  bad-gyal-4a        530 nps
2363  53291              390 nps
2367  53316              390 nps
2425  21494               60 nps
2442  40870               45 nps
2475  40x256_130          23 nps
2475  256x16.J17-swa-190  95 nps
2476  40x256_119          23 nps
2482  21583               60 nps
2491  22200               60 nps
2495  AntiFish_1.9        45 nps
2505  42420               45 nps
2513  40x256_158          23 nps
2514  41771               45 nps
2515  40x256_26           23 nps
2515  42550               45 nps
2516  33000               45 nps
2517  40x256_150          23 nps
2519  42654               45 nps
2521  40x256_107          23 nps
2521  42717               45 nps
2525  42685               45 nps
2527  Leelenstein_7       45 nps
2529  11258-256x12-se    100 nps
2530  40x256_153          23 nps
2536  42610               45 nps
2538  42487               45 nps
2543  42591               45 nps
2545  42724               45 nps
2546  49912               45 nps
2548  42700               45 nps
2549  42372               45 nps
2549  192x16.T20-swa-380 180 nps
2551  40x256_58           23 nps
2554  49906               45 nps
2554  42721               45 nps
2557  42537               45 nps
2557  192x16.T20-swa-400 180 nps
2557  42771               45 nps
2560  192x16.J20-swa-440 180 nps
2563  42633               45 nps
2566  42100               45 nps
2567  11258-200x20-se     95 nps
2567  42741               45 nps
2571  256x20.T8-swa-610   54 nps
2571  T40.T8.610          45 nps
2574  49908               45 nps
2581  49914               45 nps
2585  49904               45 nps
2580  41800-boost-7000    65 nps
2604  320x24.T13-swa-310  32 nps
2615  320x24.T13-swa-360  32 nps
2618  320x24.J13-swa-390  32 nps
2623  11141               45 nps
2625  11258               45 nps
2631  320x24.T13-swa-330  32 nps

[peter]

Update with more networks like Bad Gyal, Little Demon, the 49xxx, jhorthos 320x24 and others.

Thanks for continuing to try and show!

[Max]

Update with more networks like Bad Gyal, Little Demon, the 49xxx, jhorthos 320x24 and others.

Thanks for continuing to try and show!

Thanks Peter, here we go.

Now, as network run 42x finished and 55x was canceld, let's look back into the short Lc0 history.

STS-rankings (node = 1) of the latest - not necessarily best - policy networks.

Code: Select all

STS   latest network 	   filters(x)blocks
--------------------------------------------------
2656  11262			256x20
2597  49924			256x20 <-activ (run 3)
2584  320x24.T13-swa-410	320x24   
2581  42850			256x20
2571  T40.T8.610 	 	256x20   TCEC 15 winner
2569  T40B.2-106 	 	256x20 
2546  192x16.J20-swa-460	192x16   
2524  33005			256x20
2513  40x256_158		256x40   
2479  22202			256x20
2395  bad-gyal-5		128x10 
2367  53316			128x10
2339  52377			128x10
2315  51458			128x10
2307  36092			128x10
2291  LD2			128x10 
2287  54252			128x10
2270  56200			128x10 <-activ (run 2)
2239  50783			128x10
2213  60160 			320x24 <-activ (run 1)
2134  55109 (canceled)		128x10
1894  37082 (canceled)		128x10  

Note, that 60160, 56200 and 49924 are active and evolving!

[mesilikas]

How can i force lc0 to play with 1 node per move?

[mclane]

Lc0 with few NPS plays so impressive that computer chess makes a lot of fun again. After all these years it still can be a super hobby.

[Max]

How can i force lc0 to play with 1 node per move?

In Nibbler select Engine/Node limit/1

Node 1.png