Smallnet (128x10) run1 progresses remarkably well

[Laskos]

While the run2 (30xxx bignet 256x20) of test30 almost completely stalls for already 900 nets, almost 25 million expensive games, and some $10,000 spent uselessly on electricity bills, the smallnet run1 (35xxx, 128x10 nets) makes remarkable progress. I think they had a LR drop, but with only some 3 million "cheap" (4 times faster) games, the strength achieved is remarkable, only some 150 Elo points weaker than bignet after 40+ million (expensive) games. Someone sent me the Lc0 v0.20-dev engine which uses the new architecture of the smallnets, and I checked at short time control. One thing, either this compile is buggy or it's a feature, the speed of Lc0 v0.20-dev with a smallnet accelerates on RTX 2070 in 2-3 seconds to some 90,000+ NPS, then stalls and even decreases, having large cache and apparently good settings. I anticipated that it can easily would have increased further than for only 2-3 seconds, to some maybe 120,000 - 130,000 NPS (4x faster than the bignet). But I don't know, maybe it's a feature of the new architecture.

First I saw that it's going well in Cutechess-Cli, but for illustration I show fast games against SF8 on 4 fast i7 cores in LittleBlitzer. Lc0 is on RTX 2070.

TC: 15s + 0.25s
Openings: gm2600.pgn to 12 plies.

Code: Select all

Games Completed = 100 of 100 (Avg game length = 61.417 sec)
Settings = Gauntlet/512MB/15000ms+250ms/M 2000cp for 5 moves, D 200 moves/EPD:F:\LittleBlitzer\gm2600_12plies.epd(27202)
Time = 6417 sec elapsed, 0 sec remaining
 1.  Lc0 v0.20.0-dev 35229          	37.0/100	14-40-46  	(L: m=19 t=0 i=0 a=21)	(D: r=28 i=11 f=7 s=0 a=0)	(tpm=439.2 d=9.10 nps=1061510)
 2.  Stockfish 8 64 BMI2         	63.0/100	40-14-46  	(L: m=0 t=0 i=0 a=14)	(D: r=28 i=11 f=7 s=0 a=0)	(tpm=429.3 d=26.26 nps=9257779)

So, the performance on RTX 2070 is about 3400 CCRL 40/4 Elo points. On GTX 1060, probably some 3000 CCRL 40/4 Elo points. And their shown self-Elo is also about 3000 Elo points. The progress is remarkable. I am curious if this smallnet 35xxx can reach, after several LR drops, the level of the best test10 bignets. And if they won't start playing silly with it, I hope for improvements soon.

[jp]

I hope tablebase rescoring was removed for this run1.

[Laskos]

I hope tablebase rescoring was removed for this run1.

I don't know, but at least they are doing test net learning on 4 times smaller net, not some silly 40+ million probably barely useful expensive games with bignet.

[yanquis1972]

I’ve assumed the bignet has extended its schedule markedly after publication of several A0 parameters that have been implemented into the training. It’s also only had one LR drop and provided they are aiming for SF10+ strength with it I don’t see the issue with expense.

[Laskos]

I’ve assumed the bignet has extended its schedule markedly after publication of several A0 parameters that have been implemented into the training. It’s also only had one LR drop and provided they are aiming for SF10+ strength with it I don’t see the issue with expense.

$10,000 for almost no Elo progress (well, aside 4500 self-Elo progress, lol), $20,000 for the whole run2, just electricity bill, at lowest. I am not even interested in that, but why performing such tests, which for the last 20+ million games, proved almost fruitless, with the bignet? Do it with either 4 or 16 times "cheaper" nets. Well, I am not of any knowledge of what they are doing, but starting with a simple toy model is an often used procedure for me. And they started well with their 64x6 nets.

[jkiliani]

What is called smallnet in this thread is called test35, and its only real purpose is to test whether the SE (Squeeze-Excitation) Neural network architecture is implemented correctly in Lc0 0.20 Dev. So far it looks like that is the case, and SE nets are significantly stronger at the same network size compared to Residual Neural nets (used in test10, test30 and all other previous runs).

After Lc0 0.20 is released, test35 will be retired in favour of test40, a SE net of the same dimensions as test10 and test30, i.e. 256x20. That run is widely expected to significantly improve on test10.

[Laskos]

What is called smallnet in this thread is called test35, and its only real purpose is to test whether the SE (Squeeze-Excitation) Neural network architecture is implemented correctly in Lc0 0.20 Dev. So far it looks like that is the case, and SE nets are significantly stronger at the same network size compared to Residual Neural nets (used in test10, test30 and all other previous runs).

After Lc0 0.20 is released, test35 will be retired in favour of test40, a SE net of the same dimensions as test10 and test30, i.e. 256x20. That run is widely expected to significantly improve on test10.

Again, test40 with bignet. And then the master bignet? Well, I have no any knowledge of what you are doing, I wish you good luck and a fast success with test40.

[jp]

After Lc0 0.20 is released, test35 will be retired in favour of test40, a SE net of the same dimensions as test10 and test30, i.e. 256x20. That run is widely expected to significantly improve on test10.

Again I hope TB rescoring will not be part of test40, now we know what A0's C(s) was.

[jkiliani]

Again, test40 with bignet. And then the master bignet? Well, I have no any knowledge of what you are doing, I wish you good luck and a fast success with test40.

It has been quite conclusively shown that larger nets are more powerful in the end, they just require more resources to train. test10 was highly successful and so far has still not been surpassed. test20 failed, and arguably you're right that resources were wasted on that regard, since the belief "higher CPUCT -> better" has been taken on faith from a publication without checking it with our own data first. On the other hand, test30 has been a quite successful testbed so far, for TB rescoring which works fine and many more parameter tests.

The combination of a good set of parameters picked from test30, and the validation of an improved neural net architecture from test35 is going to make test40 a virtually guaranteed success. Any decisions on "master bignets" will have to wait until test40 is far along at the least...

[jkiliani]

After Lc0 0.20 is released, test35 will be retired in favour of test40, a SE net of the same dimensions as test10 and test30, i.e. 256x20. That run is widely expected to significantly improve on test10.

Again I hope TB rescoring will not be part of test40, now we know what A0's C(s) was.

As far as I know it will be. Why shouldn't it? TB rescoring has been shown to actually help, unlikely a lot of proposed changes to Lc0...