Uri Blass wrote: ↑Sun May 27, 2018 2:00 am
Laskos wrote: ↑Sat May 26, 2018 11:16 pm
Albert Silver wrote: ↑Sat May 26, 2018 10:39 pm
Virtual loss bug should not affect tactical performance, and scale thinking time will have zero effect in tactical suites, since it deals solely with time management.
I didn't fiddle with scale thinking time, just took the CLOP result and common sense in time usage. I put it in the description of settings to show the difference from the default. Virtual loss bug is affecting a bit the tactical performance, although I don't even know what it stands for. I know only that it affects the game-play from a given position. Using this parameter, I improved the performance on the WAC200 suite by 2 points (and the results were pretty deterministic).
Meanwhile the 20 game match finished: 12 SF wins and 8 draws, about 3350 CCRL 40/4' Elo points performance of Lc0 NN342 with these settings. One standard deviation for this result is about 50 Elo points.
For comparison the CCRL got only 2651 elo for Lc0 w323 at 40/40
Note that I do not see that they tested it at 40/4 to see if it scales better than other engines.
I wonder what is the source of the difference in rating.
possible reasons for differences
1)342 is stronger than 323
2)better hardware for Leela that CCRL did not use
3)different setting that CCRL did not use.
4)different time control(40/40 and not 40/4)
http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323
I am not sure what hardware they used. Was LCZero running on GPU? If it was CPU v0.10 version, then the rating is pretty expected one. CPU version is MUCH weaker than a moderately strong GPU GTX 1060 (like mine) version, and I used cuDNN Lc0, which is maybe 150 Elo points stronger than the official GPU v0.10 version. Also the settings. I am discovering (the sample is still small) that what seemed I fiddled wrongly with , "virtual loss bug", is VERY important. When checking with it yesterday, I got a small improvement in tactical test-suites with a setting of 60.0 (the maximum is 100.0). With these overall settings
Code: Select all
Scale thinking time=2.8
Cpuct MCTS option=15
First Play Urgency Reduction=-0.30
Virtual loss bug=60.0
I got 12 wins of SF dev (4 cores) and 8 draws. I reverted to the default "virtual loss bug" of 0.0, and the result was disastrous: 19 SF dev wins and 1 draw. Seeing this, I put "virtual loss bug" at its maximum, 100.0, and as of now have an excellent result (very small sample) of 4 draws and 1 SF dev win. Will play for a total of 20 games with these settings:
Code: Select all
Scale thinking time=2.8
Cpuct MCTS option=15
First Play Urgency Reduction=-0.30
Virtual loss bug=100.0
If the result keeps staying even remotely close to this excellent 4 draws and 1 loss, then there are serious setting changes compared to default which can improve Lc0 play by hundreds of Elo points. I maximized the performance on tactical test-suites, which is much faster to do than in games, but I didn't expect that this "virtual loss bug" setting can be so important, as in test-suites it was only slightly improving the result. The games are at 5'+ 5'', openings are 3-mover balanced GM openings. In 4-5 hours I will report the result in 20 games. The samples are all small here, 20 games each, but these things are time consuming for a single user like me.
NN used was 342, which is probably stronger than NN323 used in CCRL, but this is sure not the most important factor here.