20x256 Leela seemed quite strong

[Laskos]

Virtual loss bug should not affect tactical performance, and scale thinking time will have zero effect in tactical suites, since it deals solely with time management.

I didn't fiddle with scale thinking time, just took the CLOP result and common sense in time usage. I put it in the description of settings to show the difference from the default. Virtual loss bug is affecting a bit the tactical performance, although I don't even know what it stands for. I know only that it affects the game-play from a given position. Using this parameter, I improved the performance on the WAC200 suite by 2 points (and the results were pretty deterministic).
Meanwhile the 20 game match finished: 12 SF wins and 8 draws, about 3350 CCRL 40/4' Elo points performance of Lc0 NN342 with these settings. One standard deviation for this result is about 50 Elo points.

[Laskos]

Virtual loss bug should not affect tactical performance, and scale thinking time will have zero effect in tactical suites, since it deals solely with time management.

I didn't fiddle with scale thinking time, just took the CLOP result and common sense in time usage. I put it in the description of settings to show the difference from the default. Virtual loss bug is affecting a bit the tactical performance, although I don't even know what it stands for. I know only that it affects the game-play from a given position. Using this parameter, I improved the performance on the WAC200 suite by 2 points (and the results were pretty deterministic).
Meanwhile the 20 game match finished: 12 SF wins and 8 draws, about 3350 CCRL 40/4' Elo points performance of Lc0 NN342 with these settings. One standard deviation for this result is about 50 Elo points.

Anyway, as the impact of the virtual loss bug on tactics is small and it may affect other aspects of the game-play heavily, I reverted it to default, and will have overnight another 20 game match against SF dev on 4 cores. So, the only tuned values were CPUCT and FPUR (and they are pretty extreme to get a maximum result on WAC and ECM).

[Uri Blass]

Virtual loss bug should not affect tactical performance, and scale thinking time will have zero effect in tactical suites, since it deals solely with time management.

I didn't fiddle with scale thinking time, just took the CLOP result and common sense in time usage. I put it in the description of settings to show the difference from the default. Virtual loss bug is affecting a bit the tactical performance, although I don't even know what it stands for. I know only that it affects the game-play from a given position. Using this parameter, I improved the performance on the WAC200 suite by 2 points (and the results were pretty deterministic).
Meanwhile the 20 game match finished: 12 SF wins and 8 draws, about 3350 CCRL 40/4' Elo points performance of Lc0 NN342 with these settings. One standard deviation for this result is about 50 Elo points.

For comparison the CCRL got only 2651 elo for Lc0 w323 at 40/40
Note that I do not see that they tested it at 40/4 to see if it scales better than other engines.

I wonder what is the source of the difference in rating.
possible reasons for differences
1)342 is stronger than 323
2)better hardware for Leela that CCRL did not use
3)different setting that CCRL did not use.
4)different time control(40/40 and not 40/4)

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323

[yanquis1972]

since that rating seems extremely low, i'm assuming the answer is all of the above (even if the hardware was = CCRL probably wasn't using CUDA).

i'd guess hardware-time control-software in that order w/ settings as the wild card (think kai said they should perform worse than his previous settings but i don't know it may still be better than default)

[main line]

First time in the history of playchess, there was playing lc0 Titan V. Here are some games :

1. https://lichess.org/eqoubc3y 2. https://lichess.org/lN0SWBKo 3. https://lichess.org/IumrpN64/black
Lc0 played without openbook but opponents used very strong book.

[Albert Silver]

First time in the history of playchess, there was playing lc0 Titan V. Here are some games :

1. https://lichess.org/eqoubc3y 2. https://lichess.org/lN0SWBKo 3. https://lichess.org/IumrpN64/black
Lc0 played without openbook but opponents used very strong book.

He is known on the official Discord. I asked him to run a match against SF9 using one of my suites. I'll be sure to share if and when he finishes and shares.

[Laskos]

Virtual loss bug should not affect tactical performance, and scale thinking time will have zero effect in tactical suites, since it deals solely with time management.

I didn't fiddle with scale thinking time, just took the CLOP result and common sense in time usage. I put it in the description of settings to show the difference from the default. Virtual loss bug is affecting a bit the tactical performance, although I don't even know what it stands for. I know only that it affects the game-play from a given position. Using this parameter, I improved the performance on the WAC200 suite by 2 points (and the results were pretty deterministic).
Meanwhile the 20 game match finished: 12 SF wins and 8 draws, about 3350 CCRL 40/4' Elo points performance of Lc0 NN342 with these settings. One standard deviation for this result is about 50 Elo points.

For comparison the CCRL got only 2651 elo for Lc0 w323 at 40/40
Note that I do not see that they tested it at 40/4 to see if it scales better than other engines.

I wonder what is the source of the difference in rating.
possible reasons for differences
1)342 is stronger than 323
2)better hardware for Leela that CCRL did not use
3)different setting that CCRL did not use.
4)different time control(40/40 and not 40/4)

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323

I am not sure what hardware they used. Was LCZero running on GPU? If it was CPU v0.10 version, then the rating is pretty expected one. CPU version is MUCH weaker than a moderately strong GPU GTX 1060 (like mine) version, and I used cuDNN Lc0, which is maybe 150 Elo points stronger than the official GPU v0.10 version. Also the settings. I am discovering (the sample is still small) that what seemed I fiddled wrongly with , "virtual loss bug", is VERY important. When checking with it yesterday, I got a small improvement in tactical test-suites with a setting of 60.0 (the maximum is 100.0). With these overall settings

Code: Select all

Scale thinking time=2.8
Cpuct MCTS option=15
First Play Urgency Reduction=-0.30
Virtual loss bug=60.0

I got 12 wins of SF dev (4 cores) and 8 draws. I reverted to the default "virtual loss bug" of 0.0, and the result was disastrous: 19 SF dev wins and 1 draw. Seeing this, I put "virtual loss bug" at its maximum, 100.0, and as of now have an excellent result (very small sample) of 4 draws and 1 SF dev win. Will play for a total of 20 games with these settings:

Code: Select all

Scale thinking time=2.8
Cpuct MCTS option=15
First Play Urgency Reduction=-0.30
Virtual loss bug=100.0

If the result keeps staying even remotely close to this excellent 4 draws and 1 loss, then there are serious setting changes compared to default which can improve Lc0 play by hundreds of Elo points. I maximized the performance on tactical test-suites, which is much faster to do than in games, but I didn't expect that this "virtual loss bug" setting can be so important, as in test-suites it was only slightly improving the result. The games are at 5'+ 5'', openings are 3-mover balanced GM openings. In 4-5 hours I will report the result in 20 games. The samples are all small here, 20 games each, but these things are time consuming for a single user like me.

NN used was 342, which is probably stronger than NN323 used in CCRL, but this is sure not the most important factor here.

[Guenther]

I wonder what is the source of the difference in rating.
possible reasons for differences

IIRC no cuda no gpu at all, just cpu which is the slowest possible approach.

[jkiliani]

I didn't fiddle with scale thinking time, just took the CLOP result and common sense in time usage. I put it in the description of settings to show the difference from the default. Virtual loss bug is affecting a bit the tactical performance, although I don't even know what it stands for. I know only that it affects the game-play from a given position. Using this parameter, I improved the performance on the WAC200 suite by 2 points (and the results were pretty deterministic).
Meanwhile the 20 game match finished: 12 SF wins and 8 draws, about 3350 CCRL 40/4' Elo points performance of Lc0 NN342 with these settings. One standard deviation for this result is about 50 Elo points.

For comparison the CCRL got only 2651 elo for Lc0 w323 at 40/40
Note that I do not see that they tested it at 40/4 to see if it scales better than other engines.

I wonder what is the source of the difference in rating.
possible reasons for differences
1)342 is stronger than 323
2)better hardware for Leela that CCRL did not use
3)different setting that CCRL did not use.
4)different time control(40/40 and not 40/4)

http://www.computerchess.org.uk/ccrl/40 ... 4-bit_w323

I am not sure what hardware they used. Was LCZero running on GPU? If it was CPU v0.10 version, then the rating is pretty expected one. CPU version is MUCH weaker than a moderately strong GPU GTX 1060 (like mine) version, and I used cuDNN Lc0, which is maybe 150 Elo points stronger than the official GPU v0.10 version. Also the settings. I am discovering (the sample is still small) that what seemed I fiddled wrongly with , "virtual loss bug", is VERY important. When checking with it yesterday, I got a small improvement in tactical test-suites with a setting of 60.0 (the maximum is 100.0). With these overall settings

Code: Select all

Scale thinking time=2.8
Cpuct MCTS option=15
First Play Urgency Reduction=-0.30
Virtual loss bug=60.0

I got 12 wins of SF dev (4 cores) and 8 draws. I reverted to the default "virtual loss bug" of 0.0, and the result was disastrous: 19 SF dev wins and 1 draw. Seeing this, I put "virtual loss bug" at its maximum, 100.0, and as of now have an excellent result (very small sample) of 4 draws and 1 SF dev win. Will play for a total of 20 games with these settings:

Code: Select all

Scale thinking time=2.8
Cpuct MCTS option=15
First Play Urgency Reduction=-0.30
Virtual loss bug=100.0

If the result keeps staying even remotely close to this excellent 4 draws and 1 loss, then there are serious setting changes compared to default which can improve Lc0 play by hundreds of Elo points. I maximized the performance on tactical test-suites, which is much faster to do than in games, but I didn't expect that this "virtual loss bug" setting can be so important, as in test-suites it was only slightly improving the result. The games are at 5'+ 5'', openings are 3-mover balanced GM openings. In 4-5 hours I will report the result in 20 games. The samples are all small here, 20 games each, but these things are time consuming for a single user like me.

NN used was 342, which is probably stronger than NN323 used in CCRL, but this is sure not the most important factor here.

Virtual loss bug is a (different) type of FPU reduction, one that prunes little at the root but a lot at the leaves. Kind of strange to combine negative regular FPU reduction with positive FPU reduction by virtual loss, but if it works for you...

[jp]

Virtual loss bug is a (different) type of FPU reduction, one that prunes little at the root but a lot at the leaves. Kind of strange to combine negative regular FPU reduction with positive FPU reduction by virtual loss, but if it works for you...

So are you saying FPU prunes a lot at the root & little at the leaves? And the combination used is kind of wide but not deep?