LcZero and STS

[Rebel]

I use the MEA tool.

http://rebel13.nl/rebel13/mrl.html

Direct download - http://rebel13.nl/dl.html?file=dl/mea.zip

[Joost Buijs]

Interesting results!

Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.

390 looks obviously better, see:

http://rebel13.nl/zero1.html
http://rebel13.nl/zero2.html
http://rebel13.nl/zero3.html

Running 395 now.

Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

I have my doubts about the accuracy of STS, most of the 1500 positions were examined with engines from several years back and it is reasonable to assume that insight has changed since then. For instance Fizbo2 scores a meagre 9027 points on this test, but in fact Fizbo2 is stronger than several other engines scoring 3000 points higher, this means that there is clearly something lacking.

[Rebel]

I have my doubts about the accuracy of STS, most of the 1500 positions were examined with engines from several years back and it is reasonable to assume that insight has changed since then. For instance Fizbo2 scores a meagre 9027 points on this test, but in fact Fizbo2 is stronger than several other engines scoring 3000 points higher, this means that there is clearly something lacking.

I should (will) remove Fizbo from the lists, it doesn't obey the movetime command and starts moving instantly after only a couple of iterations.

[Rebel]

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.

Odd indeed.....

[CMCanavessi]

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.

Odd indeed.....

That's completely expected. lc0 is faster, so it produces more nodes in the same time. Hence, it's stronger.

[Rebel]

Added a tool....

With this tool you perhaps with reasonable certainty quickly can decide if a new LC-zero network weight file performs better or not. Currently version 323 has a CCRL rating of 2495 and version 390 (click on one of the 4 buttons in the upper right corner) clearly shows an improvement.

Operation
• Download LC-ZERO CPU version.
• Unzip anywhere.
• Open the folder and double-click run_mea-1s
• MEA will run the STS 1500 positions testsuite at 1 second with version 323 currently rated 2495 ELO in the CCRL 40/4 rating list and thereafter version 390. Run time ± 20-25 minutes per tested version.
• Double-click run-mrt-1s, it will open your browser and show you the results.
• That's it!

For 5 seconds testing double-click run_mea-5s and when finished double-click run_mrt-5s
For 10 seconds testing double-click run_mea-10s and when finished double-click run_mrt-10s

.....

http://rebel13.nl/rebel13/lczero.html

[Joost Buijs]

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822
LC0-090618-390           :   2884   1058     1500     0.705  12194     15000      0.813

Strange enough network 390 performs somewhat worse compared to 395 when using LC0, probably there are algorithmic differences between LC0 and LCZero in the way the network is handled.

Odd indeed.....

That's completely expected. lc0 is faster, so it produces more nodes in the same time. Hence, it's stronger.

Of course LC0 is faster and (hopefully) stronger. What I said is that with LC0 network 395 performs better on STS than network 390, and with LCZero it's the other way around, this is a bit odd, don't you agree?

[CMCanavessi]

Ah yes, that's a nice way to check for policy and value head. 1 net might be better for short tc, and another one may perform better with long tc.