LcZero and STS

[Rebel]

Okay, this is my first step in the mysterious nothing world. Got the CPU version to run together with the W323 network as used by CCRL. Did not get the GPU version to work, not even after a driver update, perhaps the NVIDIA Quadro FX 1800 is too old, I am dunno. So I will have to do with the slow CPU version for the moment.

I ran the STS test with MEA and compared with the results of (good old) Fruit 2.1



http://rebel13.nl/lczero.html

Not bad at all for a start, will run it again when a new approved network version becomes available.

Just inform me.

[CMCanavessi]

Interesting results!

Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.

[Jesse Gersenson]

...perhaps the NVIDIA Quadro FX 1800 is too old...

The FX 1800 isn't supported by tensorflow, which requires a card scoring 3.5 or higher on this page:
https://developer.nvidia.com/cuda-gpus#collapse2

Actually, 3.0 is possible with a custom compile of tensorflow.

Am not certain this is why that card isn't working.

(reference: https://www.tensorflow.org/install/)

[Milos]

...perhaps the NVIDIA Quadro FX 1800 is too old...

The FX 1800 isn't supported by tensorflow, which requires a card scoring 3.5 or higher on this page:
https://developer.nvidia.com/cuda-gpus#collapse2

Actually, 3.0 is possible with a custom compile of tensorflow.

Am not certain this is why that card isn't working.

(reference: https://www.tensorflow.org/install/)

3.0 capability is enough. Problem is Quadro FX 1800 is not even listed as CUDA capable, i.e. NVIDIA simply doesn't provide CUDA support for it.
The reason is probably that FX 1800 is way too poor since it was made 10 years ago in a prehistoric technology. So even if somehow managed to run LC0 I wouldn't expect more than 50nps at best, more probably something around 20-30.

P.S. It has nothing to do with tensorflow. TF is Google's framework. CUDA is NVIDIA's propitiatory library. LC0 is running on CUDA alone (TF framework was dropped quite some time ago).

[Joost Buijs]

Out of curiosity I ran STS with the latest LC0 on my GTX-1080ti, 1 sec. per move only because 10 secs. takes me too long.

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822

I have the impression that LC0 doesn't make full use of movetime 1000 and that it uses less, more like 700 msec. or so, maybe this has something to do with a fixed number of playouts? Certainly something that can be improved.

In 2013 I used STS to tune my own engine, I didn't change anything in the evaluation function since, with Nightmare the result is as follows:

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-6950X CPU @ 4.00GHz
Arch           : X86_64
Count          : 20

B. Engine settings
Threads        : 10
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
Nightmare v7.80          :   2884   1078     1500     0.719  12213     15000      0.814

According to STS, LC0 with network 395 on a 1080ti is roughly comparable to my own engine on 10x 4GHz. threads, which is remarkable.
I really don't know what the CCRL rating of Nightmare is because I never cared to have it tested, according to my own tests it is roughly 3000 Elo 40/40.

[CMCanavessi]

Out of curiosity I ran STS with the latest LC0 on my GTX-1080ti, 1 sec. per move only because 10 secs. takes me too long.

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822

I have the impression that LC0 doesn't make full use of movetime 1000 and that it uses less, more like 700 msec. or so, maybe this has something to do with a fixed number of playouts? Certainly something that can be improved.

In 2013 I used STS to tune my own engine, I didn't change anything in the evaluation function since, with Nightmare the result is as follows:

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-6950X CPU @ 4.00GHz
Arch           : X86_64
Count          : 20

B. Engine settings
Threads        : 10
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
Nightmare v7.80          :   2884   1078     1500     0.719  12213     15000      0.814

According to STS, LC0 with network 395 on a 1080ti is roughly comparable to my own engine on 10x 4GHz. threads, which is remarkable.
I really don't know what the CCRL rating of Nightmare is because I never cared to have it tested, according to my own tests it is roughly 3000 Elo 40/40.

Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.

[Joost Buijs]

Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.

It is a private engine, mainly used to run in the monthly HGM tournament and the ACCA tournaments. The problem is that most things are hard coded and that I never took the time to implement parameter setup via UCI, for me no problem, but for others it is unusable.

A couple of years ago I started working on a new version from scratch, the past 2 years I had other obligations, but since a few months I'm working on the engine again. I still have to do the positional evaluation, but for the rest is everything finished including the UCI protocol. When everything is finished and I'm satisfied with it, I will probably release it.

[CMCanavessi]

Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.

It is a private engine, mainly used to run in the monthly HGM tournament and the ACCA tournaments. The problem is that most things are hard coded and that I never took the time to implement parameter setup via UCI, for me no problem, but for others it is unusable.

A couple of years ago I started working on a new version from scratch, the past 2 years I had other obligations, but since a few months I'm working on the engine again. I still have to do the positional evaluation, but for the rest is everything finished including the UCI protocol. When everything is finished and I'm satisfied with it, I will probably release it.

Awesome!

[Rebel]

Interesting results!

Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.

390 looks obviously better, see:

http://rebel13.nl/zero1.html
http://rebel13.nl/zero2.html
http://rebel13.nl/zero3.html

Running 395 now.

[CMCanavessi]

What do you use to process all the STS epds and get a score? I may try the same with lc0.