LcZero and STS

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4174
Joined: Thu Aug 18, 2011 10:04 am

LcZero and STS

Post by Rebel » Thu Jun 14, 2018 8:16 am

Okay, this is my first step in the mysterious nothing world. Got the CPU version to run together with the W323 network as used by CCRL. Did not get the GPU version to work, not even after a driver update, perhaps the NVIDIA Quadro FX 1800 is too old, I am dunno. So I will have to do with the slow CPU version for the moment.

I ran the STS test with MEA and compared with the results of (good old) Fruit 2.1

Image

http://rebel13.nl/lczero.html

Not bad at all for a start, will run it again when a new approved network version becomes available.

Just inform me.
Forget this place

User avatar
CMCanavessi
Posts: 667
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LcZero and STS

Post by CMCanavessi » Thu Jun 14, 2018 11:06 am

Interesting results!

Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Jesse Gersenson
Posts: 526
Joined: Sat Aug 20, 2011 7:43 am
Contact:

Re: LcZero and STS

Post by Jesse Gersenson » Thu Jun 14, 2018 12:49 pm

Rebel wrote:
Thu Jun 14, 2018 8:16 am
...perhaps the NVIDIA Quadro FX 1800 is too old...
The FX 1800 isn't supported by tensorflow, which requires a card scoring 3.5 or higher on this page:
https://developer.nvidia.com/cuda-gpus#collapse2

Actually, 3.0 is possible with a custom compile of tensorflow.

Am not certain this is why that card isn't working.

(reference: https://www.tensorflow.org/install/)

Milos
Posts: 3153
Joined: Wed Nov 25, 2009 12:47 am

Re: LcZero and STS

Post by Milos » Thu Jun 14, 2018 1:01 pm

Jesse Gersenson wrote:
Thu Jun 14, 2018 12:49 pm
Rebel wrote:
Thu Jun 14, 2018 8:16 am
...perhaps the NVIDIA Quadro FX 1800 is too old...
The FX 1800 isn't supported by tensorflow, which requires a card scoring 3.5 or higher on this page:
https://developer.nvidia.com/cuda-gpus#collapse2

Actually, 3.0 is possible with a custom compile of tensorflow.

Am not certain this is why that card isn't working.

(reference: https://www.tensorflow.org/install/)
3.0 capability is enough. Problem is Quadro FX 1800 is not even listed as CUDA capable, i.e. NVIDIA simply doesn't provide CUDA support for it.
The reason is probably that FX 1800 is way too poor since it was made 10 years ago in a prehistoric technology. So even if somehow managed to run LC0 I wouldn't expect more than 50nps at best, more probably something around 20-30.

P.S. It has nothing to do with tensorflow. TF is Google's framework. CUDA is NVIDIA's propitiatory library. LC0 is running on CUDA alone (TF framework was dropped quite some time ago).

Joost Buijs
Posts: 729
Joined: Thu Jul 16, 2009 8:47 am
Location: Almere, The Netherlands

Re: LcZero and STS

Post by Joost Buijs » Thu Jun 14, 2018 1:12 pm

Out of curiosity I ran STS with the latest LC0 on my GTX-1080ti, 1 sec. per move only because 10 secs. takes me too long.

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822

I have the impression that LC0 doesn't make full use of movetime 1000 and that it uses less, more like 700 msec. or so, maybe this has something to do with a fixed number of playouts? Certainly something that can be improved.

In 2013 I used STS to tune my own engine, I didn't change anything in the evaluation function since, with Nightmare the result is as follows:

Code: Select all


A. Processor
Brand          : Intel(R) Core(TM) i7-6950X CPU @ 4.00GHz
Arch           : X86_64
Count          : 20

B. Engine settings
Threads        : 10
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
Nightmare v7.80          :   2884   1078     1500     0.719  12213     15000      0.814

According to STS, LC0 with network 395 on a 1080ti is roughly comparable to my own engine on 10x 4GHz. threads, which is remarkable.
I really don't know what the CCRL rating of Nightmare is because I never cared to have it tested, according to my own tests it is roughly 3000 Elo 40/40.

User avatar
CMCanavessi
Posts: 667
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LcZero and STS

Post by CMCanavessi » Thu Jun 14, 2018 1:20 pm

Joost Buijs wrote:
Thu Jun 14, 2018 1:12 pm
Out of curiosity I ran STS with the latest LC0 on my GTX-1080ti, 1 sec. per move only because 10 secs. takes me too long.

Code: Select all

A. Processor
Brand          : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch           : X86_64
Count          : 16

B. Engine settings
Threads        : 2
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
LC0-090618-395           :   2884   1081     1500     0.721  12324     15000      0.822

I have the impression that LC0 doesn't make full use of movetime 1000 and that it uses less, more like 700 msec. or so, maybe this has something to do with a fixed number of playouts? Certainly something that can be improved.

In 2013 I used STS to tune my own engine, I didn't change anything in the evaluation function since, with Nightmare the result is as follows:

Code: Select all


A. Processor
Brand          : Intel(R) Core(TM) i7-6950X CPU @ 4.00GHz
Arch           : X86_64
Count          : 20

B. Engine settings
Threads        : 10
Hash (mb)      : 128
Time(s)/pos    : 1.0

C. Test set
Filename       : STS.epd
NumPos         : 1500

D. Results
Engine                   : Rating   Top1  MaxTop1  Top1Rate  Score  MaxScore  ScoreRate
Nightmare v7.80          :   2884   1078     1500     0.719  12213     15000      0.814

According to STS, LC0 with network 395 on a 1080ti is roughly comparable to my own engine on 10x 4GHz. threads, which is remarkable.
I really don't know what the CCRL rating of Nightmare is because I never cared to have it tested, according to my own tests it is roughly 3000 Elo 40/40.
Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Joost Buijs
Posts: 729
Joined: Thu Jul 16, 2009 8:47 am
Location: Almere, The Netherlands

Re: LcZero and STS

Post by Joost Buijs » Thu Jun 14, 2018 1:54 pm

CMCanavessi wrote:
Thu Jun 14, 2018 1:20 pm
Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.
It is a private engine, mainly used to run in the monthly HGM tournament and the ACCA tournaments. The problem is that most things are hard coded and that I never took the time to implement parameter setup via UCI, for me no problem, but for others it is unusable.

A couple of years ago I started working on a new version from scratch, the past 2 years I had other obligations, but since a few months I'm working on the engine again. I still have to do the positional evaluation, but for the rest is everything finished including the UCI protocol. When everything is finished and I'm satisfied with it, I will probably release it.

User avatar
CMCanavessi
Posts: 667
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LcZero and STS

Post by CMCanavessi » Thu Jun 14, 2018 2:25 pm

Joost Buijs wrote:
Thu Jun 14, 2018 1:54 pm
CMCanavessi wrote:
Thu Jun 14, 2018 1:20 pm
Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.
It is a private engine, mainly used to run in the monthly HGM tournament and the ACCA tournaments. The problem is that most things are hard coded and that I never took the time to implement parameter setup via UCI, for me no problem, but for others it is unusable.

A couple of years ago I started working on a new version from scratch, the past 2 years I had other obligations, but since a few months I'm working on the engine again. I still have to do the positional evaluation, but for the rest is everything finished including the UCI protocol. When everything is finished and I'm satisfied with it, I will probably release it.
Awesome! :)
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

User avatar
Rebel
Posts: 4174
Joined: Thu Aug 18, 2011 10:04 am

Re: LcZero and STS

Post by Rebel » Thu Jun 14, 2018 7:43 pm

CMCanavessi wrote:
Thu Jun 14, 2018 11:06 am
Interesting results!

Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.
390 looks obviously better, see:

http://rebel13.nl/zero1.html
http://rebel13.nl/zero2.html
http://rebel13.nl/zero3.html

Running 395 now.
Forget this place

User avatar
CMCanavessi
Posts: 667
Joined: Thu Dec 28, 2017 3:06 pm
Location: Argentina

Re: LcZero and STS

Post by CMCanavessi » Thu Jun 14, 2018 8:09 pm

What do you use to process all the STS epds and get a score? I may try the same with lc0.
Follow my tournament and some Leela gauntlets live at http://twitch.tv/ccls

Post Reply