Page 1 of 2
LcZero and STS
Posted: Thu Jun 14, 2018 10:16 am
by Rebel
Okay, this is my first step in the mysterious
nothing world. Got the CPU version to run together with the W323 network as used by CCRL. Did not get the GPU version to work, not even after a driver update, perhaps the NVIDIA Quadro FX 1800 is too old, I am dunno. So I will have to do with the slow CPU version for the moment.
I ran the STS test with MEA and compared with the results of (good old) Fruit 2.1
http://rebel13.nl/lczero.html
Not bad at all for a start, will run it again when a new approved network version becomes available.
Just inform me.
Re: LcZero and STS
Posted: Thu Jun 14, 2018 1:06 pm
by CMCanavessi
Interesting results!
Would you run net 390 and/or 395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.
Re: LcZero and STS
Posted: Thu Jun 14, 2018 2:49 pm
by Jesse Gersenson
Rebel wrote: ↑Thu Jun 14, 2018 10:16 am
...perhaps the NVIDIA Quadro FX 1800 is too old...
The FX 1800 isn't supported by tensorflow, which requires a card scoring 3.5 or higher on this page:
https://developer.nvidia.com/cuda-gpus#collapse2
Actually, 3.0 is possible with a custom compile of tensorflow.
Am not certain this is why that card isn't working.
(reference:
https://www.tensorflow.org/install/)
Re: LcZero and STS
Posted: Thu Jun 14, 2018 3:01 pm
by Milos
3.0 capability is enough. Problem is Quadro FX 1800 is not even listed as CUDA capable, i.e. NVIDIA simply doesn't provide CUDA support for it.
The reason is probably that FX 1800 is way too poor since it was made 10 years ago in a prehistoric technology. So even if somehow managed to run LC0 I wouldn't expect more than 50nps at best, more probably something around 20-30.
P.S. It has nothing to do with tensorflow. TF is Google's framework. CUDA is NVIDIA's propitiatory library. LC0 is running on CUDA alone (TF framework was dropped quite some time ago).
Re: LcZero and STS
Posted: Thu Jun 14, 2018 3:12 pm
by Joost Buijs
Out of curiosity I ran STS with the latest LC0 on my GTX-1080ti, 1 sec. per move only because 10 secs. takes me too long.
Code: Select all
A. Processor
Brand : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch : X86_64
Count : 16
B. Engine settings
Threads : 2
Hash (mb) : 128
Time(s)/pos : 1.0
C. Test set
Filename : STS.epd
NumPos : 1500
D. Results
Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate
LC0-090618-395 : 2884 1081 1500 0.721 12324 15000 0.822
I have the impression that LC0 doesn't make full use of movetime 1000 and that it uses less, more like 700 msec. or so, maybe this has something to do with a fixed number of playouts? Certainly something that can be improved.
In 2013 I used STS to tune my own engine, I didn't change anything in the evaluation function since, with Nightmare the result is as follows:
Code: Select all
A. Processor
Brand : Intel(R) Core(TM) i7-6950X CPU @ 4.00GHz
Arch : X86_64
Count : 20
B. Engine settings
Threads : 10
Hash (mb) : 128
Time(s)/pos : 1.0
C. Test set
Filename : STS.epd
NumPos : 1500
D. Results
Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate
Nightmare v7.80 : 2884 1078 1500 0.719 12213 15000 0.814
According to STS, LC0 with network 395 on a 1080ti is roughly comparable to my own engine on 10x 4GHz. threads, which is remarkable.
I really don't know what the CCRL rating of Nightmare is because I never cared to have it tested, according to my own tests it is roughly 3000 Elo 40/40.
Re: LcZero and STS
Posted: Thu Jun 14, 2018 3:20 pm
by CMCanavessi
Joost Buijs wrote: ↑Thu Jun 14, 2018 3:12 pm
Out of curiosity I ran STS with the latest LC0 on my GTX-1080ti, 1 sec. per move only because 10 secs. takes me too long.
Code: Select all
A. Processor
Brand : Intel(R) Core(TM) i7-5960X CPU @ 4.00GHz
Arch : X86_64
Count : 16
B. Engine settings
Threads : 2
Hash (mb) : 128
Time(s)/pos : 1.0
C. Test set
Filename : STS.epd
NumPos : 1500
D. Results
Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate
LC0-090618-395 : 2884 1081 1500 0.721 12324 15000 0.822
I have the impression that LC0 doesn't make full use of movetime 1000 and that it uses less, more like 700 msec. or so, maybe this has something to do with a fixed number of playouts? Certainly something that can be improved.
In 2013 I used STS to tune my own engine, I didn't change anything in the evaluation function since, with Nightmare the result is as follows:
Code: Select all
A. Processor
Brand : Intel(R) Core(TM) i7-6950X CPU @ 4.00GHz
Arch : X86_64
Count : 20
B. Engine settings
Threads : 10
Hash (mb) : 128
Time(s)/pos : 1.0
C. Test set
Filename : STS.epd
NumPos : 1500
D. Results
Engine : Rating Top1 MaxTop1 Top1Rate Score MaxScore ScoreRate
Nightmare v7.80 : 2884 1078 1500 0.719 12213 15000 0.814
According to STS, LC0 with network 395 on a 1080ti is roughly comparable to my own engine on 10x 4GHz. threads, which is remarkable.
I really don't know what the CCRL rating of Nightmare is because I never cared to have it tested, according to my own tests it is roughly 3000 Elo 40/40.
Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.
Re: LcZero and STS
Posted: Thu Jun 14, 2018 3:54 pm
by Joost Buijs
CMCanavessi wrote: ↑Thu Jun 14, 2018 3:20 pm
Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.
It is a private engine, mainly used to run in the monthly HGM tournament and the ACCA tournaments. The problem is that most things are hard coded and that I never took the time to implement parameter setup via UCI, for me no problem, but for others it is unusable.
A couple of years ago I started working on a new version from scratch, the past 2 years I had other obligations, but since a few months I'm working on the engine again. I still have to do the positional evaluation, but for the rest is everything finished including the UCI protocol. When everything is finished and I'm satisfied with it, I will probably release it.
Re: LcZero and STS
Posted: Thu Jun 14, 2018 4:25 pm
by CMCanavessi
Joost Buijs wrote: ↑Thu Jun 14, 2018 3:54 pm
CMCanavessi wrote: ↑Thu Jun 14, 2018 3:20 pm
Is your engine private or available somewhere? I'd love to run it through my leagues, if it's ~3000 elo it's good enough to survive and even climb up a couple of leagues.
It is a private engine, mainly used to run in the monthly HGM tournament and the ACCA tournaments. The problem is that most things are hard coded and that I never took the time to implement parameter setup via UCI, for me no problem, but for others it is unusable.
A couple of years ago I started working on a new version from scratch, the past 2 years I had other obligations, but since a few months I'm working on the engine again. I still have to do the positional evaluation, but for the rest is everything finished including the UCI protocol. When everything is finished and I'm satisfied with it, I will probably release it.
Awesome!
Re: LcZero and STS
Posted: Thu Jun 14, 2018 9:43 pm
by Rebel
CMCanavessi wrote: ↑Thu Jun 14, 2018 1:06 pm
Interesting results!
Would you run net
390 and/or
395 to see if there's any improvement? They are the strongest networks (in actual game play) so far.
390 looks obviously better, see:
http://rebel13.nl/zero1.html
http://rebel13.nl/zero2.html
http://rebel13.nl/zero3.html
Running 395 now.
Re: LcZero and STS
Posted: Thu Jun 14, 2018 10:09 pm
by CMCanavessi
What do you use to process all the STS epds and get a score? I may try the same with lc0.