SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
corres
Posts: 3338
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by corres » Fri Aug 14, 2020 9:02 am

In my earlier post I reported my test but as I see it escaped the attention.
It is pity but the title of a post can not be modifiable later so I obliged to repeat the part of my test.

My machine
OS - Windows 10 x64
CPU - Ryzen 9 3950x 16 x 4.4 GHz Fixed, SMT=disabled PB=disabled
RAM -32 GB
GUI - Fritz 14
SYZYGY -6 men
Nalimov -5 men (for GUI only)
kTriad (NVIDIA GPUs) = 1 x RTX 2080 Ti + 2 x RTX 2060 (chess power ~ 2 x RTX 2080 Ti)

Test params
TC = 1 min + 2 sec / move
Opening positions 100 with alternated color (200 games) from my Minibook

Test 1
NNUE net = SV-2138
HASH = 2GB

Result 1
SF+NNUE PO 270720 popc 16 cores : Stockfish200718 popc 16 cores = 22(!) : 2 (176 draw) 200 games
Elo difference ~ 70 Elo


Test 2
NNUE-net = SV-2138
HASH = 2GB
Leela ver.0.25.1 Threads = 6, NNCache=20000000, Backend= Multiplexing,
BackendOptions=(backend=cudnn-fp16,gpu=0),(backend=cudnn-fp16,gpu=1)),(backend=cudnn-fp16,gpu=2)
and using Kiudee params
Leela net = SV-384x30-t60-3010

Result 2
SF+NNUE PO 270720 popc 16 cores : kTRIAD = 14 : 6 (180 draw) 200 games
Elo difference ~25 Elo

User avatar
M ANSARI
Posts: 3466
Joined: Thu Mar 16, 2006 6:10 pm

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by M ANSARI » Fri Aug 14, 2020 9:55 am

If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.

corres
Posts: 3338
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by corres » Fri Aug 14, 2020 12:30 pm

M ANSARI wrote:
Fri Aug 14, 2020 9:55 am
If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.
The NVIDIA RTX 3090 is under development this will be the major improvement for Leela.

User avatar
Laskos
Posts: 10597
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by Laskos » Fri Aug 14, 2020 1:04 pm

corres wrote:
Fri Aug 14, 2020 12:30 pm
M ANSARI wrote:
Fri Aug 14, 2020 9:55 am
If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.
The NVIDIA RTX 3090 is under development this will be the major improvement for Leela.
Might also benefit from new backend on new CUDA and cuDNN. NNUE are only good or best for such tactical games like Shogi and Chess, but in Go Alpha Zero like NNs on GPU rule. KataGo is unbelievable strong on my RTX2070 GPU, by now I can blunder check any human Go game, Leela Zero Go games and Alpha Go games. Sometimes KataGo on RTX2070 with many playouts (say half a million) seems stronger than Alpha Zero Go.

Werewolf
Posts: 1304
Joined: Thu Sep 18, 2008 8:24 pm

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by Werewolf » Fri Aug 14, 2020 1:50 pm

corres wrote:
Fri Aug 14, 2020 12:30 pm
M ANSARI wrote:
Fri Aug 14, 2020 9:55 am
If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.
The NVIDIA RTX 3090 is under development this will be the major improvement for Leela.
But how much? Some estimates for Lc0 are as low as +19% speedup, which won't be enough

Raphexon
Posts: 297
Joined: Sun Mar 17, 2019 11:00 am
Full name: Henk Drost

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by Raphexon » Fri Aug 14, 2020 2:05 pm

Laskos wrote:
Fri Aug 14, 2020 1:04 pm
corres wrote:
Fri Aug 14, 2020 12:30 pm
M ANSARI wrote:
Fri Aug 14, 2020 9:55 am
If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.
The NVIDIA RTX 3090 is under development this will be the major improvement for Leela.
Might also benefit from new backend on new CUDA and cuDNN. NNUE are only good or best for such tactical games like Shogi and Chess, but in Go Alpha Zero like NNs on GPU rule. KataGo is unbelievable strong on my RTX2070 GPU, by now I can blunder check any human Go game, Leela Zero Go games and Alpha Go games. Sometimes KataGo on RTX2070 with many playouts (say half a million) seems stronger than Alpha Zero Go.
Are there any serious Go-engine writers using AB in the past 15 years?
With increased core counts I think AB should be more viable in the past. 9x9 Go should definitely be in the range where AB is sufficiently effective.

The other problem is that somebody would either have to find good, simple heuristics for move ordering or we need a second fast NN to deal with move ordering.
Else the BF of 19x19 is going to kill the chances an AB engine may have.

User avatar
Laskos
Posts: 10597
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by Laskos » Fri Aug 14, 2020 2:22 pm

Raphexon wrote:
Fri Aug 14, 2020 2:05 pm
Laskos wrote:
Fri Aug 14, 2020 1:04 pm
corres wrote:
Fri Aug 14, 2020 12:30 pm
M ANSARI wrote:
Fri Aug 14, 2020 9:55 am
If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.
The NVIDIA RTX 3090 is under development this will be the major improvement for Leela.
Might also benefit from new backend on new CUDA and cuDNN. NNUE are only good or best for such tactical games like Shogi and Chess, but in Go Alpha Zero like NNs on GPU rule. KataGo is unbelievable strong on my RTX2070 GPU, by now I can blunder check any human Go game, Leela Zero Go games and Alpha Go games. Sometimes KataGo on RTX2070 with many playouts (say half a million) seems stronger than Alpha Zero Go.
Are there any serious Go-engine writers using AB in the past 15 years?
With increased core counts I think AB should be more viable in the past. 9x9 Go should definitely be in the range where AB is sufficiently effective.

The other problem is that somebody would either have to find good, simple heuristics for move ordering or we need a second fast NN to deal with move ordering.
Else the BF of 19x19 is going to kill the chances an AB engine may have.
I am not aware of strong Go engines of the past 15 years using AB. Until NNs, first without search, then with PUCT MCTS search, there was hardly any solid static eval of position, and move ordering heuristics were rudimentary too. Yes, I think for AB search separate value network and a much smaller policy network are needed. Or a heuristic, but good and simple move ordering heuristics are notoriously difficult for 19x19 Go since long.

corres
Posts: 3338
Joined: Wed Nov 18, 2015 10:41 am
Location: hungary

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by corres » Fri Aug 14, 2020 3:20 pm

Werewolf wrote:
Fri Aug 14, 2020 1:50 pm
corres wrote:
Fri Aug 14, 2020 12:30 pm
M ANSARI wrote:
Fri Aug 14, 2020 9:55 am
If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.
The NVIDIA RTX 3090 is under development this will be the major improvement for Leela.
But how much? Some estimates for Lc0 are as low as +19% speedup, which won't be enough
NVIDIA develops not only new GPUs but new software for their GPUs. RTX 2000 series GPUs yield ~30 %
speedup using CUDA11 and cudnn 7.6.4.
The enhancement of Leela`s effectiveness using more than two GPUs is also a possibility.
In general the greater net of Leela give more exact evaluation than the small net of NNUE, I think.

Werewolf
Posts: 1304
Joined: Thu Sep 18, 2008 8:24 pm

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by Werewolf » Fri Aug 14, 2020 6:17 pm

corres wrote:
Fri Aug 14, 2020 3:20 pm

In general the greater net of Leela give more exact evaluation than the small net of NNUE, I think.
Yes I was thinking this. And Lc0 may move to a bigger net one day too.

How deeply does the Lc0 MCTS search look ahead? It's not complete games is it?

mehmet123
Posts: 201
Joined: Sun Jan 26, 2020 9:38 pm
Location: Turkey
Full name: Mehmet Karaman

Re: SF+NNUE 16 cores aganst Stockfish-dev 16 cores and Leela

Post by mehmet123 » Fri Aug 14, 2020 6:37 pm

Laskos wrote:
Fri Aug 14, 2020 1:04 pm
corres wrote:
Fri Aug 14, 2020 12:30 pm
M ANSARI wrote:
Fri Aug 14, 2020 9:55 am
If this score holds then things look like SF will be ahead even on 8 core against 2 x 2080Ti Lc0. Not sure if your network is the strongest network for SF NN but things are progressing so fast that in 2 to 4 weeks there will probably be some fast gains for SF NN and then things will stabilize. By then it should be a good bit stronger and maybe on 8 core or even 4 cores it can go toe to toe with Lc0 at max hardware. I have a 2080Ti card in my system and it is huge! I can't imagine having 2 or 3 cards in my box !!! The thing is that 32 core is becoming more mainstream and a system with 2 x 2080Ti cards can be better compared to a 64 core system. With 64 cores I think Lc0 has no chance against SF NNUE no matter what the hardware ... unless a major improvement in Lc0 takes place.
The NVIDIA RTX 3090 is under development this will be the major improvement for Leela.
Might also benefit from new backend on new CUDA and cuDNN. NNUE are only good or best for such tactical games like Shogi and Chess, but in Go Alpha Zero like NNs on GPU rule. KataGo is unbelievable strong on my RTX2070 GPU, by now I can blunder check any human Go game, Leela Zero Go games and Alpha Go games. Sometimes KataGo on RTX2070 with many playouts (say half a million) seems stronger than Alpha Zero Go.
Is KataGO commercial, private or free program. If it's commercial or private then which is the most strongest free Go program.

Post Reply