Checking the backends with the new lc0 binary

[mwyoung]

To remark the excellent result of DX12 backend, which seems by NPS vastly superior to the other two. A glitch occurred with this command line:

lc0_v263rc1.exe benchmark --backend=cudnn-fp16 --minibatch-size=240

which sometimes exits with this error message:

Position: 1/34 rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
Unhandled exception in worker thread: CUDA error: an illegal memory access was encountered (c:\projects\lc0\src\neural\cuda\network_cudnn.cc:789)
Unhandled exception in worker thread:

I found it works better with this format:

lc0_v263rc1.exe benchmark --minibatch-size=240 --backend=cudnn-fp16

As you can see all I did was invert the two arguments.

No there is a issue 0.26.3-rc1. I tested it the day it came out in a 200 game blitz match. It was faster, but it also crashed about 23 times in 200 games. Causing a big loss in the match. I hope this will be corrected in rc2.

Rc2 is now ready for download. And the issue is claimed to be fixed. And rc2 should also run faster.

[AdminX]

I was able to replicate your results on 2070 Super

Code: Select all

DX12 

lc0.exe benchmark --minibatch-size=240 --threads=2 --backend-opts=gpu=0

===========================
Total time (ms) : 341461
Nodes searched  : 3506458
Nodes/second    : 10269

Code: Select all

Cudnn-fp16 

lc0.exe benchmark --minibatch-size=240 --threads=2 --backend-opts=gpu=0

===========================
Total time (ms) : 341514
Nodes searched  : 2762948
Nodes/second    : 8090

Someone directed me to this test version with CUDA 11.1 and cuDNN 8.04
https://appveyorcidatav2.blob.core.wind ... a-cuda.zip
and replace lc0 with this one:
https://appveyorcidatav2.blob.core.wind ... ld/lc0.exe

I am getting very much improved results (50%+ faster) for cudnn-fp16 and cuda-fp16:

cudnn-fp16
Total time (ms) : 341515
Nodes searched : 3547152
Nodes/second : 10386

cuda-fp16
Total time (ms) : 341370
Nodes searched : 3630548
Nodes/second : 10635

dx12
Total time (ms) : 341409
Nodes searched : 3077528
Nodes/second : 9014

Cuda-fp16 seems now even faster than cudnn-fp16, and both above DX12.

Links not working, files may no longer be there.

[Laskos]

I was able to replicate your results on 2070 Super

Code: Select all

DX12 

lc0.exe benchmark --minibatch-size=240 --threads=2 --backend-opts=gpu=0

===========================
Total time (ms) : 341461
Nodes searched  : 3506458
Nodes/second    : 10269

Code: Select all

Cudnn-fp16 

lc0.exe benchmark --minibatch-size=240 --threads=2 --backend-opts=gpu=0

===========================
Total time (ms) : 341514
Nodes searched  : 2762948
Nodes/second    : 8090

Someone directed me to this test version with CUDA 11.1 and cuDNN 8.04
https://appveyorcidatav2.blob.core.wind ... a-cuda.zip
and replace lc0 with this one:
https://appveyorcidatav2.blob.core.wind ... ld/lc0.exe

I am getting very much improved results (50%+ faster) for cudnn-fp16 and cuda-fp16:

cudnn-fp16
Total time (ms) : 341515
Nodes searched : 3547152
Nodes/second : 10386

cuda-fp16
Total time (ms) : 341370
Nodes searched : 3630548
Nodes/second : 10635

dx12
Total time (ms) : 341409
Nodes searched : 3077528
Nodes/second : 9014

Cuda-fp16 seems now even faster than cudnn-fp16, and both above DX12.

Links not working, files may no longer be there.

Try these:

https://appveyorcidatav2.blob.core.wind ... 3A24Z&sp=r

https://appveyorcidatav2.blob.core.wind ... 3A28Z&sp=r

[sarona]

I still cannot get them, either.

Code: Select all

<Error>
<Code>AuthenticationFailed</Code>
<Message>
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:48280c10-101e-0054-34c2-997f87000000 Time:2020-10-03T20:22:43.9719130Z
</Message>
<AuthenticationErrorDetail>
Signature not valid in the specified time frame: Start [Sat, 03 Oct 2020 19:23:24 GMT] - Expiry [Sat, 03 Oct 2020 19:29:24 GMT] - Current [Sat, 03 Oct 2020 20:22:43 GMT]
</AuthenticationErrorDetail>
</Error>

[mwyoung]

To remark the excellent result of DX12 backend, which seems by NPS vastly superior to the other two. A glitch occurred with this command line:

lc0_v263rc1.exe benchmark --backend=cudnn-fp16 --minibatch-size=240

which sometimes exits with this error message:

Position: 1/34 rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
Unhandled exception in worker thread: CUDA error: an illegal memory access was encountered (c:\projects\lc0\src\neural\cuda\network_cudnn.cc:789)
Unhandled exception in worker thread:

I found it works better with this format:

lc0_v263rc1.exe benchmark --minibatch-size=240 --backend=cudnn-fp16

As you can see all I did was invert the two arguments.

No there is a issue 0.26.3-rc1. I tested it the day it came out in a 200 game blitz match. It was faster, but it also crashed about 23 times in 200 games. Causing a big loss in the match. I hope this will be corrected in rc2.

Rc2 is now ready for download. And the issue is claimed to be fixed. And rc2 should also run faster.

All seems to be working fine with 0.26.3-rc2. And now getting much faster speed with cuda 11.1 with the big nets 384x30. Game average right now is 38.2 Knps on a 2080ti with default settings.

[corres]

Try these:

https://appveyorcidatav2.blob.core.wind ... 3A24Z&sp=r

https://appveyorcidatav2.blob.core.wind ... 3A28Z&sp=r

I also got only error messages from there.

[Laskos]

Try these:

https://appveyorcidatav2.blob.core.wind ... 3A24Z&sp=r

https://appveyorcidatav2.blob.core.wind ... 3A28Z&sp=r

I also got only error messages from there.

The links I have seem to not be working anymore, and I am unable to upload these large files.

Get the CUDA backend with all dll's from the new 0.26.3-rc2 official release. It is the fastest, faster than cuDNN backend in all my tests with different nets.

[corres]

Try these:

https://appveyorcidatav2.blob.core.wind ... 3A24Z&sp=r

https://appveyorcidatav2.blob.core.wind ... 3A28Z&sp=r

I also got only error messages from there.

The links I have seem to not be working anymore, and I am unable to upload these large files.

Get the CUDA backend with all dll's from the new 0.26.3-rc2 official release. It is the fastest, faster than cuDNN backend in all my tests with different nets.

Thanks.

[AdminX]

Try these:

https://appveyorcidatav2.blob.core.wind ... 3A24Z&sp=r

https://appveyorcidatav2.blob.core.wind ... 3A28Z&sp=r

I also got only error messages from there.

The links I have seem to not be working anymore, and I am unable to upload these large files.

Get the CUDA backend with all dll's from the new 0.26.3-rc2 official release. It is the fastest, faster than cuDNN backend in all my tests with different nets.

Or grab this one: https://ci.appveyor.com/project/LeelaCh ... /artifacts

[sarona]

Thanks very much Kai and Ted.

I downloaded (from AppVeyor) and quickly tried both v0.26.3-rc2 (3787) CUDA and cuDNN binaries using the 384x30-t60-4619.pb (Sergio) net and an RTX 2080 GPU. I am using the CUDA 11.1 Toolkit and v8.0.4.30 cuDNN libraries.



Will try the DX12 binary tonight.