Lc0 v18 rc2 for 2080 Syzygy implementation

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 10:16 am

jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
The result on 100 positions in an hour or so. At short time control, as it seems to not matter.

User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 11:09 am

Laskos wrote:
Tue Oct 02, 2018 10:16 am
jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
The result on 100 positions in an hour or so. At short time control, as it seems to not matter.
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.

jkiliani
Posts: 143
Joined: Wed Jan 17, 2018 12:26 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by jkiliani » Tue Oct 02, 2018 11:12 am

Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you for the whole Leela Dev team! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?

User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 11:22 am

jkiliani wrote:
Tue Oct 02, 2018 11:12 am
Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.

Javier Ros
Posts: 181
Joined: Fri Oct 12, 2012 10:48 am
Location: Seville (SPAIN)
Full name: Javier Ros

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Javier Ros » Tue Oct 02, 2018 11:53 am

Very interesting experiment and great work! :D
The love relationship between a chess engine tester and his computer can be summarized in one sentence:
Until heat do us part.

User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 12:33 pm

Laskos wrote:
Tue Oct 02, 2018 11:22 am
jkiliani wrote:
Tue Oct 02, 2018 11:12 am
Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.
Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.

Damir
Posts: 2140
Joined: Mon Feb 11, 2008 2:53 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Damir » Tue Oct 02, 2018 1:03 pm

jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
How about fixing a bug where engine will suddenly resign during the game out of nowhere? This happens during draw or won game, and especially in long games...

jkiliani
Posts: 143
Joined: Wed Jan 17, 2018 12:26 pm

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by jkiliani » Tue Oct 02, 2018 1:06 pm

Damir wrote:
Tue Oct 02, 2018 1:03 pm
How about fixing a bug where engine will suddenly resign during the game out of nowhere? This happens during draw or won game, and especially in long games...
To fix bugs, we need to first reproduce them. Can you give a .pgn file where this happened for you, as well as specifics about the binary you used and your command line parameters?

User avatar
Laskos
Posts: 9410
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by Laskos » Tue Oct 02, 2018 1:33 pm

Laskos wrote:
Tue Oct 02, 2018 12:33 pm
Laskos wrote:
Tue Oct 02, 2018 11:22 am
jkiliani wrote:
Tue Oct 02, 2018 11:12 am
Laskos wrote:
Tue Oct 02, 2018 11:09 am
No misses this time on 100 hard 6-men wins.
Maybe I will check on say 200 other 6-men won positions, but it seems fine now.
Great, thank you! Do you have any testing suites about holding difficult draws, or avoiding losses on TB lost positions against opponents without TB?
I tested yesterday on 100 long 6-men draws (not sure if they are exactly hard, but many of them are), no losses. Avoiding TB losses against non-TB engine needs comparison with another Syzygy-enabled engine against that non-TB engine. Might compare Lc0_Syzygy against SF8_no_TB compared to SF8_Syzygy against SF8_no_TB on 6-men hard wins. Will depend on time control used too.
Out of 50 6-men TB losses, Lc0_Syzygy (the fixed one) saves 13 against SF8_no_TB, SF8_Syzygy saves 15 against SF8_no_TB. So, behaves similarly to SF8, which has a good TB implementation (and good endgame eval, which Lc0 doesn't). Might check how many saves SF dev.
SF_dev_Syzygy saves 23 out of 50 TB losses. Might be due to different Syzygy implementation, stronger play, etc., I don't know.

nabildanial
Posts: 104
Joined: Thu Jun 05, 2014 3:29 am
Location: Malaysia

Re: Lc0 v18 rc2 for 2080 Syzygy implementation

Post by nabildanial » Tue Oct 02, 2018 1:36 pm

Damir wrote:
Tue Oct 02, 2018 1:03 pm
jkiliani wrote:
Tue Oct 02, 2018 10:12 am
Laskos wrote:
Tue Oct 02, 2018 8:41 am
crem wrote:
Tue Oct 02, 2018 6:48 am
Laskos wrote:
Mon Oct 01, 2018 9:16 pm
can00336 wrote:
Mon Oct 01, 2018 8:47 pm
Thanks for finding a bug in the implementation. A proposed fix is available here: https://github.com/LeelaChessZero/lc0/pull/408
You can download a CUDA binary to try the fix here: https://ci.appveyor.com/api/buildjobs/g ... %2Flc0.exe
No, still misses wins as 3-fold rep. at steady rate.
Hi Laskos,

We have a fix for that https://github.com/LeelaChessZero/lc0/pull/410, which we have no way to check currently. Would it be possible for you to test it? The deadline for CCCC binary submission is today, and we'll have to release v0.18.1 today with a fix.
As a last resort we can push this patch blindly, but it would be nice to have it tested. :D
The CUDA build of Lc0 with this patch is https://ci.appveyor.com/api/buildjobs/3 ... %2Flc0.exe

Thanks!
Sill misses by 3-fold, seems at lower rate, only 3 misses out of 100 positions. Time control for Lc0 is 0.5s/move, but as I already wrote, it seems to not matter, and even on easy 6-men wins it misses at pretty much the same rate. Here is the PGN of misses:

Hi, thanks a lot for testing! The dev team found another bug in the tablebase implementation, where only repeats from the current position but not from all positions since the last zeroing move were considered. There is now a fix for this: https://github.com/LeelaChessZero/lc0/pull/412, with compiled binary https://ci.appveyor.com/api/buildjobs/5 ... %2Flc0.exe. If you could run your testing suite with the new build again this would be great :D Thanks again!
How about fixing a bug where engine will suddenly resign during the game out of nowhere? This happens during draw or won game, and especially in long games...
How does a UCI engine resign in games?

Post Reply