Devlog of Leorik

Discussion of chess software programming and technical issues.

Moderator: Ras

User avatar
lithander
Posts: 915
Joined: Sun Dec 27, 2020 2:40 am
Location: Bremen, Germany
Full name: Thomas Jahn

Re: Devlog of Leorik

Post by lithander »

mvanthoor wrote: Thu Feb 17, 2022 4:30 pm I'd expect the engine to score in CCRL:

2160 (+/- 30) + 40 (staged move generation) + 20 (max? better tuned tables) = ~2220 +/- 30. Take a bit of variance into account due to playing a gauntlet against 20-30 different opponents (with probably more features for most of them), so my expectation would be 2170 - 2230 for this version. I'd be surprised to see it lower or higher than this.
I've set up a gauntlet on fast time controls with the current build and anchored all the other engines based on their CCRL blitz rating and it seems to confirm your guess!

Code: Select all

./cutechess-cli.exe -tournament gauntlet -rounds 2000 -games 2 -repeat -concurrency 5 -each tc=10+0.2 book=varied.bin option.Hash=50

Code: Select all

   # PLAYER           :  RATING  POINTS  PLAYED   (%)
   1 Dumb 1.4         :  2356.0   155.5     264    59
   2 MadChess 2.0     :  2286.0   151.0     264    57
   3 Leorik 0.2.10    :  2229.1   801.5    1587    51
   4 Bit-Genie 4      :  2229.0   133.0     265    50
   5 Ct800 1.32       :  2226.0   134.5     264    51
   6 Blunder 6.1      :  2155.0   113.5     264    43
   7 Zahak 2.0        :  2103.0    98.0     266    37
Minimal Chess (simple, open source, C#) - Youtube & Github
Leorik (competitive, in active development, C#) - Github & Lichess
User avatar
mvanthoor
Posts: 1784
Joined: Wed Jul 03, 2019 4:42 pm
Location: Netherlands
Full name: Marcel Vanthoor

Re: Devlog of Leorik

Post by mvanthoor »

Wow. That really is at the top end of the margin. It seems to bode well for the addition of staged move generatiom to Rustic.

Rustic 4 scores 2160; where it obtains 2105 with worst performance, and 2210 with best performance, depending on the opponent.
Author of Rustic, an engine written in Rust.
Releases | Code | Docs | Progress | CCRL
eligolf
Posts: 114
Joined: Sat Nov 14, 2020 12:49 pm
Full name: Elias Nilsson

Re: Devlog of Leorik

Post by eligolf »

lithander wrote: Thu Feb 17, 2022 11:42 pm I've set up a gauntlet on fast time controls with the current build and anchored all the other engines based on their CCRL blitz rating and it seems to confirm your guess!

Code: Select all

./cutechess-cli.exe -tournament gauntlet -rounds 2000 -games 2 -repeat -concurrency 5 -each tc=10+0.2 book=varied.bin option.Hash=50

Code: Select all

   # PLAYER           :  RATING  POINTS  PLAYED   (%)
   1 Dumb 1.4         :  2356.0   155.5     264    59
   2 MadChess 2.0     :  2286.0   151.0     264    57
   3 Leorik 0.2.10    :  2229.1   801.5    1587    51
   4 Bit-Genie 4      :  2229.0   133.0     265    50
   5 Ct800 1.32       :  2226.0   134.5     264    51
   6 Blunder 6.1      :  2155.0   113.5     264    43
   7 Zahak 2.0        :  2103.0    98.0     266    37
Cool to see, that is really strong with the setup you have! Out of curiosity, how much depth can you reach with that kind of time control for each move on average, not counting quiescense or other extensions depth?
User avatar
mvanthoor
Posts: 1784
Joined: Wed Jul 03, 2019 4:42 pm
Location: Netherlands
Full name: Marcel Vanthoor

Re: Devlog of Leorik

Post by mvanthoor »

eligolf wrote: Fri Feb 18, 2022 5:30 am Cool to see, that is really strong with the setup you have! Out of curiosity, how much depth can you reach with that kind of time control for each move on average, not counting quiescense or other extensions depth?
This doesn't only depend on engine features, but also on CPU power.

My engine Rustic (which now has the same features as Leorik, except staged move generation) can reach depths 7-9 in the middle game and 10+ in the end-game, on an older i7-6700K when running with 10s+0.1s time controls. Rustic doesn't have any pruning or null-move yet, so as soon as that is added, the depth will become higher.
Author of Rustic, an engine written in Rust.
Releases | Code | Docs | Progress | CCRL
User avatar
lithander
Posts: 915
Joined: Sun Dec 27, 2020 2:40 am
Location: Bremen, Germany
Full name: Thomas Jahn

Re: Devlog of Leorik

Post by lithander »

eligolf wrote: Thu Feb 17, 2022 2:22 pm Very cool! I agree that the state you are in now is my favourite because you can somewhat understand why the engine is doing as it should. With other prunings such as null moves it is so hard to find bugs and understand if things get better or just breaks. Looking forward to try it out!
Leorik reaches depths of 7-8 in the beginning and in the endgame, on an almost empty board, it can reach depth 9-10.

With the features I currently have the branching factor is around 5-6 so each next iteration takes 5-6x longer than the previous one. This rather high branching factor means that even if you give the engine a lot more time it doesn't really manage to convert that into a significantly deeper searches. If the other engines from the gaunlet have a lower branching factor due to already implementing more aggressive pruning (e.g. Null-move pruning, late move reduction, razoring etc) they will benefit more from being given extra time.

This is also confirmed by the gauntlet I ran overnight with the time control settings that CCRL uses for Blitz games, 2 minutes and 1 second increment, where Leorik 0.2.10 only achieved 2164.1 ELO against the same opponents!

That's 65 ELO less than on the ultra fast TC but I think it's to be expected. I would expect Rustic to share a similar fate when pitted against the same opponents under the same conditions.

MadChess 2.0 for example searches almost twice as deep as Leorik if you give it a few seconds but only visits 1/3 of the nodes. The engines are not comparable feature wise. Which explains why their strength scales differently based on time controls.
Minimal Chess (simple, open source, C#) - Youtube & Github
Leorik (competitive, in active development, C#) - Github & Lichess
User avatar
mvanthoor
Posts: 1784
Joined: Wed Jul 03, 2019 4:42 pm
Location: Netherlands
Full name: Marcel Vanthoor

Re: Devlog of Leorik

Post by mvanthoor »

lithander wrote: Fri Feb 18, 2022 11:36 am That's 65 ELO less than on the ultra fast TC but I think it's to be expected. I would expect Rustic to share a similar fate when pitted against the same opponents under the same conditions.
True.
MadChess 2.0 for example searches almost twice as deep as Leorik if you give it a few seconds but only visits 1/3 of the nodes. The engines are not comparable feature wise. Which explains why their strength scales differently based on time controls.
Also true. Brute speed can only get you so far. When i let Rustic run to depth 10 on the KiwiPete position (with a 32 MB TT), I get this, on a 6700K:

Code: Select all

go depth 10
info score cp 72 depth 1 seldepth 11 time 0 nodes 1032 nps 0 hashfull 411 pv e2a6 b4c3
info score cp 72 depth 2 seldepth 11 time 0 nodes 1251 nps 0 hashfull 411 pv e2a6 b4c3
info score cp -3 depth 3 seldepth 11 time 1 nodes 1574 nps 1574000 hashfull 411 pv e2a6 b4c3
info score cp -3 depth 4 seldepth 11 time 2 nodes 2902 nps 1451000 hashfull 411 pv e2a6 b4c3
info score cp -3 depth 5 seldepth 19 time 5 nodes 15264 nps 3052800 hashfull 411 pv e2a6 b4c3
info score cp -17 depth 6 seldepth 19 time 10 nodes 56182 nps 5618200 hashfull 411 pv e2a6 e6d5 c3d5 f6d5 e5c4 b6c4 a6c4 h3g2 f3g2
info score cp -39 depth 7 seldepth 21 time 14 nodes 82207 nps 5871929 hashfull 411 pv e2a6 e6d5 e1g1 e7e5 c3d5 f6d5 e4d5 h3g2 f3g2 e5d5 d2b4 d5g2 g1g2 g7b2
info score cp -39 depth 8 seldepth 22 time 31 nodes 186856 nps 6027613 hashfull 412 pv e2a6 e6d5 e1g1 h3g2 f3g2 e7e5 c3d5 f6d5 e4d5 e5d5 d2b4 d5g2 g1g2 g7b2
info score cp -26 depth 9 seldepth 25 time 4007 nodes 24016984 nps 5993757 hashfull 877 pv e2a6 e6d5 c3d5 b6d5 a6b7 h3g2 f3g2 a8b8 e4d5 b8b7
info score cp -37 depth 10 seldepth 27 time 22089 nodes 122801444 nps 5559394 hashfull 999 pv e2a6 e6d5 c3b5 d5e4 b5c7 e8d8 f3g3 h3g2 h1g1 f6h5 d2b4 h5g3 b4e7 d8c7 h2g3 g7e5 g1g2 e5b2
bestmove e2a6
It reaches depth 8 almost instantly, at 0.31s, but then it takes roughly 4 seconds and 22 seconds to reach depths 9 and 10. In ultra-fast time controls it's never going to reach those depths. Even if you give the engine _a lot_ more time, it _might_ reach depth 9 in positions such as this, but it still won't reach depth 10. This is indeed because of the branching factor, and that's the reason why engines with a larger feature set benefit more from longer time controls, and thus the result in Elo against those engines will drop.

@eligolf: staged move generation doesn't decrease the engine's branching factor. It only saves on move generation time; in case you were wondering. Branching factor gets decreased because of pruning, where you decide to just throw away/not search some branches for one reason or another.
Author of Rustic, an engine written in Rust.
Releases | Code | Docs | Progress | CCRL
User avatar
lithander
Posts: 915
Joined: Sun Dec 27, 2020 2:40 am
Location: Bremen, Germany
Full name: Thomas Jahn

Devlog of Leorik - Version 1.0 released!

Post by lithander »

Yeserday I implemented a last test I wanted to make.

I had found thousands of "mate in X" positions here and to the best of my knowledge all the pruning I do in Leorik (alpha/beta, PVS and TT hits) should be safe in such a way that if you know there's a mate in X plies the engine should be able to find it at exactly search depth X.

And Leorik passed that test with 100% accuracy on literally thousands of positions. :D I wanted to preserve this state in a first release because as soon as I implement null-moves and similar search improvements this feat is no longer possible.

So I decided it's time. But then I took way to long to clean up the codebase, create builds for all major platforms, write a github readme, worry about the license (still haven't decided so for now there's none) and in the end when I had everything ready and had just set the repository to public I looked at the clock and postponed the release for a day because it was almost 4 am in the night. O_o

When I woke up Günther had already noticed the newly public repo and announced the release of Leorik 1.0! :shock: It's great to see that people around here still care about the release of yet another mediocre engine! ;)

He's right to call it "my baby" of course and I'm curious about how the "adult" version of Leorik will turn out to be!

So... anway.

Version 1.0 is released:
https://github.com/lithander/Leorik

I'd be honored if you'd give the engine some play and report back if everything is working. Especially the builds for Linux, Mac and Linux ARM (should run on the Raspy) I can't test myself!
Minimal Chess (simple, open source, C#) - Youtube & Github
Leorik (competitive, in active development, C#) - Github & Lichess
amanjpro
Posts: 883
Joined: Sat Mar 13, 2021 1:47 am
Full name: Amanj Sherwany

Re: Devlog of Leorik - Version 1.0 released!

Post by amanjpro »

lithander wrote: Sat Feb 19, 2022 10:13 pm Yeserday I implemented a last test I wanted to make.

I had found thousands of "mate in X" positions here and to the best of my knowledge all the pruning I do in Leorik (alpha/beta, PVS and TT hits) should be safe in such a way that if you know there's a mate in X plies the engine should be able to find it at exactly search depth X.

And Leorik passed that test with 100% accuracy on literally thousands of positions. :D I wanted to preserve this state in a first release because as soon as I implement null-moves and similar search improvements this feat is no longer possible.

So I decided it's time. But then I took way to long to clean up the codebase, create builds for all major platforms, write a github readme, worry about the license (still haven't decided so for now there's none) and in the end when I had everything ready and had just set the repository to public I looked at the clock and postponed the release for a day because it was almost 4 am in the night. O_o

When I woke up Günther had already noticed the newly public repo and announced the release of Leorik 1.0! :shock: It's great to see that people around here still care about the release of yet another mediocre engine! ;)

He's right to call it "my baby" of course and I'm curious about how the "adult" version of Leorik will turn out to be!

So... anway.

Version 1.0 is released:
https://github.com/lithander/Leorik

I'd be honored if you'd give the engine some play and report back if everything is working. Especially the builds for Linux, Mac and Linux ARM (should run on the Raspy) I can't test myself!
Congrats! I am looking forward to see this kid grow up :)
User avatar
algerbrex
Posts: 608
Joined: Sun May 30, 2021 5:03 am
Location: United States
Full name: Christian Dean

Re: Devlog of Leorik - Version 1.0 released!

Post by algerbrex »

lithander wrote: Sat Feb 19, 2022 10:13 pm Yeserday I implemented a last test I wanted to make.

I had found thousands of "mate in X" positions here and to the best of my knowledge all the pruning I do in Leorik (alpha/beta, PVS and TT hits) should be safe in such a way that if you know there's a mate in X plies the engine should be able to find it at exactly search depth X.

And Leorik passed that test with 100% accuracy on literally thousands of positions. :D I wanted to preserve this state in a first release because as soon as I implement null-moves and similar search improvements this feat is no longer possible.

So I decided it's time. But then I took way to long to clean up the codebase, create builds for all major platforms, write a github readme, worry about the license (still haven't decided so for now there's none) and in the end when I had everything ready and had just set the repository to public I looked at the clock and postponed the release for a day because it was almost 4 am in the night. O_o

When I woke up Günther had already noticed the newly public repo and announced the release of Leorik 1.0! :shock: It's great to see that people around here still care about the release of yet another mediocre engine! ;)

He's right to call it "my baby" of course and I'm curious about how the "adult" version of Leorik will turn out to be!

So... anway.

Version 1.0 is released:
https://github.com/lithander/Leorik

I'd be honored if you'd give the engine some play and report back if everything is working. Especially the builds for Linux, Mac and Linux ARM (should run on the Raspy) I can't test myself!
Congratulations Thomas! I'm excited to Leorik develop, and I hope it'll be a useful and challenging sparring partner for future releases of Blunder (whenever I find the time again to work on releasing another version :lol: ).
Mike Sherwin
Posts: 965
Joined: Fri Aug 21, 2020 1:25 am
Location: Planet Earth, Sol system
Full name: Michael J Sherwin

Re: Devlog of Leorik

Post by Mike Sherwin »

A real fight this time! :)