Devlog of Leorik

[dangi12012]

Just because you specifically asked me not to.
https://github.com/lithander/Leorik/pull/6/files

Work in progress.

[lithander]

https://github.com/lithander/Leorik/pull/6/files

[pgn][Event "?"]
[Site "?"]
[Date "2023.06.11"]
[Round "?"]
[White "Leorik-NNUE"]
[Black "Leorik-2.4.3"]
[Result "0-1"]
[ECO "A00"]
[GameDuration "00:00:31"]
[GameEndTime "2023-06-11T03:29:23.608 Mitteleuropäische Sommerzeit"]
[GameStartTime "2023-06-11T03:28:51.731 Mitteleuropäische Sommerzeit"]
[Opening "Durkin's attack"]
[PlyCount "24"]
[TimeControl "40/60"]

1. Na3 {0.00/17 1.5s} e5 {+0.66/15 0.96s} 2. Nf3 {0.00/16 1.8s}
e4 {+0.87/16 0.98s} 3. Nb1 {0.00/15 0.86s} exf3 {+3.14/18 1.0s}
4. exf3 {0.00/16 2.3s} Nf6 {+3.32/17 2.2s} 5. Ke2 {0.00/15 1.4s}
Qe7+ {+M15/20 1.8s} 6. Kd3 {0.00/15 0.97s} Qd6+ {+M13/19 1.6s}
7. Ke2 {-M12/16 0.76s} Qe5+ {+M11/19 1.4s} 8. Kd3 {-M10/18 1.2s}
d5 {+M9/18 1.5s} 9. Nc3 {-M8/18 1.4s} Bf5+ {+M7/16 1.3s} 10. Ne4 {-M6/18 0.78s}
dxe4+ {+M5/16 1.1s} 11. fxe4 {-M4/16 1.0s} Qd5+ {+M3/15 1.7s}
12. Ke2 {-M2/19 1.3s} Qxe4# {+M1/18 1.1s, Black mates} 0-1[/pgn]

[dangi12012]

Yeah of course because it's returning 0 because the PR at the bottom is not finished yet.
It will be interesting to see if scalar NNUE has a chance against simd optimized static eval with pawn evals etc.
Working a bit on this yesterday reminded me why I like C# also. Quite fun.

[lithander]

I didn't realize my last post to this devlog has been almost 3 month ago.

In the past weeks I didn't make a lot of progress on Leorik but I have also not completely stopped working on it. It's not abandoned.

My problem right now is that my target for version 2.5 is 2900 CCRL Elo and I find it harder to reach than anticipated. I don't mind working a few nights on a new idea if it pays off but the last time this worked was the SIMD powered King-relative material evaluation I wrote about previously. All my ideas since have failed. I don't want to look at other engines source code but also seem to be unable to significantly improve it on my own. That creates a frustrating development cycle of fiddling around with small details and waiting for test results of which most fail. I don't know if some of you can relate but it's just not... fun? It feels like work. And that's not helping with making progress.

But I will release a version 2.5 soon. And I'm contemplating to change the license to a liberal open source license like MIT.

Originally I went closed source to be free to work at my own pace and not get my hand forced by a successful fork. But I understand now that that I'll never compete with the top engines and then the question is what value can Leorik provide? A sparring partner for other engines, sure. But also people have said nice things about the code style and it's readability. It could allow someone like Daniel, who always has a lot of ideas but not (yet) an engine of his own to try them out. If you're fine with C# but not willing to start from scratch it might provide a good platform for your own experiments!

Personally, after version 2.5, I'll probably try a few experiments led by my curiosity and not focused on the ELO race. I might look in the direction of MCTS and neural networks. But first I'll have to get 2.5 out of the door!

[JoAnnP38]

I didn't realize my last post to this devlog has been almost 3 month ago.

In the past weeks I didn't make a lot of progress on Leorik but I have also not completely stopped working on it. It's not abandoned.

[...snip...]

My problem right now is that my target for version 2.5 is 2900 CCRL Elo and I find it harder to reach than anticipated.
Personally, after version 2.5, I'll probably try a few experiments led by my curiosity and not focused on the ELO race. I might look in the direction of MCTS and neural networks. But first I'll have to get 2.5 out of the door!

I am pretty much right there with you. I haven't posted on my DevLog in quite some time either because, well like you, I have little to report. Every idea I have and try fails. I get tired of watching SPRT tests go for 30K games and produce an indecisive result! I keep on trying ideas documented in CPW only to find out they don't gain for Pedantic! I do have an idea bubbling in the back of my head though that I haven't started because the way I want to go about it may take a while to fully implement. I was just exploring CPW the other day and I came across a description of a technique called Rank Cut and the first thing I thought about was how to "improve" it (yeah, so sue me for being so optimistic!) Late move reductions and pruning are so obtuse and no really good way of tuning them other than spending 100K games experimenting. So, what if I collected stats on everything my engine has available to it (i.e. depth, ply, eval, is position in check, is pawn threatening promotion, move number, # of times best move has changed, how often searching deeper results in a better move, etc.) and create a database that I could then use to develop a machine learning model in ML.NET or some other framework? I could then use the trained model to control pruning/reductions. At least then I wouldn't just be randomly tweaking parameters. I could attack this problem the same way I handle eval -- data gen, followed by training.

Oh well, I still have a lot of ideas for enhancing eval, I just have been putting them off trying to improve my search so I have some headroom for the additional processing time they may cost.

Developing an MCTS + NN engine also sounds like a lot of fun, but I have so little background and formal training in AI and the math required for some of the techniques I am currently approaching that very carefully. I might have to spend a few months in remedial learning before I could make much progress.

Oh well, keep on plugging away. Your code and ideas you expressed in Minimal Chess were an inspiration for me. Plus, I am always waiting just what you have to post next on your DevLog!!

[eduherminio]

Despite the fact that I've also decided not to look at other people's code yet, until my engine reaches certain strength, you two folks are an inspiration for us C# engine newbies, so keep up the good work! And lithander, a liberal license would be welcome, of course

[lithander]

Oh well, keep on plugging away. Your code and ideas you expressed in Minimal Chess were an inspiration for me. Plus, I am always waiting just what you have to post next on your DevLog!!

[...] you two folks are an inspiration for us C# engine newbies, so keep up the good work! And lithander, a liberal license would be welcome, of course

Thanks a lot for the encouraging words! It sure has helped my motivation to intensify work on 2.5!

...but actually in my last post I forgot to mention something. It's that I find it hard to pinpoint Leorik's strength and the smaller the expected Elo gains of a tweak are the harder it is to verify that there's actually an improvement.

Maybe some remember that when I released Leorik 2.4 I expected to reach 2800 CCRL Elo with it. In the CCRL 40/15 list it has performed pretty consistently around that level. But after the first few hundred games in the blitz list it was at 2950 and it took ~2000 more games against mostly stronger engines to "drop" it's rating to 2856.

So I've got a similar problem with the version 2.5... after JoAnn's post I thought the current version of Pedantic should be a good sparing partner and ran 1000 games at 5s + 100ms increment time control against 0.4.1

Leorik blew Pedantic out of the water achieving a 200 Elo advantage! That would be 150 Elo more than what I can rightfully expect and so it's not a source of confidence but of doubt about my testing methodology or maybe my hardware or maybe I have misconfigured Pedantic somehow.

[JoAnnP38]

So I've got a similar problem with the version 2.5... after JoAnn's post I thought the current version of Pedantic should be a good sparing partner and ran 1000 games at 5s + 100ms increment time control against 0.4.1

Sorry, but Pedantic is most likely not a good sparing partner at that TC. I have too much overhead starting a search it seems.

[lithander]

Sorry, but Pedantic is most likely not a good sparing partner at that TC. I have too much overhead starting a search it seems.

Good to know that it's just due to the fast time controls. But there are no time forfeits so Pedantic can handle it just fine!

My impatience has made me reduce the match times bit by bit and 5+0.1 has become my default setting for selfplay! I suppose that creates a strong selection pressure for Leorik to develop into a version that's strong under tight TC settings.

[JoAnnP38]

Sorry, but Pedantic is most likely not a good sparing partner at that TC. I have too much overhead starting a search it seems.

Good to know that it's just due to the fast time controls. But there are no time forfeits so Pedantic can handle it just fine!

My impatience has made me reduce the match times bit by bit and 5+0.1 has become my default setting for selfplay! I suppose that creates a strong selection pressure for Leorik to develop into a version that's strong under tight TC settings.

Pedantic is much better than it used to be at tight TC. It has never timed out, because internally it always reserves 25ms to respond. In my SPRT testing I only go down as far as 10+0.1; however, whenever I'm testing to measure Elo I typically will try to mimic CCRL settings for either Blitz (i.e. 2+1) or 40/15. Sometimes I will use 60+0.5, but never really any lower. I also set hash table sizes to 256mb and just recently setup Syzygy tablebases for all engines that support it.