New engine releases & news H1 2024

[Graham Banks]

Starzix 4.0 NN
Ricardo Pinto, POR
https://github.com/zzzzz151/Starzix

Says that 3.0 is the latest release.

[petero2]

- does not give mate with KQ vs. KR ... games ended with draw, 50-moves-rule

Shredder 13 GUI, 3,4,5,6men Syzygys, 5"/move auto play out by Obsidian 10.0 avx2-pext- compile, single thread:

[pgn] [Event "?"] [Site "?"] [Date "2024.01.19"] [Round "?"] [White "?"] [Black "?"] [Result "*"] [SetUp "1"] [FEN "8/5k2/5r2/8/2KQ4/8/8/8 w - -"] 1. Qd7+ {+4.64/36 5s} Kg8 {+4.69/35 5s} 2. Kd4 {+4.79/44 9s} Kh8 {+4.89/43 26s} 3. Ke5 {+5.24/38 3s} Rh6 {+5.89/41 10s} 4. Kf5 {+5.94/43 4s} Kg8 {+37.20/36 9s} 5. Kg5 {+M131/62 7s} Rh7 {+M130/69 4s} 6. Qd5+ {+M130/71 5s} Kg7 {+M128/72 26s} 7. Qe4 {+M126/57 3s} Kg8 {+M18/66 9s} 8. Kg6 {+M19/1 0s} Rg7+ {+M126/71 6s} 9. Kh6 {+M120/72 1s} Kf8 {+M125/71 4s} 10. Qa8+ {+M119/72 3s} Kf7 {+M11/72 6s} 11. Qb7+ {+M11/1 0s} Ke6 {+M10/73 8s} 12. Qxg7 {+M10/1 0s} Kd6 {+M9/73 4s} 13. Qc3 {+M9/1 0s} Kd5 {+M8/73 6s} 14. Kg5 {+M8/1 0s} Kd6 {+M7/73 4s} 15. Kf5 {+M7/1 0s} Kd5 {+M6/74 4s} 16. Kf6 {+M6/1 0s} Ke4 {+M5/74 4s} 17. Qd2 {+M5/1 0s} Kf3 {Buch 0s} 18. Kf5 {+M4/1 0s} Kg3 {Buch 0s} 19. Qe2 {+M3/1 0s} Kh3 {+M2/245 0s} 20. Kf4 {+M2/1 0s} Kh4 {Buch 0s} 21. Qh2# {+M1/1 0s} * [/pgn]

Starting position is tbs- mate in 23 with 1.Kd5 as well as with 1.Qd7, tbs- hits are shown by engine in output all game long, regards

I was able to draw from that position as black against Obsidian 10.0. The time control was 60 moves in 3 minutes.
[pgn] [Event "?"] [Site "?"] [Date "2024.01.19"] [Round "?"] [White "Obsidian 10.0"] [Black "Player"] [Result "1/2-1/2"] [FEN "8/5k2/5r2/8/2KQ4/8/8/8 w - - 0 1"] [SetUp "1"] 1. Qd7+ Kg8 2. Qe8+ Kg7 3. Qe7+ Kg6 4. Kd5 Rf4 5. Qa3 Rf5+ 6. Ke4 Rf6 7. Qg3+ Kf7 8. Qb3+ Kg6 9. Qg8+ Kh5 10. Ke5 Rf3 11. Qg2 Rf8 12. Qh3+ Kg5 13. Qg3+ Kh6 14. Qh4+ Kg6 15. Qg4+ Kf7 16. Qf5+ Kg7 17. Qg5+ Kh7 18. Ke6 Rg8 19. Qh5+ Kg7 20. Ke7 Ra8 21. Qe5+ Kg6 22. Qe4+ Kg5 23. Qxa8 Kf4 24. Qd5 Ke3 25. Kf6 Ke2 26. Qc4+ Ke3 27. Qc3+ Ke4 28. Qd2 Kf3 29. Qc2 Ke3 30. Qc3+ Ke4 31. Qd2 Kf3 32. Qc2 Ke3 33. Ke5 Kf3 34. Qd2 Kg3 35. Qc2 Kf3 36. Qd2 Kg3 37. Qc2 Kf3 38. Qb2 Ke3 39. Qc3+ Ke2 40. Ke4 Kf2 41. Qc2+ Kg3 42. Qc6 Kf2 43. Qc2+ Kg3 44. Qc6 Kf2 45. Qc4 Kg3 46. Ke5 Kf3 47. Qc3+ Ke2 48. Ke4 Kf2 49. Qc4 Kg3 50. Ke5 Kf3 51. Qc6+ Ke3 52. Kf5 Kd4 53. Kf4 Kd3 54. Ke5 Ke3 55. Kf5 Kd4 56. Kf4 Kd3 57. Ke5 Ke3 58. Qe6 Kd3 59. Qb3+ Ke2 60. Kf4 Kd2 61. Qc4 Ke1 62. Qa2 Kd1 63. Qf2 Kc1 64. Qa2 Kd1 65. Qf2 Kc1 66. Qa2 Kd1 67. Qh2 Kc1 68. Kf5 Kd1 69. Kf4 Kc1 70. Kf5 Kd1 71. Ke5 Kc1 72. Qa2 Kd1 1/2-1/2 [/pgn]

[Frank Quisinsky]

Good evening Graham,

yes, back to v3.0?
Maybe the programmer found a bug ...

I downloaded v4.0 but will wait with a test.

I think better is if the programmers used this thread only?

Best
Frank

[peter]

I was able to draw from that position as black against Obsidian 10.0. The time control was 60 moves in 3 minutes.
[pgn] [Event "?"] [Site "?"] [Date "2024.01.19"] [Round "?"] [White "Obsidian 10.0"] [Black "Player"] [Result "1/2-1/2"] [FEN "8/5k2/5r2/8/2KQ4/8/8/8 w - - 0 1"] [SetUp "1"] 1. Qd7+ Kg8 2. Qe8+ Kg7 3. Qe7+ Kg6 4. Kd5 Rf4 5. Qa3 Rf5+ 6. Ke4 Rf6 7. Qg3+ Kf7 8. Qb3+ Kg6 9. Qg8+ Kh5 10. Ke5 Rf3 11. Qg2 Rf8 12. Qh3+ Kg5 13. Qg3+ Kh6 14. Qh4+ Kg6 15. Qg4+ Kf7 16. Qf5+ Kg7 17. Qg5+ Kh7 18. Ke6 Rg8 19. Qh5+ Kg7 20. Ke7 Ra8 21. Qe5+ Kg6 22. Qe4+ Kg5 23. Qxa8 Kf4 24. Qd5 Ke3 25. Kf6 Ke2 26. Qc4+ Ke3 27. Qc3+ Ke4 28. Qd2 Kf3 29. Qc2 Ke3 30. Qc3+ Ke4 31. Qd2 Kf3 32. Qc2 Ke3 33. Ke5 Kf3 34. Qd2 Kg3 35. Qc2 Kf3 36. Qd2 Kg3 37. Qc2 Kf3 38. Qb2 Ke3 39. Qc3+ Ke2 40. Ke4 Kf2 41. Qc2+ Kg3 42. Qc6 Kf2 43. Qc2+ Kg3 44. Qc6 Kf2 45. Qc4 Kg3 46. Ke5 Kf3 47. Qc3+ Ke2 48. Ke4 Kf2 49. Qc4 Kg3 50. Ke5 Kf3 51. Qc6+ Ke3 52. Kf5 Kd4 53. Kf4 Kd3 54. Ke5 Ke3 55. Kf5 Kd4 56. Kf4 Kd3 57. Ke5 Ke3 58. Qe6 Kd3 59. Qb3+ Ke2 60. Kf4 Kd2 61. Qc4 Ke1 62. Qa2 Kd1 63. Qf2 Kc1 64. Qa2 Kd1 65. Qf2 Kc1 66. Qa2 Kd1 67. Qh2 Kc1 68. Kf5 Kd1 69. Kf4 Kc1 70. Kf5 Kd1 71. Ke5 Kc1 72. Qa2 Kd1 1/2-1/2 [/pgn]

And Obsidian used tbs? And which one?

Edit: when I go to move nr. 31 of your .pgn, where DTM with tbs start to get supiciously longer instead of shorter, my installation with using 6 men and without it gives #4- output at once.

Single thread started with empty hash with 6men:

[d]8/8/5K2/8/8/5k2/3Q4/8 w - - Engine: Obsidian 10.00 (1024 MB)
von gabe

10.00 0:00 +M4 29.Kf5 (5.738) 2869 TB:43

Bester Zug: Kf6-f5 Zeit: 0:00.047 min K/s: 2.869.000 Knoten: 5.738 TB: 43

Without tbs:


8/8/5K2/8/8/5k2/3Q4/8 w - -

Engine: Obsidian 10.00 (64 MB)
von gabe

10.00 0:00 +M4 29.Kf5 (5.760) 1920

Bester Zug: Kf6-f5 Zeit: 0:00.015 min K/s: 1.920.000 Knoten: 5.760

So I just cannot reproduce your line with my installation neither, regards

[jgilchrist]

Hi

Hi! Thank you, and thanks for taking the time to do these tests, your results are very interesting!

The feature sets do differ slightly, it looks to me like Rustic uses pseudo-legal move generation while Tcheran uses fully legal move generation and so does not have to check the legality of moves during search. But it is pretty amazing how similar the feature sets are otherwise!

I also take care not to count q-search nodes twice.)

Good point - I will need to correct that in v2.1.

As you can see in the time to depth, it turns out that Rustic 4 is a (much) faster engine. (For some reason your NPS is much higher, but that can be due to the way you count it. I only count nodes that do actual work and don't count discarded / early-exit nodes. I also take care not to count q-search nodes twice.)

I have tried running them both on my machine (an M1 Macbook) as I was also curious about whether the results would look any different, assuming you're running on x86. In my tests, Rustic runs at ~7mnps and Tcheran runs at ~11mnps. Much slower than your machine but the ratio is roughly the same. I have tweaked my node counting locally to match Rustic's approach (and fixed the double-counting of the first q-search nodes) so I believe that comparison is accurate.

It looks to me like Tcheran is getting through nodes faster than Rustic, but that Rustic has to analyse many less nodes for any particular depth. For example, Tcheran v1.1 explores 165,787,095 nodes at depth 9 from the start position, while Rustic only has to explore 9,268,090 (~18x less!)

I have spent a lot of time doing profiling and optimisation - far, far too much, probably, the month between v1.0 and v1.1 was spent almost exclusively on that. I should've been spending more time on pruning techniques & the evaluation function but unfortunately I find it quite addictive Sadly as a result I'm not sure there's much low hanging fruit (or even much high-hanging fruit). But I am very curious to know how Rustic is able to search so few nodes - comparatively - at each depth despite having such a similar feature set. I did notice that Tcheran's seldepth at depth 9 is 24 while Rustic's is 23, so perhaps there's something to that, but even at depth 8 (which goes to 23 in quiescence) Tcheran is searching ~35m nodes which is still almost 4x more than Rustic searches at depth 9 (!).

... within 1 Elo-point as predicted by extrapolating Rustic 4-beta's strength from its performance against Tcheran.

Pretty incredible that it turned out that close!

I hope these insights are helpful to you. Good luck on further implementation

You've certainly given me a lot of food for thought! Thanks again for doing these tests and good luck for the release of Rustic 4

[Frank Quisinsky]

Hi there,

I updated the file:
obsidian-bug.zip

https://www.amateurschach.de/download/obsidian-bug.zip (1.3Mb).
Inside the complete 850 games in *.pgn and the *.cbh database with the 56 collected bug-games.
Maybe more of such games are in database, looking very fast in the afternoon.

Best
Frank

[petero2]

I was able to draw from that position as black against Obsidian 10.0. The time control was 60 moves in 3 minutes.
[pgn] [Event "?"] [Site "?"] [Date "2024.01.19"] [Round "?"] [White "Obsidian 10.0"] [Black "Player"] [Result "1/2-1/2"] [FEN "8/5k2/5r2/8/2KQ4/8/8/8 w - - 0 1"] [SetUp "1"] 1. Qd7+ Kg8 2. Qe8+ Kg7 3. Qe7+ Kg6 4. Kd5 Rf4 5. Qa3 Rf5+ 6. Ke4 Rf6 7. Qg3+ Kf7 8. Qb3+ Kg6 9. Qg8+ Kh5 10. Ke5 Rf3 11. Qg2 Rf8 12. Qh3+ Kg5 13. Qg3+ Kh6 14. Qh4+ Kg6 15. Qg4+ Kf7 16. Qf5+ Kg7 17. Qg5+ Kh7 18. Ke6 Rg8 19. Qh5+ Kg7 20. Ke7 Ra8 21. Qe5+ Kg6 22. Qe4+ Kg5 23. Qxa8 Kf4 24. Qd5 Ke3 25. Kf6 Ke2 26. Qc4+ Ke3 27. Qc3+ Ke4 28. Qd2 Kf3 29. Qc2 Ke3 30. Qc3+ Ke4 31. Qd2 Kf3 32. Qc2 Ke3 33. Ke5 Kf3 34. Qd2 Kg3 35. Qc2 Kf3 36. Qd2 Kg3 37. Qc2 Kf3 38. Qb2 Ke3 39. Qc3+ Ke2 40. Ke4 Kf2 41. Qc2+ Kg3 42. Qc6 Kf2 43. Qc2+ Kg3 44. Qc6 Kf2 45. Qc4 Kg3 46. Ke5 Kf3 47. Qc3+ Ke2 48. Ke4 Kf2 49. Qc4 Kg3 50. Ke5 Kf3 51. Qc6+ Ke3 52. Kf5 Kd4 53. Kf4 Kd3 54. Ke5 Ke3 55. Kf5 Kd4 56. Kf4 Kd3 57. Ke5 Ke3 58. Qe6 Kd3 59. Qb3+ Ke2 60. Kf4 Kd2 61. Qc4 Ke1 62. Qa2 Kd1 63. Qf2 Kc1 64. Qa2 Kd1 65. Qf2 Kc1 66. Qa2 Kd1 67. Qh2 Kc1 68. Kf5 Kd1 69. Kf4 Kc1 70. Kf5 Kd1 71. Ke5 Kc1 72. Qa2 Kd1 1/2-1/2 [/pgn]

And Obsidian used tbs? And which one?

It was configured to use all 3,4,5-man wdl and dtz syzygy tablebases. It showed tb hits and searched more than 3 million nodes/s using one search thread.

I don't know what tbs it actually used though. Based on how it plays, maybe it is not using dtz tables and only relies on wdl tables.

[peter]

I was able to draw from that position as black against Obsidian 10.0. The time control was 60 moves in 3 minutes.
[pgn] [Event "?"] [Site "?"] [Date "2024.01.19"] [Round "?"] [White "Obsidian 10.0"] [Black "Player"] [Result "1/2-1/2"] [FEN "8/5k2/5r2/8/2KQ4/8/8/8 w - - 0 1"] [SetUp "1"] 1. Qd7+ Kg8 2. Qe8+ Kg7 3. Qe7+ Kg6 4. Kd5 Rf4 5. Qa3 Rf5+ 6. Ke4 Rf6 7. Qg3+ Kf7 8. Qb3+ Kg6 9. Qg8+ Kh5 10. Ke5 Rf3 11. Qg2 Rf8 12. Qh3+ Kg5 13. Qg3+ Kh6 14. Qh4+ Kg6 15. Qg4+ Kf7 16. Qf5+ Kg7 17. Qg5+ Kh7 18. Ke6 Rg8 19. Qh5+ Kg7 20. Ke7 Ra8 21. Qe5+ Kg6 22. Qe4+ Kg5 23. Qxa8 Kf4 24. Qd5 Ke3 25. Kf6 Ke2 26. Qc4+ Ke3 27. Qc3+ Ke4 28. Qd2 Kf3 29. Qc2 Ke3 30. Qc3+ Ke4 31. Qd2 Kf3 32. Qc2 Ke3 33. Ke5 Kf3 34. Qd2 Kg3 35. Qc2 Kf3 36. Qd2 Kg3 37. Qc2 Kf3 38. Qb2 Ke3 39. Qc3+ Ke2 40. Ke4 Kf2 41. Qc2+ Kg3 42. Qc6 Kf2 43. Qc2+ Kg3 44. Qc6 Kf2 45. Qc4 Kg3 46. Ke5 Kf3 47. Qc3+ Ke2 48. Ke4 Kf2 49. Qc4 Kg3 50. Ke5 Kf3 51. Qc6+ Ke3 52. Kf5 Kd4 53. Kf4 Kd3 54. Ke5 Ke3 55. Kf5 Kd4 56. Kf4 Kd3 57. Ke5 Ke3 58. Qe6 Kd3 59. Qb3+ Ke2 60. Kf4 Kd2 61. Qc4 Ke1 62. Qa2 Kd1 63. Qf2 Kc1 64. Qa2 Kd1 65. Qf2 Kc1 66. Qa2 Kd1 67. Qh2 Kc1 68. Kf5 Kd1 69. Kf4 Kc1 70. Kf5 Kd1 71. Ke5 Kc1 72. Qa2 Kd1 1/2-1/2 [/pgn]

And Obsidian used tbs? And which one?

It was configured to use all 3,4,5-man wdl and dtz syzygy tablebases. It showed tb hits and searched more than 3 million nodes/s using one search thread.

I don't know what tbs it actually used though. Based on how it plays, maybe it is not using dtz tables and only relies on wdl tables.

Anyhow, with my 6men tbs- installation no problem with letting the engine play for White against me, when I was always chosing the move with longest possible DTM for Black, Obsidian had 6men, a single thread and 10"/move:

[pgn] [Event "?"] [Site "?"] [Date "2024.01.19"] [Round "?"] [White "Obsidian 10.0"] [Black "Player"] [Result "1-0"] [SetUp "1"] [FEN "8/5k2/5r2/8/2KQ4/8/8/8 w - -"] 1. Qd7+ Kg8 2. Qe8+ Kg7 3. Qe7+ Kg6 4. Kd5 Rf4 5. Qa3 Rf5+ 6. Ke4 Rf6 7. Qg3+ Kf7 8. Qb3+ Kg6 9. Qg8+ Kh5 10. Ke5 Rf3 11. Qg2 Rf8 12. Qh3+ Kg5 13. Qg3+ Kh6 14. Qh4+ Kg6 15. Qg4+ Kf7 16. Qf5+ Kg7 17. Qg5+ Kh7 18. Ke6 Rg8 19. Qh5+ Kg7 20. Ke7 Ra8 21. Qe5+ Kg6 22. Qe4+ Kg5 23. Qxa8 Kf4 24. Qd5 Ke3 25. Kf6 Ke2 26. Qc4+ Ke3 27. Qc3+ Ke4 28. Kg5 {+M21/12 2s} Kd5 {34s} 29. Kf5 {+M22/1 0s} Kd6 {16s} 30. Qc4 {+M22/1 0s} Kd7 {10s} 31. Qc5 {+M13/1 0s} Ke8 {11s} 32. Ke6 {+M12/1 0s} Kd8 {9s} 33. Qc4 {+M6/1 0s} Ke8 {13s} 34. Qc8# {+M1/1 0s} 1-0 [/pgn]
Then I gave it a second try with same conditions except not loading tbs at all. After restart of GUI from 28th move onwards again, result was just the same as above. Obsidian of course should be able and was at my trials to find such short distances to mate without tbs as well as with them.
As told I cannot try with 5men subset, have only complete 3,4,5,6men in one file, dtz in a second and wdl in a third one, regards

[mvanthoor]

Hi! Thank you, and thanks for taking the time to do these tests, your results are very interesting!

Indeed. I have noticed in the past that some authors pile lots of features on top of their engine as fast as possible to climb the rating list, without measuring if the feature actually gains the expected amount of strength. Thus I've been testing this religiously in my engine (I even test refactors), and keeping scores of newly developed engines how many points they gain per feature.

(I dislike it when authors just throw out new versions of engines without actually keeping a changelog, or they add like 10 features at once gaining 500 points in one go without describing what they did. Others can't learn anything from that. It feels like someone just wanting to climb the rating list as fast as possible. That's fine of course, but I don't follow these engines because I can't learn anything from them.)

Therefore I also test new engines against my own, and if one has a massive feature set while not being noticeably stronger (or a similar feature set with being noticeably weaker), then I run a test and write a report like this.

The feature sets do differ slightly, it looks to me like Rustic uses pseudo-legal move generation while Tcheran uses fully legal move generation and so does not have to check the legality of moves during search. But it is pretty amazing how similar the feature sets are otherwise!

In the end, move generation makes up only 5-10% of the code use, so even if the move generator is a bit faster, it wouldn't matter more than 10 Elo or so. A move generator twice as fast, that would mean 50-70 Elo. That'd be worth it.

I have tried running them both on my machine (an M1 Macbook) as I was also curious about whether the results would look any different, assuming you're running on x86. In my tests, Rustic runs at ~7mnps and Tcheran runs at ~11mnps. Much slower than your machine but the ratio is roughly the same. I have tweaked my node counting locally to match Rustic's approach (and fixed the double-counting of the first q-search nodes) so I believe that comparison is accurate.

So you are saying that if you match Tcheran's node counting to Rustic's, your engine is 50% (!) faster than mine? That would be massive. Could it be that your full legal move generator makes the engine 50% faster? If that is true, I'll have to look at this at some point, because it would gain at least 35 Elo. With regard to node counts... it's hard to measure engine speed by that, because everybody does it differently. Still, if I could scrap the legality check (and maybe even the is_check function, that uses the same square_attacked function) from the engine after the move generator is fully legal, then it could maybe become 50% faster. It could also still just be a difference in node counts. First though, I'll look into easier to add features, including staged move generation.

It looks to me like Tcheran is getting through nodes faster than Rustic, but that Rustic has to analyse many less nodes for any particular depth. For example, Tcheran v1.1 explores 165,787,095 nodes at depth 9 from the start position, while Rustic only has to explore 9,268,090 (~18x less!)

I have spent a lot of time doing profiling and optimisation - far, far too much, probably, the month between v1.0 and v1.1 was spent almost exclusively on that. I should've been spending more time on pruning techniques & the evaluation function but unfortunately I find it quite addictive Sadly as a result I'm not sure there's much low hanging fruit (or even much high-hanging fruit). But I am very curious to know how Rustic is able to search so few nodes - comparatively - at each depth despite having such a similar feature set. I did notice that Tcheran's seldepth at depth 9 is 24 while Rustic's is 23, so perhaps there's something to that, but even at depth 8 (which goes to 23 in quiescence) Tcheran is searching ~35m nodes which is still almost 4x more than Rustic searches at depth 9 (!).

Our feature-set is the same, and the only two node-cutting features both engines have are PVS and the Transposition Table. The TT adds 150-160 Elo which is... uh... exact the amount you seem to be missing. (PVS added 54 Elo in self-play in my case.) Off the bat, my guess would be that your transposition table is having an issue. (I did check that the setting was indeed 256 MB when running this test. At least, Tcheran showed about 258.4 MB in use during this run.)

edit: even this could be your node count. Obvioiusly, I could search "more" nodes by moving my node count up. Rustic counts the node right after the q-search call, so it does NOT count any nodes that get aborted by things like out of time. (q-search obviously also counts its nodes.)

My suggestion would be: disable your PVS and Transposition Table to get a base version. You can test this against Rustic Alpha 1. (Just use the latest version; they just update crates and lints to keep up with Rust.) They should be about equal. Then run this base version against a version that has a TT (added ~150 Elo for me in self-play) and then add the PVS (added ~54 Elo for me). If your gains are a lot lower, you probably have a bug somewhere.

Pretty incredible that it turned out that close!

I hope it stays that way in the CCRL list. I don't know how their new calibration is going to affect engine placement. The range in the list is stretched now. (Before, it was compressed.) We'll see. If I don't gain at least 250 points with Rustic 4, I'll be... disapPOINTed. Eh. Sorry. Couldn't resist.

You've certainly given me a lot of food for thought! Thanks again for doing these tests and good luck for the release of Rustic 4

Thanks. Good luck I've just recently picked up Rustic's development again, after an almost 3 years (...) hiatus in which I only massively refactored the code. In hindsight... Alpha 1-3 are not really... Rust-like. It's more C-with-Rust-syntax.

I'll be watching your engine (as I'm watching most Rust engines that don't crash on me when I start them).

[Graham Banks]

Good evening Graham,

yes, back to v3.0?
Maybe the programmer found a bug ...

I downloaded v4.0 but will wait with a test.

I think better is if the programmers used this thread only?

Best
Frank

Starzix 4 is there now, Frank.