Progress on Rustic

[mvanthoor]

This evening I've had to do a few fixes to make the transposition table usable on 32-bit systems:

1. Set the transposition table size to a maximum of 2048 MB (and enforce this, obviously)
2. Fix a few "usize" types to "u64" ("usize" is 32-bit on a 32-bit system, 64-bit on a 64-bit system)
3. Point 2 also fixed an integer wrap-around in the TT percentage used calculation.

Overnight, I'm going to run a match between Alpha 2 64-bit and 32-bit. If everything is OK (= no crashes), I'll make a release tomorrow evening.

Time control in Rustic alpha seems to be problematic if selecting st=5 in cutechess. (5 seconds per move).
Cutechess-cli is very strict and a few ms passed the allotted time causes a timeout.
Will this get fixed in next release or even in a patch release on alpha?

I forgot to subtract an overhead from the move time per second. At some point the GUI would realize that the move comes in late (maybe only 0.01 seconds) and forfeit the game. In MoveTime mode, Rustic Alpha 1 was losing all of its games against Belofte 2.1.0. I've decided to also include this fix into Alpha 1. This version, Alpha 1.1, is now playing a 200 game match against Belofte 2.1.0, and after 21 games, it is up 21 - 0, instead of down 0 - 21. The problem seems to be fixed.

Engines just don't take this mode into account for testing, apparently. (I didn't, either.), I tested 3 other engines (Clueless, Fractal, and Pulse) and they all lose games on time in MoveTime mode.

So, if there are no issues in the overnight matches, I'll be creating an Alpha 1.1 and Alpha 2 release tomorrow.

[mvanthoor]

Hi

Rustic Alpha 2 has been released:

https://github.com/mvanthoor/rustic/releases

In this release, the transposition table, and sorting on the TT Move have been implemented.
The expected rating increase is about 105 Elo compared to Alpha 1.

New Features:
- Transposition table for search and perft.
- Ordering on transposition table move.
- Set TT size through --hash option or UCI parameter.

Improvement:
- Move check extension higher up in the search routine, to prevent
quiescence search while in check.

Changes:
- seldepth: report max ply reached during the search, instead of
selective depth at last completed iteration.
- Count all nodes visited, instead of only nodes which generated moves.
- Change random number generator from SmallRng to ChaChaRng for
reproducible behavior between platforms/OS's/architectures/versions.

Cleanup
- Change Root PV handling to remove redundant code.
- Miscellaneous small renames, refactors, and cleanups.
- Add rand_chacha and remove SmallRng number generators.
- Update Rand library to 0.8.3.

I've also released version Alpha 1.1, which fixes the forfeits on time in MoveTime mode. This fix is obviously also in Alpha 2. I'll add the binaries for Alpha 1.1 tomorrow (probably).

...

@ The CCRL people: thanks again for testing, whomever is going to do this. As lithander said somewhere else: without you, it wouldn't be half as fun

[mvanthoor]

Alpha 2 release retracted due to possibly massive bug in the TT implementation.

I kept suspecting something was wrong, and it seems there is:

http://talkchess.com/forum3/viewtopic.p ... 44#p887342

I'll have to change this and retest again. The next release will have to wait a week or so.

[mvanthoor]

Well. I fixed the bug yesterday evening, and had my computer run a 1000 game gauntlet, with the following results:

Code: Select all

0 Rustic Alpha 2 TT Fix          65      20    1000   59.3%   17.8% 
1 Deepov 0.4                     28      60     100   54.0%   24.0% 
2 Clueless 1.4                   14      67     100   52.0%    6.0% 
3 Pigeon 1.5.1                    7      55     100   51.0%   36.0% 
4 Wukong JS 1.4                 -35      63     100   45.0%   18.0% 
5 CDrill Build 4                -42      66     100   44.0%   10.0% 
6 TSCP 1.81                     -96      66     100   36.5%   15.0% 
7 Celestial 1.0                -104      68     100   35.5%   11.0% 
8 Shallow Blue 2.0             -127      69     100   32.5%   11.0% 
9 FracTal 1.0                  -160      60     100   28.5%   31.0% 
10 Mizar 3                     -164      69     100   28.0%   16.0% 

1000 of 1000 games finished.

Code: Select all

Rank Name			Elo    +    - games score oppo. draws
   1 Clueless 1.4		1874   43   42   200   60%  1789    8%
   2 Deepov 0.4			1854   40   40   200   58%  1789   20%
   3 Rustic Alpha 2 rc6		1843   18   18  1000   59%  1773   18%
   4 Pigeon 1.5.1		1824   38   38   200   55%  1789   35%
   5 Wukong JS 1.4		1824   40   40   200   54%  1789   17%
   6 CDrill Build 4		1801   42   41   200   51%  1789   11%
   7 Rustic Alpha 2 rc5.1	1793   23   23   600   57%  1741   20%
   8 TSCP 1.81			1742   41   41   200   44%  1789   13%
   9 Celestial 1.0		1730   40   41   200   43%  1789   17%
  10 Shallow Blue 2.0         	1708   41   42   200   40%  1789   14%
  11 FracTal 1.0		1706   38   39   200   38%  1789   35%
  12 Rustic Alpha 1           	1677   23   24   600   42%  1741   19%
  13 Mizar 3                  	1667   41   43   200   34%  1789   15%
  14 MinimalChess 0.3         	1582   40   41   200   29%  1735   25%

Very nice; Alpha 1 scored -81 in this pool. Alpha 2 rc5.1 (with the TT bug) scored +15. Alpha 2 rc6 without the TT bug scored +65. Total Elo gain of Alpha 2 over Alpha 1, by adding the TT + TT Move ordering, is 166 Elo.

I expect the engine to score somewhere around 1840 in CCRL Blitz 2m+1s.

The goal of attaining Vice's rating of 2050, with AT MAXIMUM feature parity and NO evaluation terms besides material and PST's, seems attainable now, if I get the benefits with each feature I see in other engines.

In Alpha 3, I'll add, killer moves and history heuristics, aspiration windows, and PVS. (expectation: +50 Elo). Rustic 4 should have tuned PST's and material value (expectation +80), version 5, Null move (expectation +100 Elo). That would put Rustic 5 at 2070 Elo. Feature-wise it would be on par with Vice, but without any of its evaluation terms (but tuning in place of that).

Maybe everything shifts one version up if I decide to do aspiration windows and PVS in a seperate version. Depends on the gains made by the sorting functions.

In the next gauntlet, I'll be replacing Mizar, Fractal, Celestial and ShallowBlue, as these engines are now performing at more than 100 Elo below Rustic. For example, Fractal's and Shallow Blue's rating hardly changed. Playing games against them doesn't gain a lot of Elo points anymore; it just costs time. (For some reason, Fractal and Celestial are still heavily overrated. They play better against Rustic in my own tournaments, especially Fractal, as they do in CCRL. Must be the opening book or the shorter time control.)

[lithander]

Very nice; Alpha 1 scored -81 in this pool. Alpha 2 rc5.1 (with the TT bug) scored +15. Alpha 2 rc6 without the TT bug scored +65. Total Elo gain of Alpha 2 over Alpha 1, by adding the TT + TT Move ordering, is 166 Elo.

Very nice. Finally all the work put into adding TT got rewarded by some significant strength gains! Do you have statistics on how much deeper you can search now? How your average branching factor changed?

I've spent hours trying to improve the move ordering (without using TTs) of MinimalChess 0.3 to no avail. I have thought about trying SEE next to improve upon MVV-LVA... did you consider it? I don't see it mentioned in your roadmap.

[Ras]

I've spent hours trying to improve the move ordering (without using TTs) of MinimalChess 0.3 to no avail.

What about quiet move history and depth killers?

[mvanthoor]

Very nice. Finally all the work put into adding TT got rewarded by some significant strength gains! Do you have statistics on how much deeper you can search now? How your average branching factor changed?

I've spent hours trying to improve the move ordering (without using TTs) of MinimalChess 0.3 to no avail. I have thought about trying SEE next to improve upon MVV-LVA... did you consider it? I don't see it mentioned in your roadmap.

In the end, if you have your terminology straight (i.e., "best move" is any move that improves the evaluation, not the move that last raised alpha), the TT is not that difficult to implement. (Except if you're doing it in Rust and are working with traits, trait objects, and generics, all at the same time, for the first time...)

I don't have any statistics. Maybe I should implement something for that at some point, and then backport it to all versions...

"Play best move from the first iteration first" is the largest move ordering optimization you can make, and you have that already (in my case, it comes with the TT, as far as I can tell). Then comes MVV-LVA (which can be improved by SEE), then Killers, then History Heuristic. If you don't have the latter two, you could implement them. In other engines they reduce the search tree to about half the size as compared to MVV-LVA only, and they're not very hard to implement. You can also reduce the tree size by Aspiration Windows, and Principle Variation Search. (Guess what will be on the menu for Alpha 3 (maybe 4, depending on the gains per feature)? Killers, History, Aspriation Windows, and PVS...)

I'll look into SEE later. I consider that a more advanced feature, as it optimizes MVV-LVA, which is already an optimization of the move ordering. There's much to implement before I get to that

Yesterday I saw I made a mistake. My gauntlet pool has been running the other engines with their default TT sizes (if they have any); but fortunately, they did so for both Alpha 1 and 2, so the rating increase of the engine is still correct, because testing conditions are the same.

For Alpha 3, I'm going to create a completely new pool with 1750+ engines only, because the sub-1700 engines are losing too many games now. And as you've noticed, it's harder to find stable engines (or even WORKING engines for that matter) than you might think...

[lithander]

For Alpha 3, I'm going to create a completely new pool with 1750+ engines only, because the sub-1700 engines are losing too many games now. And as you've noticed, it's harder to find stable engines (or even WORKING engines for that matter) than you might think...

I can give you a build of MinimalChess 0.3 with PeSTO evaluations that plays in that range. It's interesting because the search is still shallow but the PSTs are just very good. I won't release it officially (because I don't want to copy the full evaluation of another engine outside of a temporary experiment) but you can play it if you want.

I've spent hours trying to improve the move ordering (without using TTs) of MinimalChess 0.3 to no avail.

What about quiet move history and depth killers?

Haven't tried that yet! I'll give it a shot! Googling "Quiet move history" didn't turn up many results, though. Do you have a link?

I just thought I could use the better, tapered evaluation of PeSTO to sort moves based on the rating of the position they would produce but that didn't work out. And I tried to extend recording the principal variation well into the QSearch phase and continue to play the move from the PV first if available but that also didn't work out. And I tried Internal Iterative Deepening but I think my move generator is too slow for that to make it worth the additional cost at the rather low search depths (7-8) I achieve at the moment.

[mvanthoor]

I can give you a build of MinimalChess 0.3 with PeSTO evaluations that plays in that range. It's interesting because the search is still shallow but the PSTs are just very good. I won't release it officially (because I don't want to copy the full evaluation of another engine outside of a temporary experiment) but you can play it if you want.

Thanks; I'd like that. It's one more 1750+ engine I know will work (and will actually provide some output when I decide to watch a game or two). It's motivating to see that a good tapering/PST-implementation has boosted your engine by about 200-250 points. It means Rustic 4 may get a massive Elo boost again, if I can get this to work, without actually adding any features

[mvanthoor]

Haven't tried that yet! I'll give it a shot! Googling "Quiet move history" didn't turn up many results, though. Do you have a link?

Search for History Heuristic.