Progress on Rustic

[mvanthoor]

...

Confirmed. Rustic is losing on time. When using controls such as move time = 5 seconds, it uses 100% of the time. Overhead protection probably just got lost during refactoring when I was jacking around with the time management a few weeks ago, to prepare Alpha 1. As said, in incremental controls, this is never a problem, as Rustic does things such as using only a percentage of its time, and stops overshooting allocated time, if its time left becomes lower than a certain value.

I'll fix it and backport the fix to Alpha 1 => 1.1. (PS: Rustic Alpha 1 was a stupid name. It should have been Rustic 1 Alpha. Ah, well. Doesn't really matter. At some point, the Alpha part will be dropped. It's there because I don't consider the engine finished yet.)

By the way, I have a few bugs with regard to belofte to report in return; at least, in version 2.1.0. (There is no exe yet for version 2.1.1; I try to not compile other engines, and prefer to use what the author thinks is the best compile).

- It either has no qsearch, or it is very buggy. The engine either sacrifices pieces for no reason, or leaves pieces hanging. I stopped testing after 10 games.
- The engine often shuffles pieces back and forth; so in effect, it does nothing for several moves.
- Sometimes the engine just starts pulling its pieces -backward-, un-developing them?!
- At a depth of let's say 4 ply, my engine plays instantly (it goes up to depth 7 in under 2 seconds on my system; obviously newer versions will search even deeper in less time). Your engine, when playing at a fixed depth of 4, sometimes sits there for multiple minutes before it makes a move. Either you have only mini-max and no alpha-beta, no move sorting, or you have bugs in the search. If your engine needs minutes to reach depth 4, you'll forever be stuck at 1200-1300 Elo thereabouts. As I can't see what the engine is doing (no UCI output), I can't determine speed or depth, and I'm not going to wait minutes per move for the engine to play... sorry.

I would be happy to test your engine against Rustic and report weak points and strange things in normal play, but you have to fix the above issues first. With these issues in place, it's no use for me to test the engine because it doesn't play normal games from beginning to end, or its going to take huge amounts of time on a fixed depth. Sometimes a game is basically over by move 9, because right after the opening, Belofte decides to go Qc7xh2, Rustic goes Kg1xh2, and from there on, playing the game to the end is virtually pointless because Belofte is 800 cp down.

Maybe you did fix these things in version 2.1.1; and if your post above actually compares version 2.1.0 vs. 2.1.1, and the 2.1.1 is a few hundred Elo stronger because of the fixes, I can certainly believe that. If you can make the 2.1.1 binary available, I'll try and test it.

[ydebilloez]

Confirmed. Rustic is losing on time. When using controls such as move time = 5 seconds, it uses 100% of the time. Overhead protection probably just got lost during refactoring when I was jacking around with the time management a few weeks ago, to prepare Alpha 1. As said, in incremental controls, this is never a problem, as Rustic does things such as using only a percentage of its time, and stops overshooting allocated time, if its time left becomes lower than a certain value.

Hint, in belofte I use 50 ms overhead protection to act on ? / movenow command and remove it from available time. Take care to flush output on bestmove (std::endl in C++) so output gets sent directly and is not buffered.

I'll fix it and backport the fix to Alpha 1 => 1.1. (PS: Rustic Alpha 1 was a stupid name. It should have been Rustic 1 Alpha. Ah, well. Doesn't really matter. At some point, the Alpha part will be dropped. It's there because I don't consider the engine finished yet.)

No release required for me. It was just a bug report. But if you fix this, I will certainly use the updated version.

- It either has no qsearch, or it is very buggy. The engine either sacrifices pieces for no reason, or leaves pieces hanging. I stopped testing after 10 games.
- The engine often shuffles pieces back and forth; so in effect, it does nothing for several moves.
- Sometimes the engine just starts pulling its pieces -backward-, un-developing them?!
- At a depth of let's say 4 ply, my engine plays instantly (it goes up to depth 7 in under 2 seconds on my system; obviously newer versions will search even deeper in less time). Your engine, when playing at a fixed depth of 4, sometimes sits there for multiple minutes before it makes a move. Either you have only mini-max and no alpha-beta, no move sorting, or you have bugs in the search. If your engine needs minutes to reach depth 4, you'll forever be stuck at 1200-1300 Elo thereabouts. As I can't see what the engine is doing (no UCI output), I can't determine speed or depth, and I'm not going to wait minutes per move for the engine to play... sorry.

Most issues are fixed in 2.1.1. To compile out of repository, just fetch the 2.1.1 (or when merged the trunk branch) and run ./compile.sh in the project folder. (or make) Version 2.1.0 uses a handicapped AB that resembles more to mini-max.
Version 2.1.1 should be released this week.

There is not yet something like developing pieces, just scores on certain fields for certain pieces.

Thanks to your remarks, I have been reading again through the xboard sd/uci depth specification and indeed, I am incorrectly using it as full search depth while it should be set as maxdepth. Will be fixed in 2.1.1 and probably back-ported to 2.1.0.

For your information, currently, the SD is used as a level, setting time to infinite, and setting search depth incrementally from 1 to x... but setting quiescence to x times 4 with a minimum of 8. Setting depth to 4 caused trees to go to 16! Expect a fix very very soon.

Currently, UCI output must be turned on with post command or debug verbose x, will be fixed as well so default there is at least some output. Running in xboard mode does give output.

[mvanthoor]

For your information, currently, the SD is used as a level, setting time to infinite, and setting search depth incrementally from 1 to x... but setting quiescence to x times 4 with a minimum of 8. Setting depth to 4 caused trees to go to 16! Expect a fix very very soon.

This is normal. I often see a 10-12 ply extension for QSearch in Rustic as well. It's not a problem. (It's actually what QSearch is intended for: making the position totally quiet.)

Currently, UCI output must be turned on with post command or debug verbose x, will be fixed as well so default there is at least some output. Running in xboard mode does give output.

UCI doesn't have something like "post" or "nopost". I would advise to not mix the protocols; it's not going to work. A GUI is using one or the other, not both.

PS: My UCI / XBoard implementation (XBoard not finished yet) are built on top of the same engine infrastructure. My engine has a -quiet command line option, which the engine also uses for post/nopost in XBoard. I use it with "./rustic -q" to suppress the intermediate search stat updates, so I only see an update every depth, when quickly analyzing a position on the command line.

[ydebilloez]

This is normal. I often see a 10-12 ply extension for QSearch in Rustic as well. It's not a problem. (It's actually what QSearch is intended for: making the position totally quiet.)

Question, what are you doing if depth parameter on go is issued? Limit QSearch to that depth or let run QSearch to position that is quiet?
If I read the specification, I assume that even QSearch should be cut at depth... but this will result in erratic play.

UCI doesn't have something like "post" or "nopost". I would advise to not mix the protocols; it's not going to work. A GUI is using one or the other, not both.
PS: My UCI / XBoard implementation (XBoard not finished yet) are built on top of the same engine infrastructure. My engine has a -quiet command line option, which the engine also uses for post/nopost in XBoard. I use it with "./rustic -q" to suppress the intermediate search stat updates, so I only see an update every depth, when quickly analyzing a position on the command line.

Xboard commands can be used even if uci mode has been selected in my engine. Same for perft command that does not exist in uci neither... I merely use it in test scripts or on the command line. But when typing help, the hidden commands will not show. Normally you should be able to use --post on the command line (or --nopost as an alternative on -quiet) in belofte. Any command not taking parameters can be issued directly from the command line. e.g. belofte --uci --nopost --ucinewgame --isready

Some information I forgot to share, belofte 2.1.1 is about 400 elo stronger in bullet controls compared to 2.1.0. Not compared to rustic.

[mvanthoor]

Question, what are you doing if depth parameter on go is issued? Limit QSearch to that depth or let run QSearch to position that is quiet?
If I read the specification, I assume that even QSearch should be cut at depth... but this will result in erratic play.

As far as I know, UCI only limits the search depth, not the number of ply's used within that depth. QSearch is an extension on top of the search depth, just like the check extension is. So you go to (say) depth = 4, but because of check extensions and/or QSearch, you could end up 10 plies deep. AFAIK, UCI does not limit this.

Xboard commands can be used even if uci mode has been selected in my engine. Same for perft command that does not exist in uci neither... I merely use it in test scripts or on the command line. But when typing help, the hidden commands will not show. Normally you should be able to use --post on the command line (or --nopost as an alternative on -quiet) in belofte. Any command not taking parameters can be issued directly from the command line. e.g. belofte --uci --nopost --ucinewgame --isready

Some information I forgot to share, belofte 2.1.1 is about 400 elo stronger in bullet controls compared to 2.1.0. Not compared to rustic.

OK; that's possible. You can handle non-UCI commands in UCI, and non-XBoard commands in XBoard, obviously. So if you want to have a post/nopost command in UCI you can (which is akin to my --quiet parameter, indeed), but don't expect that a GUI will ever issue it when communicating with a UCI engine.

[hgm]

The depth limits in UCI and CECP (go depth / sd) are generally interpreted as a limit to the nominal depth (which is what is reported in the thinking output while iterating in the root). Not as a limit to the selective depth.

[mvanthoor]

The transposition table is in, and it works in both Perft and Search, and can be disabled by setting a hash size of 0 MB.

Some preliminary results:
TT Move sorting: shrinks the search tree by 20-40%, depending on the position and game phase.
TT Search cuts: Rustic+TT outcalculates Rustic without TT (same otherwhise) by 1-4 ply, depending on the position and game phase.
Probing the TT slows the engine down by 500 kN/s on an i7-6700K, which is about 10-15% depending on the position.
Elo difference in self-play: 100 Elo, +/- 50 Elo error bars (not enough games yet).

Mate scoring is not being adjusted for yet. The engine now botches many simple endgames, where it (for example) states "mate in 5" and then actually doesn't mate. Quite a few unnecessary draws. I've seen the difference in rating drop by 15 to 20 points sometimes, when the engine draws a game it should have won, and is in front a lot of Elo. (Drawing / losing against a weaker opponent costs a lot of points.) I expect another jump in rating when I fix this. That'll be over the weekend.

PS: Big refactoring as well. The engine is now tidy again. Also: the hash table is a generic. Same code used for both Perft and Search, but with different Entry types.

After the TT is completely finished and tested, I'll have to make the UCI options for it, and then Alpha 2 will be done.

[mvanthoor]

I've now added hash move sorting (I could have postponed this to Alpha 3, where I'm going to do more sorting), and it seems to decrease the number of visited nodes, and the time needed to reach some specific depth. In the start position, "go depth 9" decreases from 51 seconds to 44 seconds. When playing games against a version of Rustic that does not yet sort hash moves however, the strength difference seems to be zilch; the version with hash move sorting only occasionally reaches a ply of extra depth.

Previously, I only searched the hash-table when not in the root, because cutting on the TT-value in the root sometimes got me an illegal move and no PV. Now I do probe the hash table in the root to get the hash move, but don't cut on the root anymore; this solves the illegal move problem, and gives the engine the opportunity to sort the hash move; which it does, which makes it search less and faster, as described above, but doesn't gain any strength.

Strange, as sorting the has move should be the same as sorting the best move of the previous iteration in my engine, as I put each move that increases alpha (where the PV is also updated) into the hash table. I had expected more from this to be honest.

[mvanthoor]

An example, KiwiPete position, search to depth 10.

MVV-LVA, no TT Move ordering:

Code: Select all

info score cp 25 depth 1 seldepth 16 time 0 nodes 4271 nps 0 pv e2a6 b4c3 b2c3 e6d5
info score cp 25 depth 2 seldepth 16 time 1 nodes 8576 nps 8576000 pv e2a6 b4c3 b2c3 e6d5
info score cp 20 depth 3 seldepth 20 time 2 nodes 23305 nps 11652500 pv e2a6 e6d5 a6b7 b4c3 b7a8 c3d2
info score cp 20 depth 4 seldepth 20 time 6 nodes 58083 nps 9680500 pv e2a6 e6d5 a6b7 b4c3 b7a8 c3d2
info score cp 5 depth 5 seldepth 22 time 26 nodes 220026 nps 8462538 pv e2a6 b4c3 d2c3 e6d5 e4d5 f6d5
info score cp 15 depth 6 seldepth 22 time 102 nodes 748591 nps 7339127 hashfull 2 pv e2a6 e6d5 c3d5 b6d5 a6b7 a8d8 b7d5 f6d5
info score cp 5 depth 7 seldepth 24 time 435 nodes 3284474 nps 7550515 hashfull 9 pv e2a6 e6d5 c3d5 f6d5 e5c4 f7f5 c4b6 a7b6
info score cp -15 depth 8 seldepth 26 time 1901 nodes 12742578 nps 6703092 hashfull 54 pv e2a6 e6d5 c3d5 f6d5 e5d3 f7f5 g2h3 f5e4
info score cp -40 depth 9 seldepth 27 time 9617 nodes 64729880 nps 6730777 hashfull 274 pv d5e6 e7e6 e2a6 e6e5 g2h3 b4c3 d2c3 e5e6 c3f6 g7f6
info score cp -40 depth 10 seldepth 29 time 43549 nodes 276216520 nps 6342660 hashfull 883 pv d5e6 e7e6 e2a6 e6e5 g2h3 b4c3 d2c3 e5e6 c3f6 g7f6

MVV-LVA, put TT Move before any other move:

Code: Select all

info score cp 25 depth 1 seldepth 16 time 0 nodes 4271 nps 0 pv e2a6 b4c3 b2c3 e6d5
info score cp 25 depth 2 seldepth 16 time 1 nodes 8576 nps 8576000 pv e2a6 b4c3 b2c3 e6d5
info score cp 20 depth 3 seldepth 20 time 2 nodes 23305 nps 11652500 pv e2a6 e6d5 a6b7 b4c3 b7a8 c3d2
info score cp 20 depth 4 seldepth 20 time 6 nodes 55845 nps 9307500 pv e2a6 e6d5 a6b7 b4c3 b7a8 c3d2
info score cp 5 depth 5 seldepth 20 time 24 nodes 217885 nps 9078542 pv e2a6 b4c3 d2c3 e6d5 e4d5 f6d5
info score cp 15 depth 6 seldepth 22 time 98 nodes 727764 nps 7426163 hashfull 2 pv e2a6 e6d5 c3d5 b6d5 a6b7 a8d8 b7d5 f6d5
info score cp 5 depth 7 seldepth 24 time 404 nodes 3103598 nps 7682173 hashfull 8 pv e2a6 e6d5 c3d5 f6d5 e5c4 f7f5 c4b6 a7b6
info score cp -15 depth 8 seldepth 26 time 1767 nodes 11859116 nps 6711441 hashfull 51 pv e2a6 e6d5 c3d5 f6d5 e5d3 f7f5 g2h3 f5e4
info score cp -40 depth 9 seldepth 27 time 8832 nodes 59641169 nps 6752850 hashfull 255 pv d5e6 e7e6 e2a6 e6e5 g2h3 b4c3 d2c3 e5e6 c3f6 g7f6
info score cp -40 depth 10 seldepth 30 time 39769 nodes 255350305 nps 6420838 hashfull 861 pv d5e6 e7e6 e2a6 e6e5 g2h3 b4c3 d2c3 e5e6 c3f6 g7f6

As can be seen, the number of searched nodes is clearly less, and time to depth is faster:

depth 8: 12.74 M (1.90s) => 11.86 M (1.77s)
depth 9: 64.72 M (9.60s) => 59.64 M (8.83s)
depth 10: 276.2 M (43.6s) => 255.4 M (39.77s)

And, I can clearly see the version with TT Move ordering often gain 1 ply, and in the late middle game/end game it gains 2 ply. In practice / self-play, there doesn't seem to be any difference btween the versions though. It's all within the margin of error.

I also see that both versions sometimes do really weird things, which they also did before, when reaching higher depths; now they just start doing it earlier: such as postponing a capture by checks, in-between moves, and tactical round-about antics, finally making the capture in some sort of weird, unexpected way. This is something I can't explain. No human would ever do something like this. Why would you leave a rook that you can just capture with a bishop (which would for all intents and purposes, be a normal move) hanging on a square, go on a wild checking-spree, to finally fork the enemy king and rook, so you can capture the rook with the knight? The only reason I can explain is that, because of the higher search depth, the engine has determined that having the knight on that square instead of the bishop, gains 0.05 from the PST's. For a human, that would be a VERY risky thing to try to do, and it makes for very strange games to look at.

IT could be because there isn't much of an evaluation yet, which basically makes any move that doesn't win material or detract from the PST-score "good enough". Maybe I should first finish the hash table and move sorting, and then write the Texel tuner, all of which will make moves more different from one another.

[mvanthoor]

I've tested a bunch of positions from the perft results wiki page. It doesn't matter which position I test, the difference is measurable and in some positions, quite pronounced, where de time to depth 9 drops with 60% (!); however in self-play the version with TTMove ordering doesn't seem to be stronger; not even on longer time controls.

Could be that I need a bunch of move ordering optimizations on top of one another before I'm actually going to see a strength increase. Maybe 1-2 ply here or there on short time controls such as 1m+0.6 seconds isn't enough. On longer time controls (and with better time management, probably), the difference would be more pronounced.

Maybe I'm expecting too much of small improvement such as these, on very short time controls.