cutechess-cli 0.8.0 beta (with pondering)

mvk · Post by **mvk** » Wed Nov 11, 2015 1:00 am

ilari wrote:The fix is in! Here's the new beta version:

Are those the same versions as on github? (which doesn't identify itself as 0.8.0 BTW, but does have "ponder").

Reason I ask is I experience problems when the losing engine wants to ponder on a move that gets him checkmated (at the end of the game).

Cutechess sends "go ponder", but never sends "stop" or "ponderhit" when the winning engine plays the checkmate. Instead it sends "isready" and "ucinewgame" to the losing engine. By the time I get "position startpos" as well I stop the search thread, but that is when the search thread emits "bestmove 0000", because every "go" must be followed by a "bestmove". Cutechess doesn't expect that, considers that as an illegal move in the new game, and delivers another loss to the losing engine. Not fair.

I think it starts to go wrong where cutechess assumes that the reply to "isready" means that there is no search running, but that is not how it works. Searches are stopped by "stop" or "ponderhit", and must always give a "bestmove" reply, correct? (Pondering is uci is new for me also, and I try to follow what shredder does when the specs are unclear to me.). And "isready" during a search will always be replied to by "readyok". The uci spec is explicit about that.

I would prefer not to write code that avoids pondering on a game-ending move, if possible (could also be a stalemate).

The github version I'm using was copied on Nov/08. It seems to include such kind of fix, but I still see the above scenario.

MikeB · Post by **MikeB** » Wed Nov 11, 2015 6:15 am

That would be great except for one little detail. The two biggest platforms are Windows and Mac OS.

https://en.m.wikipedia.org/wiki/Usage_s ... ng_systems

ilari · Post by **ilari** » Wed Nov 11, 2015 12:45 pm

mvk wrote:
ilari wrote:The fix is in! Here's the new beta version:
Are those the same versions as on github? (which doesn't identify itself as 0.8.0 BTW, but does have "ponder").

The GitHub version is actually newer, but I haven't updated the beta binaries because the pondering feature hasn't seen updates since the last beta.

mvk wrote:Reason I ask is I experience problems when the losing engine wants to ponder on a move that gets him checkmated (at the end of the game).

Cutechess sends "go ponder", but never sends "stop" or "ponderhit" when the winning engine plays the checkmate. Instead it sends "isready" and "ucinewgame" to the losing engine. By the time I get "position startpos" as well I stop the search thread, but that is when the search thread emits "bestmove 0000", because every "go" must be followed by a "bestmove". Cutechess doesn't expect that, considers that as an illegal move in the new game, and delivers another loss to the losing engine. Not fair.

I think it starts to go wrong where cutechess assumes that the reply to "isready" means that there is no search running, but that is not how it works. Searches are stopped by "stop" or "ponderhit", and must always give a "bestmove" reply, correct? (Pondering is uci is new for me also, and I try to follow what shredder does when the specs are unclear to me.). And "isready" during a search will always be replied to by "readyok". The uci spec is explicit about that.

I would prefer not to write code that avoids pondering on a game-ending move, if possible (could also be a stalemate).

The github version I'm using was copied on Nov/08. It seems to include such kind of fix, but I still see the above scenario.

Sounds definitely like a bug in the CuteChess beta. Is there some free UCI engine that I can reproduce this problem with?

ilari · Post by **ilari** » Wed Nov 11, 2015 12:48 pm

MikeB wrote:That would be great except for one little detail. The two biggest platforms are Windows and Mac OS.

https://en.m.wikipedia.org/wiki/Usage_s ... ng_systems

I think in computer chess the biggest ones are Windows and Linux though. But don't worry, we're committed to supporting Mac OS as well. It's just the beta that's Windows/Linux only because I don't have a usable Mac at the moment. The final 0.8.0 is going to have a Mac version too.

mvk · Post by **mvk** » Wed Nov 11, 2015 7:52 pm

ilari wrote:Sounds definitely like a bug in the CuteChess beta. Is there some free UCI engine that I can reproduce this problem with?

Sure: this dev version of floyd https://github.com/kervinck/floyd/tree/dev
Specifically, commit: 0472c2912c394defdfd4157eddd9f7ef23389f69

After compiling, the binary should identify itself as "Version x61374713". Pondering is new and I haven't stress-tested it under other interfaces yet. I do have my doubts about this sequence though. The "bestmove 0000" is in response to the "position startpos" while still pondering (in fact: waiting for "stop" or "ponderhit" which never came).

Typical log file of this scenario: https://marcelk.net/2015-11-11/fponder.out.bz2

Relevant snippet:

Code: Select all

49912 <FloydA(0): bestmove f2f3 ponder h2h1q
49912 >FloydB(1): ponderhit
49912 >FloydA(0): position startpos moves e2e4 g7g6 c2c4 f8g7 d2d4 d7d6 f1e2 b8c6 c1e3 e7e5 d4d5 c6d4 g1f3 d4f3 e2f3 g7h6 e3h6 g8h6 h2h4 e8g8 b1c3 f7f5 h4h5 g6g5 d1d2 f5f4 e1c1 c8g4 f3g4 h6g4 h5h6 g8f7 d2e2 d8d7 c1c2 f7g6 c2b3 f8g8 a2a4 a7a6 e2f3 a8f8 a4a5 f8b8 b3a3 b7b6 a5b6 c7b6 f3h3 b8d8 d1f1 d7c8 b2b3 g6f6 h3f3 f6g6 f1c1 c8d7 c3a2 a6a5 a2c3 g4f6 f3e2 g5g4 h1h4 g6g5 c1h1 d7c8 a3b2 c8c5 g2g3 g8e8 c3b5 a5a4 h1a1 d8a8 a1a3 a4b3 b2b3 a8a3 b5a3 e8a8 a3b5 a8a1 h4h2 a1b1 b3a2 b1b4 b5a3 b6b5 a3b5 b4a4 a2b1 c5b4 e2b2 b4e1 b1c2 a4c4 b5c3 f6e4 c2b3 c4c3 b2c3 e1c3 b3a4 c3c4 a4a3 e4c5 g3f4 e5f4 h2h3 g4h3 a3b2 c5d3 b2b1 h3h2 f2f3 h2h1q
49912 >FloydA(0): isready
49912 <FloydB(1): info time 0 depth 0 score mate 1 nodes 7 nps 1174405 hashfull 0 pv h2h1q
49912 <FloydB(1): info time 0 depth 1 score mate 1 nodes 41 nps 1169840 hashfull 0 pv h2h1q
49912 <FloydB(1): bestmove h2h1q
49912 >FloydB(1): isready
49912 <FloydA(0): readyok
49912 >FloydA(0): go ponder wtime 4855 btime 4581 winc 250 binc 250
49912 >FloydA(0): isready
49912 <FloydB(1): readyok
49912 <FloydA(0): readyok
Finished game 1 (FloydA vs FloydB): 0-1 {Black mates}
Score of FloydA vs FloydB: 0 - 1 - 0  [0.000] 1
Started game 2 of 2 (FloydB vs FloydA)
49913 >FloydB(1): ucinewgame
49913 >FloydB(1): setoption name Ponder value true
49913 >FloydB(1): position startpos
49913 >FloydA(0): ucinewgame
49913 >FloydA(0): setoption name Ponder value true
49913 >FloydA(0): position startpos
49913 >FloydB(1): position startpos moves e2e4
49913 >FloydA(0): position startpos moves e2e4

[... snip ...]

49913 >FloydB(1): position startpos moves e2e4 g7g6 c2c4 f8g7 d2d4 d7d6 f1e2 b8c6 c1e3
49913 >FloydA(0): position startpos moves e2e4 g7g6 c2c4 f8g7 d2d4 d7d6 f1e2 b8c6 c1e3
49913 >FloydA(0): isready
49913 <FloydA(0): info time 0 depth 0 score mate 0 nodes 1 nps 466034 hashfull 0
49913 <FloydA(0): bestmove 0000
49913 >FloydB(1): isready
49913 <FloydA(0): readyok
49913 >FloydA(0): isready
49913 <FloydA(0): readyok
49913 <FloydB(1): readyok
Finished game 2 (FloydB vs FloydA): 1-0 {Black makes an illegal move: 0000}

mvk · Post by **mvk** » Fri Nov 13, 2015 11:14 pm

Update: I tried a couple of workarounds so I can continue testing, but no 100% success.

Workaround 1: don't give a pondermove unless the PV is at least 3 ply
Workaround 2: stop the search when receiving `ucinewmove'.

Even with these, this scenario still happens:

Code: Select all

3753236 <FloydA(0): info time 5 depth 120 score cp 0 nodes 20544 nps 4097179 hashfull 0 pv g6h7 e7e6 h7h8 e6f7 h8h7 f7f8 h7h8 f8f7
3753236 <FloydA(0): bestmove g6h7 ponder e7e6
3753236 >FloydA(0): position startpos moves d2d4 f7f5 g1f3 g7g6 g2g3 f8g7 f1g2 g8f6 b2b3 e8g8 c1b2 d7d5 c2c4 c7c6 e1g1 b8a6 f3e5 c8e6 b1d2 d5c4 d2c4 a6b4 d1d2 b4d5 d2g5 f6d7 a1e1 d7e5 d4e5 d8d7 h2h3 a8c8 e1d1 b7b5 c4e3 b5b4 g3g4 c8b8 a2a4 b8c8 f2f4 d7b7 g2d5 c6d5 b2d4 a7a5 d1d3 c8c6 d3d1 h7h6 g5h4 f5g4 h3g4 f8f4 f1f4 g6g5 h4h1 g5f4 e3d5 g7f8 h1e4 b7d7 e4g6 f8g7 d5f4 e6f7 g6g7 g8g7 e5e6 d7d4 d1d4 f7e6 f4e6 c6e6 g1f2 e6f6 f2e1 f6c6 d4d5 c6c3 d5a5 c3b3 a5b5 b3b2 a4a5 b2a2 e1f2 a2a3 b5b4 a3a5 b4b6 h6h5 g4h5 a5h5 f2f3 h5h6 b6b7 h6f6 f3e3 f6e6 e3f2 e6f6 f2e1 f6e6 e1f2 e6f6 f2g3 f6e6 g3f3 e6f6 f3e3 f6e6 e3d3 e6d6 d3c3 d6e6 c3d2 e6d6 d2c3 g7f6 b7b5 e7e5 e2e4 d6d4 b5b4 f6f7 b4d4 e5d4 c3d4 f7e8 e4e5 e8f8 e5e6 f8e8 d4e5 e8e7 e5f5 e7e8 f5g6 e8e7 g6h7 e7e6
3753236 >FloydA(0): isready
3753237 <FloydA(0): readyok
3753237 >FloydA(0): go ponder wtime 4720 btime 4962 winc 250 binc 250
3753237 >FloydA(0): isready
3753237 <FloydA(0): readyok
Finished game 71 (FloydA vs FloydB): 1/2-1/2 {Draw by insufficient mating material}
Score of FloydA vs FloydB: 28 - 25 - 18  [0.521] 71
Started game 72 of 2000 (FloydB vs FloydA)
3753237 >FloydA(0): ucinewgame
3753237 >FloydA(0): setoption name Ponder value true
3753237 >FloydA(0): position startpos
3753237 >FloydA(0): position startpos moves d2d4
[... snip ...]
3753238 >FloydA(0): position startpos moves d2d4 f7f5 g1f3 g7g6 g2g3 f8g7 f1g2 g8f6 b2b3 e8g8 c1b2 d7d5 c2c4 c7c6 e1g1
3753238 >FloydA(0): isready
3753238 <FloydA(0): info time 0 depth 0 score cp 0 nodes 1 nps 322639 hashfull 0
3753238 <FloydA(0): info time 0 depth 1 score cp 0 nodes 7 nps 302682 hashfull 0 pv h7h8
3753238 <FloydA(0): info time 0 depth 2 score cp 0 nodes 21 nps 524288 hashfull 0 pv h7h8 e6f7
[... snip ...]
3753238 <FloydA(0): info time 1 depth 35 score cp 0 nodes 2810 nps 2908686 hashfull 0 pv h7h8 e6f7 h8h7 f7f8 h7h8 f8f7
3753238 <FloydA(0): info time 1 depth 36 score cp 0 nodes 2916 nps 2927379 hashfull 0 pv h7h8 e6f7 h8h7 f7f8 h7h8 f8f7
3753238 <FloydA(0): info time 1 nodes 2936 nps 2895480 hashfull 0
3753238 <FloydA(0): bestmove h7h8 ponder e6f7
3753238 <FloydA(0): readyok
3753238 >FloydA(0): isready
3753238 <FloydA(0): readyok
Finished game 72 (FloydB vs FloydA): 1-0 {Black makes an illegal move: h7h8}
Score of FloydA vs FloydB: 28 - 26 - 18  [0.514] 72

I can imagine more engines must encounter the same scenario, because in this case one has to recognise conversion to KvK as a game terminating condition and then not announce that move as a pondermove. I doubt everybody does that.

Ferdy · Post by **Ferdy** » Sat Nov 14, 2015 3:08 am

Normally ucinewgame is sent when the engine is not searching. It is indeed reasonable to stop the search when you receive it while searching.

I tried testing matching crafty and deuterium dev version all were fine so far, this was at tc 60s without adjudications.

Testing this feature is painful, you don't see the board.

Tried with deuterium dev version and arasan and some games were time forfeits from arasan, this was at tc 60s + 100ms. I have not examined deeply since this is time forfeit.

ilari · Post by **ilari** » Sun Nov 15, 2015 4:20 pm

I have updated the beta to 0.8.0b3. Now cutechess doesn't ask engines to ponder on game-ending moves. I'm still going to tweak some special pondering cases a bit, but matches between two Floyds (commit 0472c2912c394defdfd4157eddd9f7ef23389f69) seems to work okay now.

Updated binaries:

* Windows 32-bit: http://koti.mbnet.fi/~ilaripih/bin/cute ... -win32.zip
* Linux 64-bit: http://koti.mbnet.fi/~ilaripih/bin/cute ... x64.tar.gz

Ferdy · Post by **Ferdy** » Tue Nov 17, 2015 2:00 am

Can't download the win32 ATM, webpage is not available.

ilari · Post by **ilari** » Tue Nov 17, 2015 7:35 am

Ferdy wrote:Can't download the win32 ATM, webpage is not available.

Maybe mbnet was doing maintenance. The file should be available now. I have removed the earlier betas though.

cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)

Re: cutechess-cli 0.8.0 beta (with pondering)