Romichess bug (Michael?)

[michiguel]

After playing a bit more than 5000 games between Romichess and Gaviota, I got this 6 games in which Romichess tried to play an illegal move right after the book moves fed by cutechess-cli in Linux, 32 bits.
The games are 40 moves in 8 seconds. Anybody seen this?

I have the feeling the bug is related to some sort of a timing issue at this fast speed.

The command to call the engine is wrapped in romichess.sh, which has

Code: Select all

#!/bin/sh
wine RomiChessP3k.exe

BTW, I love Romichess, it is a fantastic partner for Gaviota.

Miguel

Code: Select all

[Event "?"]
[Site "?"]
[Date "2010.07.25"]
[Round "146"]
[White "Romichess-P3K"]
[Black "Gaviota-p08"]
[Result "0-1"]
[PlyCount "20"]
[TimeControl "40/8"]

1. e4 {book} c5 {book} 2. Nf3 {book} e6 {book} 3. d4 {book} cxd4 {book}
4. Nxd4 {book} a6 {book} 5. Bd3 {book} Bc5 {book} 6. Nb3 {book} Ba7 {book}
7. Nc3 {book} Nc6 {book} 8. Qe2 {book} d6 {book} 9. Be3 {book} Nf6 {book}
10. Bxa7 {book} Rxa7 {book, White makes an illegal move: a8g8} 0-1

[Event "?"]
[Site "?"]
[Date "2010.07.25"]
[Round "600"]
[White "Romichess-P3K"]
[Black "Gaviota-p08"]
[Result "0-1"]
[PlyCount "20"]
[TimeControl "40/8"]

1. e4 {book} c5 {book} 2. Nf3 {book} d6 {book} 3. d4 {book} Nf6 {book}
4. Nc3 {book} cxd4 {book} 5. Nxd4 {book} a6 {book} 6. Be3 {book} Ng4 {book}
7. Bc1 {book} Nf6 {book} 8. Be3 {book} Ng4 {book} 9. Bg5 {book} h6 {book}
10. Bh4 {book} g5 {book, White makes an illegal move: d7f6} 0-1

[Event "?"]
[Site "?"]
[Date "2010.08.27"]
[Round "716"]
[White "Romichess-P3K"]
[Black "Gaviota-p2bsc"]
[Result "0-1"]
[PlyCount "20"]
[TimeControl "40/8"]

1. d4 {book} Nf6 {book} 2. c4 {book} g6 {book} 3. Nc3 {book} d5 {book}
4. Bf4 {book} Bg7 {book} 5. e3 {book} O-O {book} 6. Qb3 {book} c5 {book}
7. cxd5 {book} cxd4 {book} 8. exd4 {book} Nbd7 {book} 9. Be2 {book} Nb6 {book}
10. Bf3 {book} e6 {book, White makes an illegal move: d1d5} 0-1


[Event "?"]
[Site "?"]
[Date "2010.08.27"]
[Round "902"]
[White "Romichess-P3K"]
[Black "Gaviota-p2bsc"]
[Result "0-1"]
[PlyCount "16"]
[TimeControl "40/8"]

1. d4 {book} Nc6 {book} 2. c4 {book} e5 {book} 3. d5 {book} Nce7 {book}
4. e4 {book} Nf6 {book} 5. Nc3 {book} Ng6 {book} 6. Be3 {book} Bb4 {book}
7. f3 {book} Bxc3+ {book} 8. bxc3 {book}
d6 {book, White makes an illegal move: g6h5} 0-1


[Event "?"]
[Site "?"]
[Date "2010.08.28"]
[Round "50"]
[White "Romichess-P3K"]
[Black "Gaviota-p2b"]
[Result "0-1"]
[PlyCount "20"]
[TimeControl "40/8"]

1. d4 {book} d5 {book} 2. c4 {book} e6 {book} 3. Nf3 {book} c6 {book}
4. cxd5 {book} exd5 {book} 5. Nc3 {book} Nf6 {book} 6. Bg5 {book} Bf5 {book}
7. e3 {book} Nbd7 {book} 8. Bd3 {book} Bxd3 {book} 9. Qxd3 {book} Be7 {book}
10. Rb1 {book} O-O {book, White makes an illegal move: e6d6} 0-1


[Event "?"]
[Site "?"]
[Date "2010.08.28"]
[Round "40"]
[White "Romichess-P3K"]
[Black "Gaviota-p2b"]
[Result "0-1"]
[PlyCount "20"]
[TimeControl "40/8"]

1. d4 {book} Nf6 {book} 2. Nf3 {book} d5 {book} 3. c4 {book} e6 {book}
4. g3 {book} Be7 {book} 5. Bg2 {book} O-O {book} 6. O-O {book} dxc4 {book}
7. Qc2 {book} a6 {book} 8. Qxc4 {book} b5 {book} 9. Qc2 {book} Bb7 {book}
10. Bd2 {book} Be4 {book, White makes an illegal move: e3d3} 0-1

[gaard]

I've seen this with cutechess-cli with a few engines over hundreds of thousands of games. Engines that otherwise are virtually fault free. If it is reproducible I'd check cutechess-cli's debug log first because I have a hunch it is unrelated to RomiChess.

[Michael Sherwin]

Hi Miguel,

I am a big Gaviota fan! I now test Romi more against Gavi than I do any other engine. Most games are just out of this world fantastic to watch. Like watching a fight from a "Rocky" movie! What Gavi lacks in depth it more than makes up with evaluation!! Romi needs all the extra depth it gets just to stay in most games.

As far as the problem that you are having with Cutechess-cli I haven't a clue. If there was something wrong with Romi then it should happen more often than 6 times in 5000. Especially since It does not seem to happen with other GUI. However, I will have a look at the games to see if anything stands out about them. Do you use my compile or one by someone else?

Edit: Romi does sometimes stay in memory. If that happens then maybe that Romi does not have the correct position from book. If Cute has a kill command then maybe try that. Just a guess.

[michiguel]

Hi Miguel,

I am a big Gaviota fan! I now test Romi more against Gavi than I do any other engine. Most games are just out of this world fantastic to watch. Like watching a fight from a "Rocky" movie! What Gavi lacks in depth it more than makes up with evaluation!! Romi needs all the extra depth it gets just to stay in most games.

As far as the problem that you are having with Cutechess-cli I haven't a clue. If there was something wrong with Romi then it should happen more often than 6 times in 5000. Especially since It does not seem to happen with other GUI. However, I will have a look at the games to see if anything stands out about them. Do you use my compile or one by someone else?

Edit: Romi does sometimes stay in memory. If that happens then maybe that Romi does not have the correct position from book. If Cute has a kill command then maybe try that. Just a guess.

It kind of surprised me because other than this, Romi has always been rock solid.

Comments from Martin and you oriented me to look at the right place: Romi try to play a move from the previous game (it makes sense if you look at the move sent at the final position of the previous game). In the previous game, it resigned, but sent a move anyway. Cutechess-cli must have processed that move after the next game started.

I think this have been a common synchronization problem with many engines/GUIs.

I just grep your sources with "resign" and I found that Romi sends a move and then, if the score < resignlimit, sends "resign" to the GUI. Maybe it should not send any move if the decision is to resign. This may be a problem with cutechess-cli, but there is no advantage to send a move and then resign. From the point of view of the protocol, I think it is safer to resign when it is not the opponent's turn, and wait for "result".

I hope this helps!

Miguel

[ilari]

There's the problem. Cutechess-cli expects the game to be over when an engine resigns, so no further moves should be sent. With Winboard engines (especially ones that don't support the ping feature) it's sometimes difficult to know when it's safe to start a new game.

Cutechess-cli has the "-wait <n>" option to solve this problem. Try a value between 10 to 100 milliseconds first, it should be enough for Romi to send any pending moves before the next game.

[Matthias Gemuh]

I just grep your sources with "resign" and I found that Romi sends a move and then, if the score < resignlimit, sends "resign" to the GUI. Maybe it should not send any move if the decision is to resign. This may be a problem with cutechess-cli, but there is no advantage to send a move and then resign. From the point of view of the protocol, I think it is safer to resign when it is not the opponent's turn, and wait for "result".

I hope this helps!

Miguel

Good point, Miguel.

[michiguel]

There's the problem. Cutechess-cli expects the game to be over when an engine resigns, so no further moves should be sent. With Winboard engines (especially ones that don't support the ping feature) it's sometimes difficult to know when it's safe to start a new game.

Cutechess-cli has the "-wait <n>" option to solve this problem. Try a value between 10 to 100 milliseconds first, it should be enough for Romi to send any pending moves before the next game.

Good suggestions, I ran with -wait 100 and not illegal move appear but something else showed up.

Romichess lost by "White's connection stalls". After that, it stayed in memory spinning. This the weird thing:

Romichess and Gaviota repeated a position for the third time after a gaviota's move. At that point, the game should have been over because in those cases, Gaviota moves and sends draw to claim it.

I think there must be some sort of an issue of how draw offers and claims are done/received among these two engines and cutechess-cli.

Miguel
PS: Does Romi recognize castling when different positions are compared for repetition? Just a wild guess.

[Michael Sherwin]

This is reminiscent of my health problems--as soon as I get control of one ailment, another magically appears to take its place. I will look into these new problems with Romi as my strength and time permit.

[michiguel]

This is reminiscent of my health problems--as soon as I get control of one ailment, another magically appears to take its place. I will look into these new problems with Romi as my strength and time permit.

In this position, Gaviota (black) played 45. ... h5
[D]8/bpr1kp1p/4pNp1/P5P1/5P1P/8/3RK3/8 b - - 2 45

and reached this position

[D]8/bpr1kp2/4pNp1/P5Pp/5P1P/8/3RK3/8 w - h6 0 46

and the game proceeded with

46. Ng8+ {-0.56/11 0.20s}
Ke8 {0.00/8 0.17s} 47. Nf6+ {+0.53/11 0.21s} Ke7 {0.00/9 0.17s}

[D]8/bpr1kp2/4pNp1/P5Pp/5P1P/8/3RK3/8 w - - 4 48

48. Ng8+ {-0.53/13 0.26s} Ke8 {0.00/9 0.16s} 49. Nf6+ {-0.52/13 0.20s}
Ke7 {0.00/9 0.16s, White's connection stalls} 0-1

[D]8/bpr1kp2/4pNp1/P5Pp/5P1P/8/3RK3/8 w - - 8 50

This seems to be a third repetition, but it is not (I was wrong in the previous post), because the first position white has the ability to capture en passant at h6, but not in the other two positions.

My theory is that Romichess thought it was a third repetition, sent a draw and left the game thinking it was a draw.

Miguel

[ilari]

My theory is that Romichess thought it was a third repetition, sent a draw and left the game thinking it was a draw.
Miguel

You're probably right. When cutechess-cli receives a draw claim in a position that is not a draw, it expects that the next move from the engine draws the game (eg. Crafty does this). If the engine doesn't send any move, you get the "connection stalls" error. I'll try to fix this so that it only results in a forfeit.