Komodo TCEC: Serious castling bug !

[mjlef]

I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark

[Vinvin]

I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr.
...
Mark

May be Komodo-TCEC-r1 is a better (more clear) name.

[lkaufman]

I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr.
...
Mark

May be Komodo-TCEC-r1 is a better (more clear) name.

I thought there was no need to make the name longer with the "1" because we do not anticipate another bugfix version, and if there is one we can still put a "2" after it.

[ouachita]

(http://komodochess.com/store/secure/log ... /index.php

doesn't work for me; is it working for anyone else?

[lkaufman]

I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark

Testing shows that using the LittleBlitzer GUI and the same book as used by LS list, the bugfix tested as +2.5 elo over the initial release after 4000 bullet games. Using our own huge private book it was just +1 elo after about 18,000 games. Neither result is near the margin of error, but they seem about right to me for the fix. More important, no more time forfeits!
If anyone knows of any GUI other than LittleBlitzer in which the initial release was time-forfeiting in engine matches (I'm not talking about server play, that's a whole different problem) please let us know here, so we can post a warning for those who use those GUIs to get the bugfix.
Testers who were using LittleBlitzer or any GUI that was showing time forfeits should retest; for others thaere is no need to do so.

[pohl4711]

I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark

Testing shows that using the LittleBlitzer GUI and the same book as used by LS list, the bugfix tested as +2.5 elo over the initial release after 4000 bullet games. Using our own huge private book it was just +1 elo after about 18,000 games. Neither result is near the margin of error, but they seem about right to me for the fix. More important, no more time forfeits!
If anyone knows of any GUI other than LittleBlitzer in which the initial release was time-forfeiting in engine matches (I'm not talking about server play, that's a whole different problem) please let us know here, so we can post a warning for those who use those GUIs to get the bugfix.
Testers who were using LittleBlitzer or any GUI that was showing time forfeits should retest; for others thaere is no need to do so.

Thanx for the quick bugfix. LS-testrun will follow after the testrun of Stockfish 131210.

Stefan

[lkaufman]

I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark

Testing shows that using the LittleBlitzer GUI and the same book as used by LS list, the bugfix tested as +2.5 elo over the initial release after 4000 bullet games. Using our own huge private book it was just +1 elo after about 18,000 games. Neither result is near the margin of error, but they seem about right to me for the fix. More important, no more time forfeits!
If anyone knows of any GUI other than LittleBlitzer in which the initial release was time-forfeiting in engine matches (I'm not talking about server play, that's a whole different problem) please let us know here, so we can post a warning for those who use those GUIs to get the bugfix.
Testers who were using LittleBlitzer or any GUI that was showing time forfeits should retest; for others thaere is no need to do so.

Thanx for the quick bugfix. LS-testrun will follow after the testrun of Stockfish 131210.

Stefan

I look forward to it. But it seems to me that you should limit the testing of any engine family to one version per month. Testing new stockfish versions every couple weeks introduces "survivor bias" in that one version will just get lucky and make Stockfish look artificially strong as you prune the weaker versions. It would be okay if you tested every engine with the same frequency, but you don't get new versions of most engines that often. Basically there has to be good reason to believe that a version has gained ten elo or so since the last version to overcome the margin of error for the difference between two versions to avoid survivor bias.
Anyway it's of course your call; I just want everyone to be aware of survivor bias when too many versions of one engine are tested.

Best regards,
Larry

[pohl4711]

I look forward to it. But it seems to me that you should limit the testing of any engine family to one version per month. Testing new stockfish versions every couple weeks introduces "survivor bias" in that one version will just get lucky and make Stockfish look artificially strong as you prune the weaker versions. It would be okay if you tested every engine with the same frequency, but you don't get new versions of most engines that often. Basically there has to be good reason to believe that a version has gained ten elo or so since the last version to overcome the margin of error for the difference between two versions to avoid survivor bias.
Anyway it's of course your call; I just want everyone to be aware of survivor bias when too many versions of one engine are tested.

Best regards,
Larry

First point: To test every engine with the same frequency is impossible (Houdini is updated only once in a year!).
Second point: No more than 3 development-versions of Stockfish (or other engines) are part of the LS-ratinglist and the LS-gamebase, so there is no distortion of the ratinglist - all older, outdated versions and their games are deleted out of the database. So they have no influence on the bayeselo-calculation of the LS-ratinglist.
Third point: "Basically there has to be good reason to believe that a version has gained ten elo or so". Thats what I am doing. LS-testruns of a new development-Stockfish are mostly done, when there were some patches included, which seem to bring some Elo-gain. Stockfish 131210, which is tested at the moment, contains 3 new patches with some Elo-gain in the Stockfish-Framework-selftests. So I think, a test makes sense, although that version is not much newer than Stockfish DD.
Here the 3 patches of Stockfish 131210 (since Stockfish DD):

Update History and Counter move on TT hit
LLR: 2.96 (-2.94,2.94) [0.00,6.00]
Total: 5555 W: 935 L: 808 D: 3812

Research at intermediate depth if LMR is very high
LLR: 2.95 (-2.94,2.94) [0.00,6.00]
Total: 7292 W: 1195 L: 1061 D: 5036

Broader condition for dangerous pawn moves
LLR: 2.95 (-2.94,2.94) [0.00,6.00]
Total: 21459 W: 3486 L: 3286 D: 14687


Between Stockfish 131111 and Stockfish DD there was only one Elo-gain-patch (with a very, very, small gain), but Stockfish DD is an official release, so I had to test it (otherwise I wouldnt have tested it). And the result was nearly the same rating like Stockfish 131111 (-1 Elo)(as I expected).
And Stockfish DD adapC is an experimental Stockfish-mod, which I wanted to test...It will be handled like a Stockfish-development-version and will be deleted, when 3 newer development-versions of Stockfish are part of the LS-ratinglist.

Best regards - Stefan

[OneTrickPony]

Thanks, this version works correctly for me in Tarrasch GUI.

[lkaufman]

Thanks, this version works correctly for me in Tarrasch GUI.

Just to be clear, does this mean that you were getting the time forfeits/hangups with the older version? If so we should add Tarrasch GUI to the list of ones that need the fixed version.