Komodo TCEC: Serious castling bug !

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
mjlef
Posts: 1427
Joined: Thu Mar 30, 2006 12:08 pm
Contact:

Re: Komodo TCEC: Serious castling bug !

Post by mjlef » Wed Dec 11, 2013 9:35 pm

I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark

Vinvin
Posts: 4390
Joined: Thu Mar 09, 2006 8:40 am
Full name: Vincent Lejeune

Re: Komodo TCEC: Serious castling bug !

Post by Vinvin » Wed Dec 11, 2013 10:25 pm

mjlef wrote:I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr.
...
Mark
May be Komodo-TCEC-r1 is a better (more clear) name.

lkaufman
Posts: 3733
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo TCEC: Serious castling bug !

Post by lkaufman » Wed Dec 11, 2013 11:46 pm

Vinvin wrote:
mjlef wrote:I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr.
...
Mark
May be Komodo-TCEC-r1 is a better (more clear) name.
I thought there was no need to make the name longer with the "1" because we do not anticipate another bugfix version, and if there is one we can still put a "2" after it.

ouachita
Posts: 454
Joined: Tue Jan 15, 2013 3:33 pm
Location: Ritz-Carlton, NYC
Full name: Bobby Johnson

Re: Komodo TCEC: Serious castling bug !

Post by ouachita » Wed Dec 11, 2013 11:54 pm

doesn't work for me; is it working for anyone else?
SIM, PhD, MBA, PE

lkaufman
Posts: 3733
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo TCEC: Serious castling bug !

Post by lkaufman » Wed Dec 11, 2013 11:54 pm

mjlef wrote:I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark
Testing shows that using the LittleBlitzer GUI and the same book as used by LS list, the bugfix tested as +2.5 elo over the initial release after 4000 bullet games. Using our own huge private book it was just +1 elo after about 18,000 games. Neither result is near the margin of error, but they seem about right to me for the fix. More important, no more time forfeits!
If anyone knows of any GUI other than LittleBlitzer in which the initial release was time-forfeiting in engine matches (I'm not talking about server play, that's a whole different problem) please let us know here, so we can post a warning for those who use those GUIs to get the bugfix.
Testers who were using LittleBlitzer or any GUI that was showing time forfeits should retest; for others thaere is no need to do so.

pohl4711
Posts: 1099
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Komodo TCEC: Serious castling bug !

Post by pohl4711 » Thu Dec 12, 2013 4:57 am

lkaufman wrote:
mjlef wrote:I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark
Testing shows that using the LittleBlitzer GUI and the same book as used by LS list, the bugfix tested as +2.5 elo over the initial release after 4000 bullet games. Using our own huge private book it was just +1 elo after about 18,000 games. Neither result is near the margin of error, but they seem about right to me for the fix. More important, no more time forfeits!
If anyone knows of any GUI other than LittleBlitzer in which the initial release was time-forfeiting in engine matches (I'm not talking about server play, that's a whole different problem) please let us know here, so we can post a warning for those who use those GUIs to get the bugfix.
Testers who were using LittleBlitzer or any GUI that was showing time forfeits should retest; for others thaere is no need to do so.

Thanx for the quick bugfix. LS-testrun will follow after the testrun of Stockfish 131210.

Stefan

lkaufman
Posts: 3733
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo TCEC: Serious castling bug !

Post by lkaufman » Thu Dec 12, 2013 5:14 am

pohl4711 wrote:
lkaufman wrote:
mjlef wrote:I am pleased to announce a revised version of Komodo-TCEC, which we call Komodo-TCECr. It fixes the reported error in fen diagrams followed by castling moves. It also fixes a minor issue of an error message when uses run Komodo in a Windows cmd window and close it using the title bar X instead of using the “quit” command.

Komodo TCEC owners can log into account on the Komodochess.com site (http://komodochess.com/store/secure/log ... /index.php) to download the revised version.

Note that users running matches using “Little Blitzer” will want to get the download, since Little Blitzer apparently converts PGNs to fens during match play. Since the error disallowed castling after many fen positions, Komodo might not find the castling move after a fen, which could hurt play on testers using fens. This is fixed in the TCECr version.

Thanks again for reporting the bug. We want Komodo to be rock solid and really appreciate it when you let us know about problems so we can fix them quickly.

I wish to thank Jeremy Bernstein for making the Windows X close patch.

Mark
Testing shows that using the LittleBlitzer GUI and the same book as used by LS list, the bugfix tested as +2.5 elo over the initial release after 4000 bullet games. Using our own huge private book it was just +1 elo after about 18,000 games. Neither result is near the margin of error, but they seem about right to me for the fix. More important, no more time forfeits!
If anyone knows of any GUI other than LittleBlitzer in which the initial release was time-forfeiting in engine matches (I'm not talking about server play, that's a whole different problem) please let us know here, so we can post a warning for those who use those GUIs to get the bugfix.
Testers who were using LittleBlitzer or any GUI that was showing time forfeits should retest; for others thaere is no need to do so.

Thanx for the quick bugfix. LS-testrun will follow after the testrun of Stockfish 131210.

Stefan
I look forward to it. But it seems to me that you should limit the testing of any engine family to one version per month. Testing new stockfish versions every couple weeks introduces "survivor bias" in that one version will just get lucky and make Stockfish look artificially strong as you prune the weaker versions. It would be okay if you tested every engine with the same frequency, but you don't get new versions of most engines that often. Basically there has to be good reason to believe that a version has gained ten elo or so since the last version to overcome the margin of error for the difference between two versions to avoid survivor bias.
Anyway it's of course your call; I just want everyone to be aware of survivor bias when too many versions of one engine are tested.

Best regards,
Larry

pohl4711
Posts: 1099
Joined: Sat Sep 03, 2011 5:25 am
Location: Berlin, Germany
Contact:

Re: Komodo TCEC: Serious castling bug !

Post by pohl4711 » Thu Dec 12, 2013 7:26 am

lkaufman wrote:
I look forward to it. But it seems to me that you should limit the testing of any engine family to one version per month. Testing new stockfish versions every couple weeks introduces "survivor bias" in that one version will just get lucky and make Stockfish look artificially strong as you prune the weaker versions. It would be okay if you tested every engine with the same frequency, but you don't get new versions of most engines that often. Basically there has to be good reason to believe that a version has gained ten elo or so since the last version to overcome the margin of error for the difference between two versions to avoid survivor bias.
Anyway it's of course your call; I just want everyone to be aware of survivor bias when too many versions of one engine are tested.

Best regards,
Larry
First point: To test every engine with the same frequency is impossible (Houdini is updated only once in a year!).
Second point: No more than 3 development-versions of Stockfish (or other engines) are part of the LS-ratinglist and the LS-gamebase, so there is no distortion of the ratinglist - all older, outdated versions and their games are deleted out of the database. So they have no influence on the bayeselo-calculation of the LS-ratinglist.
Third point: "Basically there has to be good reason to believe that a version has gained ten elo or so". Thats what I am doing. LS-testruns of a new development-Stockfish are mostly done, when there were some patches included, which seem to bring some Elo-gain. Stockfish 131210, which is tested at the moment, contains 3 new patches with some Elo-gain in the Stockfish-Framework-selftests. So I think, a test makes sense, although that version is not much newer than Stockfish DD.
Here the 3 patches of Stockfish 131210 (since Stockfish DD):

Update History and Counter move on TT hit
LLR: 2.96 (-2.94,2.94) [0.00,6.00]
Total: 5555 W: 935 L: 808 D: 3812

Research at intermediate depth if LMR is very high
LLR: 2.95 (-2.94,2.94) [0.00,6.00]
Total: 7292 W: 1195 L: 1061 D: 5036

Broader condition for dangerous pawn moves
LLR: 2.95 (-2.94,2.94) [0.00,6.00]
Total: 21459 W: 3486 L: 3286 D: 14687


Between Stockfish 131111 and Stockfish DD there was only one Elo-gain-patch (with a very, very, small gain), but Stockfish DD is an official release, so I had to test it (otherwise I wouldnt have tested it). And the result was nearly the same rating like Stockfish 131111 (-1 Elo)(as I expected).
And Stockfish DD adapC is an experimental Stockfish-mod, which I wanted to test...It will be handled like a Stockfish-development-version and will be deleted, when 3 newer development-versions of Stockfish are part of the LS-ratinglist.

Best regards - Stefan

OneTrickPony
Posts: 95
Joined: Mon Apr 29, 2013 11:29 pm

Re: Komodo TCEC: Serious castling bug !

Post by OneTrickPony » Thu Dec 12, 2013 8:32 am

Thanks, this version works correctly for me in Tarrasch GUI.

lkaufman
Posts: 3733
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Komodo TCEC: Serious castling bug !

Post by lkaufman » Thu Dec 12, 2013 2:58 pm

OneTrickPony wrote:Thanks, this version works correctly for me in Tarrasch GUI.
Just to be clear, does this mean that you were getting the time forfeits/hangups with the older version? If so we should add Tarrasch GUI to the list of ones that need the fixed version.

Post Reply