Strange behavior/bug in Komodo

[OneTrickPony]

Here is the game:
1.d4 Nf6 2.Nf3 g6 3.c4 Bg7 4.Nc3 d5 5.cxd5 Nxd5 6.e4 Nxc3 7.bxc3 c5 8.Be3 Qa5 9.Nd2 O-O 10.Be2 cxd4 11.cxd4 f5 12.exf5 gxf5 13.Rc1 Be6

If you run it starting from position after black's 12...gf5 and wait till D20 you get 13.Rc1 +0.25 with Be6, 14.0-0 f4 as PV. Then after playing 13.Rc1 Be6 Komodo quickly realizes it's ~0.00.
Now when you go back two moves (taking back Be6 and Rc1) it's back to +0.29 even at D21 with 13. Rc1 Be6 14.0-0 f4 as PV.
Here again you move Rc1 Be6 and it's instantly back to 0.04. You go back two moves again and you see big + eval again and so forth.
This can be reproduced even if you let it run for quite long time after actually playing 13. Rc1 Be6 and then again letting it run for significant time after taking it back.
It looks like it's getting confused with hash somehow, ideas ?

(I am using x64 version, i7, 6threads, 4096MB hash and multiPV 4)

EDIT:
I let it run for some more time and it still is at 13.Rc1 +0.33 at depth 23 with 13.Rc1 Nc6 14.d5 Nd4 as PV. Now when I play 13.Rc1 Be6 it's instantly back to 0.00 and stays there until big depths (actually even moves to -0.07 at D24).
I wonder if it ever discovers 13.Rc1 doesn't give advantage on its own.

[OneTrickPony]

Ok, at D26 (which I think is big for Komodo) it's still at +0.36 for Rc1 with
PV: 13.Rc1 Be6 14.O-O f4 15.Nb3 Qb6 16.Bd2 Nc6 17.Nc5 Bf5 18.Bc3 Rad8

and again after playing Be6 it's back to 0.00 in no-time.
Hopefully it's possible to reproduce on other machines because this kind of bug is quite a big one when it comes to analysis.

[carldaman]

Ok, at D26 (which I think is big for Komodo) it's still at +0.36 for Rc1 with
PV: 13.Rc1 Be6 14.O-O f4 15.Nb3 Qb6 16.Bd2 Nc6 17.Nc5 Bf5 18.Bc3 Rad8

and again after playing Be6 it's back to 0.00 in no-time.
Hopefully it's possible to reproduce on other machines because this kind of bug is quite a big one when it comes to analysis.

What version of Komodo is exhibiting this behavior? Komodo 6 or TCEC ?
Have you tried the same position with a different version?

[OneTrickPony]

It's Komodo TCEC. I haven't tried different version as I don't have any.
I am digging deeper in to this and it seems that there is in fact an advantage in PV line after 13.Rc1.
Komodo sees it at move 13 and after playing:
13.Rc1 Be6 14.0-0 f4 15.Nb3

It doesn't see it after 13...Be6 exactly with 14.0-0 not appearing anywhere in multiPV 4.
So maybe the bug is somehow eliminating 14.0-0 as an option at root after 13...Be6 ?
As of now after 13.Rc1 Be6 Komodo has 14.Rc5 as 4th best move with -0.43 evaluation while 14.0-0 doesn't appear (so must be worse). If you play 14.0-0 Komodo doesn't need much time to see 14..f4 15.Nb3 with a + score (and improving with time). Once you take 14.0-0 back it's back to clueless.

EDIT: I downloaded Komodo 5 and it sees 14.0-0 quite quickly so it seems it's bug in TCEC version.

To summarize the bug: it sees the 14.0-0 15.Nb3 line either at move 13 (before playing 13.Rc1) or after playing 14.0-0 but not in position exactly after 13...Be6
The most amazing thing is that it's actually putting 13...Be6 on the board which causes it to miss the 14.0-0 15.Nb3 line. If you take Be6 back it's very quick to see it

[carldaman]

It's Komodo TCEC. I haven't tried different version as I don't have any.
I am digging deeper in to this and it seems that there is in fact an advantage in PV line after 13.Rc1.
Komodo sees it at move 13 and after playing:
13.Rc1 Be6 14.0-0 f4 15.Nb3

It doesn't see it after 13...Be6 exactly with 14.0-0 not appearing anywhere in multiPV 4.
So maybe the bug is somehow eliminating 14.0-0 as an option at root after 13...Be6 ?
As of now after 13.Rc1 Be6 Komodo has 14.Rc5 as 4th best move with -0.43 evaluation while 14.0-0 doesn't appear (so must be worse). If you play 14.0-0 Komodo doesn't need much time to see 14..f4 15.Nb3 with a + score (and improving with time). Once you take 14.0-0 back it's back to clueless.

EDIT: I downloaded Komodo 5 and it sees 14.0-0 quite quickly so it seems it's bug in TCEC version.

To summarize the bug: it sees the 14.0-0 15.Nb3 line either at move 13 (before playing 13.Rc1) or after playing 14.0-0 but not in position exactly after 13...Be6
The most amazing thing is that it's actually putting 13...Be6 on the board which causes it to miss the 14.0-0 15.Nb3 line. If you take Be6 back it's very quick to see it

Thanks for pointing out this bug.

[mjlef]

Thanks for the report. I am swampped with orders and customer questions but I promise to look at it when things calm down.

Mark

[OneTrickPony]

OK, problem solved sorry for panic mode.
It's almost surely a bug in Tarrash GUI (it sends castling as illegal probably) as there aren't any problems in Arena with neither position.

I was taken aback by this... sorry again.

[OneTrickPony]

On the other hand all other engines see 14.0-0 in Tarrasch GUI... so maybe it sends moves/FEN something in format Komodo doesn't understand but SF/Houdini do. Very strange.
In either case it doesn't seem to be a bug in engine's logic.

[mjlef]

On the other hand all other engines see 14.0-0 in Tarrasch GUI... so maybe it sends moves/FEN something in format Komodo doesn't understand but SF/Houdini do. Very strange.
In either case it doesn't seem to be a bug in engine's logic.

Thanks for letting us know.

[shrapnel]

I really feel this Komodo TCEC is weak...even Komodo 6 was better !
Anyway, I strongly recommend the Komodo authors to enable Syzygy support in future versions !
In my tests and online games, Stockfish and Houdini 4, both with Syzygy support, are CONSISTENTLY beating Komodo TCEC !