Komodo 8 underperforming

[titanD]

I have the latest version of Stockfish and Komodo.

I've been running some engine tests and Komodo is consistently scoring worse.

Last engine tourney I ran of 600 games at 5+0 and Stockfish scored 335 vs Komodo's 265.

The one before was Stockfish 343 vs Komodo 257

My hash is set to 128 mb.

The engine files I'm running are:
stockfish_14053109_x64_modern.exe
komodo-8-64bit.exe


I feel I must be doing something wrong. I know 1200 games isn't as statistically significant as it could be, but the performance seemed pretty consistent throughout both tournaments.

[Laskos]

I have the latest version of Stockfish and Komodo.

I've been running some engine tests and Komodo is consistently scoring worse.

Last engine tourney I ran of 600 games at 5+0 and Stockfish scored 335 vs Komodo's 265.

The one before was Stockfish 343 vs Komodo 257

My hash is set to 128 mb.

The engine files I'm running are:
stockfish_14053109_x64_modern.exe
komodo-8-64bit.exe


I feel I must be doing something wrong. I know 1200 games isn't as statistically significant as it could be, but the performance seemed pretty consistent throughout both tournaments.

It seems that Komodo's performance against Stockfish has a U shape. At ultra-fast, Komodo is ahead, at say 5 minutes/ game, SF is ahead, at LTC they are sensibly equal. We will see TCEC results soon, which will probably end in K-SF superfinal.

[titanD]

Interesting. I look forward to seeing LTC results.

I was told that Komodo may have a slightly wider search, which would lead me to suspect that it may have better performance at correspondence time controls.

[reflectionofpower]

Frazier breaks Ali's jaw in first fight and he loses,Ali beats Foreman so Frazier will beat Foreman, W-R-O-N-G. Foreman destroys Frazier in Rd. 2 at 1:35 (TKO) Tic for tac in programs is similar concept. The top 10 programs are all strong.

[lkaufman]

I have the latest version of Stockfish and Komodo.

I've been running some engine tests and Komodo is consistently scoring worse.

Last engine tourney I ran of 600 games at 5+0 and Stockfish scored 335 vs Komodo's 265.

The one before was Stockfish 343 vs Komodo 257

My hash is set to 128 mb.

The engine files I'm running are:
stockfish_14053109_x64_modern.exe
komodo-8-64bit.exe


I feel I must be doing something wrong. I know 1200 games isn't as statistically significant as it could be, but the performance seemed pretty consistent throughout both tournaments.

SF has improved quite a bit since Komodo 8 was released, and at blitz time controls SF is now stronger, though the CCRL and CEGT tests show that Komodo is much stronger relative to SF at longer time controls than at blitz. Of course Komodo has also improved, but we don't release new versions daily like SF does. Another point is that testing at sudden death, without increment, favors Stockfish, because Stockfish uses near-sudden death (very tiny increment) for their testing, while we use a normal increment to correspond with the way most testing is done. So your test conditions favor SF. I suggest 3 or 4 minutes plus two seconds increment if you want results that have more to do with the quality of play than with whether one engine runs out of time (or nearly so) after 200 moves in a drawn endgame.

[lkaufman]

Frazier breaks Ali's jaw in first fight and he loses,Ali beats Foreman so Frazier will beat Foreman, W-R-O-N-G. Foreman destroys Frazier in Rd. 2 at 1:35 (TKO) Tic for tac in programs is similar concept. The top 10 programs are all strong.

"Strong" has no meaning except relative to something else. Of course the top ten would easily defeat Carlsen or Caruana in a match, even giving huge time odds, but both Stockfish and Komodo will easily win any long match from any other engine at pretty much any reasonable time control and/or number of cores used. Strength in computer chess is not perfectly transitive, as you say, but it is fairly close to it. Komodo tends to beat other engines a bit more decisively than Stockfish (assuming fairly comparable versions of each), while SF tends to come out ahead of Komodo if they are scoring equally against other engines. But the effect is not huge.

[yanquis1972]

interesting re: decisiveness of victory, as that would seem to contradict each engine's general reputation (possibly its just my opinion, but i think its fairly common).

[Solrac1970]

Frazier breaks Ali's jaw in first fight and he loses,Ali beats Foreman so Frazier will beat Foreman, W-R-O-N-G. Foreman destroys Frazier in Rd. 2 at 1:35 (TKO) Tic for tac in programs is similar concept. The top 10 programs are all strong.

As a fan of pugilism I feel compelled to respond-it was Ken Norton who broke Ali's jaw and not Smoking Joe Frazier. Also, Ali beat Frazier twice and lost to him once. In boxing we have a saying that styles make fights and this is seemingly true also in computer chess.

[shrapnel]

Mr Kaufmann saved me the bother of replying.
I would just like to add that Komodo 8 is now even far behind in LTCs.
IPMAN confirms that Komodo 8 is behind latest Stockfish, by around 50-60 ELO !!
No point using Komodo 8 in any kind of online matches until an update is generally available.
Regards