Komodo benefits less from TBs?

[Laskos]

6-men Syzygy on SSD, 16GB RAM, games at 0.25s/per move, no time losses, no adjudications. Cutechess-Cli used.

The engines are the latest incarnations of top 3 dogs:
Komodo 11.2.2
Houdini 6.02
BrainFish 2 Oct 2017 (no any book, just the fastest compile of the current master Stockfish)

The TB usage is the default in all three. Contempt=0 in all three.

1/ Starting positions are from 14-men unbalanced suite resulting from "Peasant Revolt" chess endgame variant. This endgame suite is very sensitive to TBs. The typical position looks like this:

[D]2n3n1/3k4/2n1p3/8/PP4P1/2PPPP1P/8/4K3 w - -

3000 games Round-Robin, 1000 games each engine with and without 6-men Syzygy. Result in Ordo:

Code: Select all

   # PLAYER                  : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)

   1 Houdini_602 Syzygy-6    :   51.6   12.2     567.0    1000    56.7      87    
   2 Komodo 11.2 Syzygy-6    :   40.8   12.5     553.0    1000    55.3     100    
   3 Komodo 11.2 NO TB       :   13.5   12.3     517.5    1000    51.8      97    
   4 BrainFish Syzygy-6      :   -4.6   12.6     494.0    1000    49.4     100    
   5 Houdini_602 NO TB       :  -30.7   12.1     460.0    1000    46.0     100    
   6 BrainFish NO TB         :  -70.6   12.5     408.5    1000    40.9     ---    

The correct pentanomial error margins are 1.6-2.0 smaller than those shown in Ordo, because the positions are very unbalanced (about 50% Black Wins and 50% Draws).

The ELO gain from TBs (6-men Syzygy):

• Komodo 11.2.2: 27.3
  BrainFish: 66.0
  Houdini 6.02: 82.3

I estimate the 2SD error margins when comparing the gain to about 10 ELO points, so the result is statistically very significant.

2/ To see that this is not just a property of this particular endgame variant, I also checked with a 7-8-9 men opening suite of unbalanced endgames occurring in normal games. This suite is not that sensitive to TBs as the previous one, but contains a large diversity of endgames.

3000 games Round-Robin, 1000 games each engine with and without 6-men Syzygy. Result in Ordo:

Code: Select all

   # PLAYER                  : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)

   1 Houdini_602 Syzygy-6    :   29.6   13.7     550.5    1000    55.0      91    
   2 BrainFish Syzygy-6      :   15.2   13.7     526.0    1000    52.6      85    
   3 Komodo 11.2 Syzygy-6    :    3.8   13.7     506.5    1000    50.6      93    
   4 Komodo 11.2 NO TB       :  -12.6   13.6     478.5    1000    47.9      52    
   5 BrainFish NO TB         :  -13.2   13.7     477.5    1000    47.8      81    
   6 Houdini_602 NO TB       :  -22.9   13.7     461.0    1000    46.1     ---    

The correct pentanomial error margins are again 1.6-2.0 smaller than those shown in Ordo, because the positions are again very unbalanced.

The ELO gain from TBs (6-men Syzygy):

• Komodo 11.2.2: 16.4
  BrainFish: 28.4
  Houdini 6.02: 52.5

The result is again statistically significant, and here not only Komodo underperforms with TBs, but Houdini using TBs climbs from the last place to the first.

[Sam Watson]

Excellent topic!

Was surprised that Komodo did so poor, an area that Komodo can get some Elo it seems. I thought Stockfish would be behind Houdini because Stockfish doesn't have a method of testing tablebases on the Fishtest framework. While I recall that Robert spent time making Houdini good with tablebases a few versions back.

I suspect if you tested with Houdini 5, you would get a similar Elo plus with use of tablesbases.

[shrapnel]

Komodo underperforms with TBs

I've suspected this from my observations of online matches for quite some time. You confirmed it. TBs may no doubt help in the End-Game, but I've long suspected that using TBs ruin Komodo's rather Unique Middle-Game.
Thanks.

[Uri Blass]

Komodo underperforms with TBs

I've suspected this from my observations of online matches for quite some time. You confirmed it. TBs may no doubt help in the End-Game, but I've long suspected that using TBs ruin Komodo's rather Unique Middle-Game.
Thanks.

This is not the conclusion.

Komodo get less elo from tablebases but tablebases also help komodo.

[shrapnel]

but tablebases also help komodo.

Never said they don't. Read my Post again.
It's complicated though, mainly because its difficult to pinpoint exactly where the Middle Game (where TBs don't really help Komodo and may in fact be detrimental) ends and the End-Game ( where TBs certainly help) begins.
What makes it more complicated is that one can't really go by RESULTS per se !
For example, if Komodo plays WITHOUT TBs, it may get an advantage over Stockfish in the Middle Game, but since no TBs are being used the End-Game suffers and so Result is DRAW.
On the other hand, if Komodo plays with TBs, no advantage accrues to it in the Middle Game ; reaching the End-Game with TBs, which both Engines are using, result is again DRAW !
See ?
It would require careful analysis of EVERY game from beginning to end, to note exactly where the advantage, if any, starts and then peters out, WITHOUT bothering about the End-Result !

[Modern Times]

Possibly tweaking Komodo's sysygy parameters may help.

My totally unscientific observation of Komodo is that it hits tablebases more heavily than other engines. Happy to be proven wrong on that, but if it is true then maybe that is a factor.

[Evert]

It is possible that Komodo has end-game knowledge that helps it in case there are no tablebases. If that is the case, then there is less to gain from tablebases.
This is just speculation, of course, but I wouldn't be surprised if Komodo has such knowledge.

[Laskos]

It is possible that Komodo has end-game knowledge that helps it in case there are no tablebases. If that is the case, then there is less to gain from tablebases.
This is just speculation, of course, but I wouldn't be surprised if Komodo has such knowledge.

I also thought of that possibility. Amongst these 3 top engines, at this time control, Komodo is significantly weaker than the other two in normal games with no TBs. But here, in endgames without TBs, it comes as the strongest, especially pronouncedly in the first endgame suite. In other words, without TBs it overperforms in endgames compared to general play.

So, yes, this speculation seems to me a possibility.

[Laskos]

Possibly tweaking Komodo's sysygy parameters may help.

My totally unscientific observation of Komodo is that it hits tablebases more heavily than other engines. Happy to be proven wrong on that, but if it is true then maybe that is a factor.

I may test with different Syzygy Probe Depth parameter, although in some my earlier tests I found the default as the best from fast SSD.

[Laskos]

Excellent topic!

Was surprised that Komodo did so poor, an area that Komodo can get some Elo it seems. I thought Stockfish would be behind Houdini because Stockfish doesn't have a method of testing tablebases on the Fishtest framework. While I recall that Robert spent time making Houdini good with tablebases a few versions back.

I suspect if you tested with Houdini 5, you would get a similar Elo plus with use of tablesbases.

Yes, Houdini seems to be overperforming with TBs compared to Stockfish, especially with the second, less sensitive, but more diverse endgame suite.