Komodo benefits less from TBs?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Komodo benefits less from TBs?

Post by Laskos » Sat Oct 07, 2017 12:05 pm

6-men Syzygy on SSD, 16GB RAM, games at 0.25s/per move, no time losses, no adjudications. Cutechess-Cli used.

The engines are the latest incarnations of top 3 dogs:
Komodo 11.2.2
Houdini 6.02
BrainFish 2 Oct 2017 (no any book, just the fastest compile of the current master Stockfish)

The TB usage is the default in all three. Contempt=0 in all three.

1/ Starting positions are from 14-men unbalanced suite resulting from "Peasant Revolt" chess endgame variant. This endgame suite is very sensitive to TBs. The typical position looks like this:

[D]2n3n1/3k4/2n1p3/8/PP4P1/2PPPP1P/8/4K3 w - -

3000 games Round-Robin, 1000 games each engine with and without 6-men Syzygy. Result in Ordo:

Code: Select all

   # PLAYER                  : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)

   1 Houdini_602 Syzygy-6    :   51.6   12.2     567.0    1000    56.7      87    
   2 Komodo 11.2 Syzygy-6    :   40.8   12.5     553.0    1000    55.3     100    
   3 Komodo 11.2 NO TB       :   13.5   12.3     517.5    1000    51.8      97    
   4 BrainFish Syzygy-6      :   -4.6   12.6     494.0    1000    49.4     100    
   5 Houdini_602 NO TB       :  -30.7   12.1     460.0    1000    46.0     100    
   6 BrainFish NO TB         :  -70.6   12.5     408.5    1000    40.9     ---    
The correct pentanomial error margins are 1.6-2.0 smaller than those shown in Ordo, because the positions are very unbalanced (about 50% Black Wins and 50% Draws).

The ELO gain from TBs (6-men Syzygy):
  • Komodo 11.2.2: 27.3
    BrainFish: 66.0
    Houdini 6.02: 82.3
I estimate the 2SD error margins when comparing the gain to about 10 ELO points, so the result is statistically very significant.

2/ To see that this is not just a property of this particular endgame variant, I also checked with a 7-8-9 men opening suite of unbalanced endgames occurring in normal games. This suite is not that sensitive to TBs as the previous one, but contains a large diversity of endgames.

3000 games Round-Robin, 1000 games each engine with and without 6-men Syzygy. Result in Ordo:

Code: Select all

   # PLAYER                  : RATING  ERROR    POINTS  PLAYED     (%)   CFS(next)

   1 Houdini_602 Syzygy-6    :   29.6   13.7     550.5    1000    55.0      91    
   2 BrainFish Syzygy-6      :   15.2   13.7     526.0    1000    52.6      85    
   3 Komodo 11.2 Syzygy-6    :    3.8   13.7     506.5    1000    50.6      93    
   4 Komodo 11.2 NO TB       :  -12.6   13.6     478.5    1000    47.9      52    
   5 BrainFish NO TB         :  -13.2   13.7     477.5    1000    47.8      81    
   6 Houdini_602 NO TB       :  -22.9   13.7     461.0    1000    46.1     ---    
The correct pentanomial error margins are again 1.6-2.0 smaller than those shown in Ordo, because the positions are again very unbalanced.

The ELO gain from TBs (6-men Syzygy):
  • Komodo 11.2.2: 16.4
    BrainFish: 28.4
    Houdini 6.02: 52.5
The result is again statistically significant, and here not only Komodo underperforms with TBs, but Houdini using TBs climbs from the last place to the first.

Sam Watson
Posts: 3
Joined: Thu Aug 17, 2017 12:31 pm

Re: Komodo benefits less from TBs?

Post by Sam Watson » Sat Oct 07, 2017 2:04 pm

Excellent topic!

Was surprised that Komodo did so poor, an area that Komodo can get some Elo it seems. I thought Stockfish would be behind Houdini because Stockfish doesn't have a method of testing tablebases on the Fishtest framework. While I recall that Robert spent time making Houdini good with tablebases a few versions back.

I suspect if you tested with Houdini 5, you would get a similar Elo plus with use of tablesbases.

shrapnel
Posts: 1267
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: Komodo benefits less from TBs?

Post by shrapnel » Sat Oct 07, 2017 4:36 pm

Komodo underperforms with TBs
I've suspected this from my observations of online matches for quite some time. You confirmed it. TBs may no doubt help in the End-Game, but I've long suspected that using TBs ruin Komodo's rather Unique Middle-Game.
Thanks.
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

Uri Blass
Posts: 8697
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Komodo benefits less from TBs?

Post by Uri Blass » Sat Oct 07, 2017 5:06 pm

shrapnel wrote:
Komodo underperforms with TBs
I've suspected this from my observations of online matches for quite some time. You confirmed it. TBs may no doubt help in the End-Game, but I've long suspected that using TBs ruin Komodo's rather Unique Middle-Game.
Thanks.
This is not the conclusion.

Komodo get less elo from tablebases but tablebases also help komodo.

shrapnel
Posts: 1267
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: Komodo benefits less from TBs?

Post by shrapnel » Sat Oct 07, 2017 7:11 pm

but tablebases also help komodo.
Never said they don't. Read my Post again.
It's complicated though, mainly because its difficult to pinpoint exactly where the Middle Game (where TBs don't really help Komodo and may in fact be detrimental) ends and the End-Game ( where TBs certainly help) begins.
What makes it more complicated is that one can't really go by RESULTS per se !
For example, if Komodo plays WITHOUT TBs, it may get an advantage over Stockfish in the Middle Game, but since no TBs are being used the End-Game suffers and so Result is DRAW.
On the other hand, if Komodo plays with TBs, no advantage accrues to it in the Middle Game ; reaching the End-Game with TBs, which both Engines are using, result is again DRAW !
See ?
It would require careful analysis of EVERY game from beginning to end, to note exactly where the advantage, if any, starts and then peters out, WITHOUT bothering about the End-Result !
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

Modern Times
Posts: 2459
Joined: Thu Jun 07, 2012 9:02 pm

Re: Komodo benefits less from TBs?

Post by Modern Times » Sat Oct 07, 2017 7:37 pm

Possibly tweaking Komodo's sysygy parameters may help.

My totally unscientific observation of Komodo is that it hits tablebases more heavily than other engines. Happy to be proven wrong on that, but if it is true then maybe that is a factor.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

User avatar
Evert
Posts: 2925
Joined: Fri Jan 21, 2011 11:42 pm
Location: NL
Contact:

Re: Komodo benefits less from TBs?

Post by Evert » Sat Oct 07, 2017 7:58 pm

It is possible that Komodo has end-game knowledge that helps it in case there are no tablebases. If that is the case, then there is less to gain from tablebases.
This is just speculation, of course, but I wouldn't be surprised if Komodo has such knowledge.

User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Komodo benefits less from TBs?

Post by Laskos » Sat Oct 07, 2017 8:15 pm

Evert wrote:It is possible that Komodo has end-game knowledge that helps it in case there are no tablebases. If that is the case, then there is less to gain from tablebases.
This is just speculation, of course, but I wouldn't be surprised if Komodo has such knowledge.
I also thought of that possibility. Amongst these 3 top engines, at this time control, Komodo is significantly weaker than the other two in normal games with no TBs. But here, in endgames without TBs, it comes as the strongest, especially pronouncedly in the first endgame suite. In other words, without TBs it overperforms in endgames compared to general play.

So, yes, this speculation seems to me a possibility.

User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Komodo benefits less from TBs?

Post by Laskos » Sun Oct 08, 2017 7:49 am

Modern Times wrote:Possibly tweaking Komodo's sysygy parameters may help.

My totally unscientific observation of Komodo is that it hits tablebases more heavily than other engines. Happy to be proven wrong on that, but if it is true then maybe that is a factor.
I may test with different Syzygy Probe Depth parameter, although in some my earlier tests I found the default as the best from fast SSD.

User avatar
Laskos
Posts: 9843
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Komodo benefits less from TBs?

Post by Laskos » Sun Oct 08, 2017 8:02 am

Sam Watson wrote:Excellent topic!

Was surprised that Komodo did so poor, an area that Komodo can get some Elo it seems. I thought Stockfish would be behind Houdini because Stockfish doesn't have a method of testing tablebases on the Fishtest framework. While I recall that Robert spent time making Houdini good with tablebases a few versions back.

I suspect if you tested with Houdini 5, you would get a similar Elo plus with use of tablesbases.
Yes, Houdini seems to be overperforming with TBs compared to Stockfish, especially with the second, less sensitive, but more diverse endgame suite.

Post Reply