Komodo 8 - 5-men Syzygy tablebases

fastgm · Post by **fastgm** » Sat Jan 10, 2015 7:26 pm

What are the benefits of using 5-men Syzygy tablebases?

Here a test with Komodo 8 on one core with 128 MB Hash.
The tablebases were on a solid-state drive - Samsung 840 EVO.
Four different time controls were tested, each 3000 games or 1520 at highest time control.
15 sec + 0.15 sec / 60 sec + 0.60 sec / 240 sec + 2.40 sec / 960 sec + 9.60 sec

Code: Select all

TC  = 15"+0.15"
TB5 = 5-men Syzygy tablebases
T1  = 1 thread
 
    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3005   8   8   3000    51.4 %   2995   60.8 %
  2 Komodo 8 T1      &#58; 2995   8   8   3000    48.6 %   3005   60.8 %

Result &#58; 1542.5/3000 (+630,=1825,-545&#41;
 
Elo  +10 
 
TC  = 60"+0.6"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3009   7   7   3000    52.5 %   2991   68.2 %
  2 Komodo 8 T1      &#58; 2991   7   7   3000    47.5 %   3009   68.2 %

Result &#58; 1576.0/3000 (+553,=2046,-401&#41;
 
Elo  +18 
 
TC  = 4'+2.4"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3006   6   6   3000    51.7 %   2994   75.4 %
  2 Komodo 8 T1      &#58; 2994   6   6   3000    48.3 %   3006   75.4 %

Result &#58; 1551.5/3000 (+420,=2263,-317&#41;
 
Elo  +12
  
TC  = 16'+9.6"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3006   8   8   1520    51.7 %   2994   79.8 %
  2 Komodo 8 T1      &#58; 2994   8   8   1520    48.3 %   3006   79.8 %

Result &#58; 785.5/1520 (+179,=1213,-128&#41;
 
Elo  +12

fenchel · Post by **fenchel** » Sat Jan 10, 2015 9:10 pm

fastgm wrote:What are the benefits of using 5-men Syzygy tablebases?

Here a test with Komodo 8 on one core with 128 MB Hash.
The tablebases were on a solid-state drive - Samsung 840 EVO.
Four different time controls were tested, each 3000 games or 1520 at highest time control.
15 sec + 0.15 sec / 60 sec + 0.60 sec / 240 sec + 2.40 sec / 960 sec + 9.60 sec

Code: Select all

TC  = 15"+0.15"
TB5 = 5-men Syzygy tablebases
T1  = 1 thread
 
    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3005   8   8   3000    51.4 %   2995   60.8 %
  2 Komodo 8 T1      &#58; 2995   8   8   3000    48.6 %   3005   60.8 %

Result &#58; 1542.5/3000 (+630,=1825,-545&#41;
 
Elo  +10 
 
TC  = 60"+0.6"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3009   7   7   3000    52.5 %   2991   68.2 %
  2 Komodo 8 T1      &#58; 2991   7   7   3000    47.5 %   3009   68.2 %

Result &#58; 1576.0/3000 (+553,=2046,-401&#41;
 
Elo  +18 
 
TC  = 4'+2.4"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3006   6   6   3000    51.7 %   2994   75.4 %
  2 Komodo 8 T1      &#58; 2994   6   6   3000    48.3 %   3006   75.4 %

Result &#58; 1551.5/3000 (+420,=2263,-317&#41;
 
Elo  +12
  
TC  = 16'+9.6"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3006   8   8   1520    51.7 %   2994   79.8 %
  2 Komodo 8 T1      &#58; 2994   8   8   1520    48.3 %   3006   79.8 %

Result &#58; 785.5/1520 (+179,=1213,-128&#41;
 
Elo  +12

Thank you so much for performing this test. I think it is especially valuable at this present time (e.g., with all the endgame knowledge debates going on with SF, the recent TCEC, etc).

Also, thanks very much for trying different TCs, though of course for now they have overlapping error bars.

I think it would be great to see more of these kinds of tests =) (6-man, SF, small error bars, threads, etc)

Leto · Post by **Leto** » Sat Jan 10, 2015 11:36 pm

Thanks for performing this test. A similar test with 6 men tablebases would be even more interesting, I'm certain they will add more than 20 elo at the 960 seconds + 9.60 seconds time control, maybe 50 elo.

Game in 15 minutes + 9.60 sec is rapid, would a slower time control such as game in 2 hours show more elo gains from 6 men tablebases. I'd say yes, but I don't know how much.

cc2150dx · Post by **cc2150dx** » Sun Jan 11, 2015 10:52 am

I mainly use 5 men syzygy bases to determine the wins/loss/draws quicker.

And, off-topic I use the bases on a regular HDD with Houdini, SF, K with no slowdowns.

Jason

fastgm · Post by **fastgm** » Wed Jan 21, 2015 8:20 pm

Here another test, this time Komodo 8 against Stockfish 5, Komodo once without and then with tablebases, 1 thread, 128 MB Hash.
Three different time controls (TC) were tested, each 3000 or 1550 games at highest time control.
60 sec + 0.60 sec / 240 sec + 2.40 sec / 960 sec + 9.60 sec

Code: Select all

TC  = 60"+0.60"
TB5 = 5-men Syzygy tablebases
T1  = 1 Thread


     Program            Elo    +   -   Games   Score   Av.Op.  Draws
  --------------------------------------------------------------------
   1 Stockfish 5 T1   &#58; 3005   9   9   3000    51.4 %   2995   46.1 %
   2 Komodo 8 T1      &#58; 2995   9   9   3000    48.6 %   3005   46.1 %
 
Result &#58; 1540.5/3000 (+849,=1383,-768&#41;

Elo  +9


     Program            Elo    +   -   Games   Score   Av.Op.  Draws
  --------------------------------------------------------------------
   1 Stockfish 5 T1   &#58; 3002   9   9   3000    50.5 %   2998   48.2 %
   2 Komodo 8 T1 TB5  &#58; 2998   9   9   3000    49.5 %   3002   48.2 %

Result &#58; 1516.5/3000 (+793,=1447,-760&#41;

Elo  +4                   TB benefit&#58; 5 Elo

--------------------------------------------------------------------

 
TC  = 4'+2.4"

     Program            Elo    +   -   Games   Score   Av.Op.  Draws
  --------------------------------------------------------------------
   1 Stockfish 5 T1   &#58; 3004   9   9   3000    51.2 %   2996   52.2 %
   2 Komodo 8 T1      &#58; 2996   9   9   3000    48.8 %   3004   52.2 %

Result &#58; 1536.5/3000 (+754,=1565,-681&#41;
 
Elo  +8


     Program            Elo    +   -   Games   Score   Av.Op.  Draws
  --------------------------------------------------------------------
   1 Stockfish 5 T1   &#58; 3002   9   9   3000    50.5 %   2998   52.1 %
   2 Komodo 8 T1 TB5  &#58; 2998   9   9   3000    49.5 %   3002   52.1 %

Result &#58; 1516.5/3000 (+735,=1563,-702&#41;

Elo  +4                   TB benefit&#58; 4 Elo

-------------------------------------------------------------------- 

  
TC  = 16'+9.6"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 --------------------------------------------------------------------
  1 Komodo 8 T1      &#58; 3004   11  11  1550    51.3 %   2996   60.4 %
  2 Stockfish 5 T1   &#58; 2996   11  11  1550    48.7 %   3004   60.4 %

Result &#58; 795.0/1550 (+327,=936,-287&#41;

Elo  +9


    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 --------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3006   11  11  1550    51.7 %   2994   59.2 %
  2 Stockfish 5 T1   &#58; 2994   11  11  1550    48.3 %   3006   59.2 %

Result &#58; 802.0/1550 (+343,=918,-289&#41;

Elo  +12                   TB benefit&#58; 3 Elo

The use of Syzygy tablebases seems only marginally increasing the playing strength of Komodo 8 against Stockfish 5, in average 4 Elo. The gain decreases with increasing time limit.
But due to the error bars, even with this large number of games, this statement is not necessarily cogent, even if the trend suggests this.

Once again Komodo shows a increase in strenght with increasing time limit.

Next is a test with the upcoming Stockfish 6, then the test Komodo 8 against Stockfish 6.

Andreas

shrapnel · Post by **shrapnel** » Thu Jan 22, 2015 4:33 pm

cc2150dx wrote:I mainly use 5 men syzygy bases to determine the wins/loss/draws quicker.

And, off-topic I use the bases on a regular HDD with Houdini, SF, K with no slowdowns.

Jason

You may not notice it, but there will DEFINITELY be slowdowns if you don't use an SSD.
SSDs are so much faster than HDDs....almost unbelievable the difference it makes.
And off-topic, I recently wanted to add a large-capacity HDD to my System, but the thought of installing a large, clunky HDD so put me off that I decided to go with a large capacity SSD instead, even though it cost me more.
No regrets.
Believe me, all your programs ( not just chess) will run so much faster with an SSD that it will make you wonder how you ever put up with HDDs !
On second thoughts, DON'T upgrade to SSD !

Once you do, there is no going back to HDDs !

Norm Pollock · Post by **Norm Pollock** » Fri Jan 23, 2015 2:22 pm

In between no TBs and SSD, are usb 2.0 and usb 3.0 flash drives. Does anyone know how they rate compared to an SSD?

syzygy · Post by **syzygy** » Sat Jan 24, 2015 1:39 am

cc2150dx wrote:And, off-topic I use the bases on a regular HDD with Houdini, SF, K with no slowdowns.

Since the usual ignoramus felt a need to contest what you wrote here, I'll just say that 5-piece TBs on a regular HDD indeed won't cause a slowdown on a modern PC as the data will be cached in RAM (ignoring the first seconds of TB access after a reboot of the PC).

lucasart · Post by **lucasart** » Sat Jan 24, 2015 5:37 am

fastgm wrote:What are the benefits of using 5-men Syzygy tablebases?

Here a test with Komodo 8 on one core with 128 MB Hash.
The tablebases were on a solid-state drive - Samsung 840 EVO.
Four different time controls were tested, each 3000 games or 1520 at highest time control.
15 sec + 0.15 sec / 60 sec + 0.60 sec / 240 sec + 2.40 sec / 960 sec + 9.60 sec

Code: Select all

TC  = 15"+0.15"
TB5 = 5-men Syzygy tablebases
T1  = 1 thread
 
    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3005   8   8   3000    51.4 %   2995   60.8 %
  2 Komodo 8 T1      &#58; 2995   8   8   3000    48.6 %   3005   60.8 %

Result &#58; 1542.5/3000 (+630,=1825,-545&#41;
 
Elo  +10 
 
TC  = 60"+0.6"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3009   7   7   3000    52.5 %   2991   68.2 %
  2 Komodo 8 T1      &#58; 2991   7   7   3000    47.5 %   3009   68.2 %

Result &#58; 1576.0/3000 (+553,=2046,-401&#41;
 
Elo  +18 
 
TC  = 4'+2.4"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3006   6   6   3000    51.7 %   2994   75.4 %
  2 Komodo 8 T1      &#58; 2994   6   6   3000    48.3 %   3006   75.4 %

Result &#58; 1551.5/3000 (+420,=2263,-317&#41;
 
Elo  +12
  
TC  = 16'+9.6"

    Program            Elo    +   -   Games   Score   Av.Op.  Draws
 ---------------------------------------------------------------------
  1 Komodo 8 T1 TB5  &#58; 3006   8   8   1520    51.7 %   2994   79.8 %
  2 Komodo 8 T1      &#58; 2994   8   8   1520    48.3 %   3006   79.8 %

Result &#58; 785.5/1520 (+179,=1213,-128&#41;
 
Elo  +12

This is very much in line with the result I measured myself a while ago, which was +10 elo in 15"+0.15". However, I never had resources and patience to do longer tests.

It is interesting to note that the elo gain does not shrink at longer tc, even at the very long 16'+9.6". Normally any elo gain tends to shrink at longer tc, as draw rate increases, so anything that does not shrink at this very long tc is remarkable.

I wish Marco Costalba could read this, and stop claiming that syzygy TB have zero elo impact.

Jouni · Post by **Jouni** » Sat Jan 24, 2015 10:00 am

So we get +4 ELO from 5 piece set. IPON test gave +6 for Stockfish. Isn't these very modest gain from 960 MB of data?? There has been +5 gain from 10 parameters change in Stockfish testing.

Komodo 8 - 5-men Syzygy tablebases

Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases

Re: Komodo 8 - 5-men Syzygy tablebases