What happens using egbb

[michiguel]

I managed to get a surprisingly fast conclusive result for Shredder 12 using endgame bases. Shredder 12 + Nalimov + egbb against Shredder 12 without any bases. Standard opening 8-move positions, TC 15''+0.15'', LOS 99.9% as stopping rule.

Code: Select all

Games Completed = 3315 of 30000 (Avg game length = 43.826 sec)
Settings = Gauntlet/16MB/15000ms+150ms/M 700000cp for 1000 moves, D 150000 moves/PGN:C:\LittleBlitzer\swcr.pgn(5120)
Time = 39180 sec elapsed, 315392 sec remaining
 1.  Shredder EGBB            	1723.5/3315	881-749-1685  	(L: m=749 t=0 i=0 a=0)	(D: r=1344 i=283 f=50 s=8 a=0)	(tpm=312.2 d=10.25 nps=765881)
 2.  Shredder                 	1591.5/3315	749-881-1685  	(L: m=881 t=0 i=0 a=0)	(D: r=1344 i=283 f=50 s=8 a=0)	(tpm=328.0 d=11.79 nps=892042)

The NPS is significantly lower using bases, but the net benefit is evident:

Code: Select all

    Program                            Score       %    Elo    +   -    Draws

  1 Shredder EGBB                  : 1723.5/3315  52.0    7    8   8   50.8 %
  2 Shredder                       : 1591.5/3315  48.0   -7    8   8   50.8 %
 

14 +/- 8 (2SD) ELO points benefit for Shredder using Nalimov + egbb, LOS 99.95%. This is the first time I get a conclusive result using endgame bases, and the benefit is pretty substantial. Ferdy got a benefit with Scorpio egbb, and all this amounts to disproving the skeptics of endgame bases (especially egbb) ELO-wise benefits. Maybe I will manage a match Shredder + Nalimov + egbb vs. Shredder + Nalimov.

Most of the criticisms on bases in general were based on hearsay and whatever the people tried or heard someone tried with Nalimovs. Moreover, the myth was generated that engine may even play worse.

Or course there is caveat regarding how they are implemented.

Miguel

[Laskos]

I managed to get a surprisingly fast conclusive result for Shredder 12 using endgame bases. Shredder 12 + Nalimov + egbb against Shredder 12 without any bases. Standard opening 8-move positions, TC 15''+0.15'', LOS 99.9% as stopping rule.

Code: Select all

Games Completed = 3315 of 30000 (Avg game length = 43.826 sec)
Settings = Gauntlet/16MB/15000ms+150ms/M 700000cp for 1000 moves, D 150000 moves/PGN:C:\LittleBlitzer\swcr.pgn(5120)
Time = 39180 sec elapsed, 315392 sec remaining
 1.  Shredder EGBB            	1723.5/3315	881-749-1685  	(L: m=749 t=0 i=0 a=0)	(D: r=1344 i=283 f=50 s=8 a=0)	(tpm=312.2 d=10.25 nps=765881)
 2.  Shredder                 	1591.5/3315	749-881-1685  	(L: m=881 t=0 i=0 a=0)	(D: r=1344 i=283 f=50 s=8 a=0)	(tpm=328.0 d=11.79 nps=892042)

The NPS is significantly lower using bases, but the net benefit is evident:

Code: Select all

    Program                            Score       %    Elo    +   -    Draws

  1 Shredder EGBB                  : 1723.5/3315  52.0    7    8   8   50.8 %
  2 Shredder                       : 1591.5/3315  48.0   -7    8   8   50.8 %
 

14 +/- 8 (2SD) ELO points benefit for Shredder using Nalimov + egbb, LOS 99.95%. This is the first time I get a conclusive result using endgame bases, and the benefit is pretty substantial. Ferdy got a benefit with Scorpio egbb, and all this amounts to disproving the skeptics of endgame bases (especially egbb) ELO-wise benefits. Maybe I will manage a match Shredder + Nalimov + egbb vs. Shredder + Nalimov.

Most of the criticisms on bases in general were based on hearsay and whatever the people tried or heard someone tried with Nalimovs. Moreover, the myth was generated that engine may even play worse.

Or course there is caveat regarding how they are implemented.

Miguel

I myself was a bit skeptical, as with Nalimov EGTB I had a totally inconclusive large test a year or so ago

Code: Select all

    Program                            Score         %      Elo   +   -   Draws

  1 Houdini 3                      : 24159.5/48300  50.0     0    2   2   64.4 %
  2 Houdini 3 Nalimov              : 24140.5/48300  50.0    -0    2   2   64.4 %

So, Nalimov TB give 0 +/- 2 points to Houdini, with default settings of TB usage and on HD, maybe on SSD it's different. It seems that RAM loaded 345 egbb really make the difference, in my current test I am already getting a LOS of 98% in Shredder + Nalimov + egbb vs. Shredder + Nalimov, will have to wait for a probable stop at 99.9%. I remember testing Nalimov with other engines years ago, with inconclusive results. As you say, the implementation and the usage are very important.

[Laskos]

The test Shredder + Nalimov + egbb vs. Shredder + Nalimov shows that all of the benefit from using endgame bases in Shredder comes from egbb. Standard opening 8-move positions, TC 15''+0.15'', LOS 99.9% as stopping rule.

Code: Select all

    Program                            Score       %      Elo    +   -    Draws

  1 Shredder Nalimov + EGBB        : 1633.5/3141  52.0      7    9   9   48.9 %
  2 Shredder Nalimov               : 1507.5/3141  48.0     -7    9   9   48.9 % 

14 +/- 9 (2SD) ELO points benefit, identical to the previous test, LOS=99.92%. It seems that, at least in Shredder, Nalimov TBs by themselves don't bring ELO-wise benefit, while egbb loaded into RAM give some 14 ELO points.

[Daniel Shawul]

Note that Shredder uses Nalimov tbs only at the root in all cases since it assumes egbbs are available. Many other engines use Nalimov tbs alone at all internal nodes, so this test only shows egbbs do help. Shredder uses Nalimov to guide mating while scorpio doesn't need it and it seems that both perform similarly comparing yours and Ferd's results. However to test if there is any gain from DTM tables, you will need an engine that actually uses them inside search. But you said you tested DTM tables in the past with a 0+-2 gain, so the story probably haven't changed there.

[Laskos]

Note that Shredder uses Nalimov tbs only at the root in all cases since it assumes egbbs are available. Many other engines use Nalimov tbs alone at all internal nodes, so this test only shows egbbs do help. Shredder uses Nalimov to guide mating while scorpio doesn't need it and it seems that both perform similarly comparing yours and Ferd's results. However to test if there is any gain from DTM tables, you will need an engine that actually uses them inside search. But you said you tested DTM tables in the past with a 0+-2 gain, so the story probably haven't changed there.

Yes, those were Nalimovs with Houdini, 0 +/- 2 points.

[Daniel Shawul]

Unless Houdart screwed up there too using Nalimove , I think we can make some kind of preliminary conclusions. Thanks for the tests.

[Ryan Benitez]

Thanks Norman! I warned him that the audience is not gullible...

Sorry but his audience is that gullible. He could point his user directly to the robbolito web site to download robbobases and they would pay him extra.

[Laskos]

Note that Shredder uses Nalimov tbs only at the root in all cases since it assumes egbbs are available. Many other engines use Nalimov tbs alone at all internal nodes, so this test only shows egbbs do help. Shredder uses Nalimov to guide mating while scorpio doesn't need it and it seems that both perform similarly comparing yours and Ferd's results. However to test if there is any gain from DTM tables, you will need an engine that actually uses them inside search. But you said you tested DTM tables in the past with a 0+-2 gain, so the story probably haven't changed there.

Yes, those were Nalimovs with Houdini, 0 +/- 2 points.

The test with Syzygy 3-4-5 TBs on HD and Stockfish 21102013 single core. TC 15''+0.05'', 8-move opening positions, LOS 99.9% stopping.

Code: Select all

    Program                            Score      %    Elo    +   -   Draws

  1 SF 21102013 Syzygy            : 1349.5/2590  52.1    7    8   8   61.1 %
  2 SF 21102013 No TB             : 1240.5/2590  47.9   -7    8   8   61.1 %
 

15 +/- 8 (2SD) Elo points benefit from Syzygy TBs for Stockfish (LOS 99.97%), similar to that of Scorpio EGBBs and Shredderbases. The NPS drop was roughly 2%.