My troubles with MultiPV and Syzygy in Stockfish 7

[hgm]

Yes, I guess you are right and that it works very poorly. That must be why so many engine authors have instead decided to implement your imaginary system.

You seem unaware that what you call an 'imaginary system' is actually the standard method underlying the design of DTZ EGTs. The primary goal is to convert, and only when that is not possible the secondary goal is to make progress towards a conversion.

I guess the lesson is that when you use a hideous kludge instead of a logical method, you cannot expect it to work automatically in any situation that it was not specifically designed for.

[petero2]

Wow, Little Blitzer is buggy when adjudicating 50 move draw. The missed Win cases are hard to reproduce. I will check later this night when I get back home.

Although for Wins I am still getting some draws with Texel (one in 500 games or so) in Cutechess-cli, I observe some odd behavior, the draws seem to cluster and appear at the start of the run. Also, the wrong move is the first move in the game. I guess it's UI-engine communication. Here are 2 games:

From the first game:

Code: Select all

1. Ra1 {0.00/1 0.42s}

After 0.42 seconds texel had not been able to finish the depth 2 search. I guess this happens because the DTZ files are on a mechanical disk and they have not been used recently enough to be in cache. The texel TB implementation prefers to not use TBs if it would overstep the allocated thinking time too much, but the stockfish implementation would probe DTZ at the root regardless of how much time that would take I think.

So this seems to be a combination of using hyper bullet time control, starting from a position already in the TBs, having the DTZ tables on a slow (mechanical) disk drive, and using an engine that prefers not using the TBs over using too much thinking time.

In a game not starting from a TB position, the first DTZ probe in texel would typically happen inside a deep search when texel has proven a TB win and starts trying to optimize the DTM score. Using the same hardware, there would likely be 0.42s delay there too, but that would not really matter because the search has already found a forced win, and when starting a new search after the opponents next move, the required part of the DTZ table would already be cached.

[syzygy]

Yes, I guess you are right and that it works very poorly. That must be why so many engine authors have instead decided to implement your imaginary system.

You seem unaware that what you call an 'imaginary system'

Imaginary I call something that exists only in your imagination. Build something first, then talk. But no, you prefer to show off your ignorance about "Syzygy in Stockfish 7" (see thread title).

[Laskos]

From the first game:

Code: Select all

1. Ra1 {0.00/1 0.42s}

After 0.42 seconds texel had not been able to finish the depth 2 search. I guess this happens because the DTZ files are on a mechanical disk and they have not been used recently enough to be in cache. The texel TB implementation prefers to not use TBs if it would overstep the allocated thinking time too much, but the stockfish implementation would probe DTZ at the root regardless of how much time that would take I think.

So this seems to be a combination of using hyper bullet time control, starting from a position already in the TBs, having the DTZ tables on a slow (mechanical) disk drive, and using an engine that prefers not using the TBs over using too much thinking time.

In a game not starting from a TB position, the first DTZ probe in texel would typically happen inside a deep search when texel has proven a TB win and starts trying to optimize the DTM score. Using the same hardware, there would likely be 0.42s delay there too, but that would not really matter because the search has already found a forced win, and when starting a new search after the opponents next move, the required part of the DTZ table would already be cached.

Yes, it's confirmed by the fact that from Syzygy on USB flash drive these things don't happen.

I also tried to tentatively test the quality of implementation by using a file of hard 6-men Wins having only 5-men Syzygy. On these openings, the sensitivity to Syzygy is high. It seems Syzygy helps Texel significantly more than they help Stockfish:

Code: Select all

Score of SF7 No TB vs Texel 1.06 No TB: 291 - 74 - 353  [0.651] 718
ELO difference: 108
Finished match

Score of SF7 Syzygy vs Texel 1.06 Syzygy: 257 - 166 - 295  [0.563] 718
ELO difference: 44
Finished match

Do you have a reason for Texel implementation of Syzygy being better than Stockfish one?

[syzygy]

Do you have a reason for Texel implementation of Syzygy being better than Stockfish one?

If Stockfish profits less, it is because Stockfish has less need for them when playing these endgames.

It is to be expected that giving both engines "perfect knowledge" of 5-piece endgames will bring the two engines closer to each other in strength.

[Laskos]

Do you have a reason for Texel implementation of Syzygy being better than Stockfish one?

If Stockfish profits less, it is because Stockfish has less need for them when playing these endgames.

The result seems too peculiar in its magnitude. Also, Stockfish, as good as it might be, still misses the majority of the hard 6-men wins even having the perfect 5-men game, so the magnitude of the effect you describe should be moderate.

[syzygy]

Do you have a reason for Texel implementation of Syzygy being better than Stockfish one?

If Stockfish profits less, it is because Stockfish has less need for them when playing these endgames.

The result seems too peculiar in its magnitude.

Why?

If you give each engine access to 6-piece endgames, the result will be 0 Elo difference. Does that mean that Texel's implementation is "better"?

[Laskos]

Do you have a reason for Texel implementation of Syzygy being better than Stockfish one?

If Stockfish profits less, it is because Stockfish has less need for them when playing these endgames.

The result seems too peculiar in its magnitude.

Why?

If you give each engine access to 6-piece endgames, the result will be 0 Elo difference. Does that mean that Texel's implementation is "better"?

Stockfish play even enabled with 5-men bases is far from perfect, in fact it misses about a half of the 6-men hard Wins. I am not saying, though, that the effect described by you is necessarily small, that's why I asked Peter whether it is possible Texel has a different game-play implementation (not just aesthetically different).

[petero2]

I also tried to tentatively test the quality of implementation by using a file of hard 6-men Wins having only 5-men Syzygy. On these openings, the sensitivity to Syzygy is high. It seems Syzygy helps Texel significantly more than they help Stockfish:

Code: Select all

Score of SF7 No TB vs Texel 1.06 No TB: 291 - 74 - 353  [0.651] 718
ELO difference: 108
Finished match

Score of SF7 Syzygy vs Texel 1.06 Syzygy: 257 - 166 - 295  [0.563] 718
ELO difference: 44
Finished match

Do you have a reason for Texel implementation of Syzygy being better than Stockfish one?

I would like Ronald guess that this is mainly caused by texel being weaker to begin with.

Another difference is that texel has "TB swindle mode" permanently enabled, which means that it does not give a 0 score for a TB draw. Cursed wins are scored between 0.35 and 0.70 depending on how far away they are from a real win, and "normal draws" are scored between 0 and 0.34 depending on how good texel's normal search+eval think the position is. (And correspondingly for negative scores between 0 and -0.70).

I don't know if this swindle mode gives any elo advantage in your setup. I can prepare a special version that lets you disable swindle mode if you want to test this theory.

[Laskos]

I also tried to tentatively test the quality of implementation by using a file of hard 6-men Wins having only 5-men Syzygy. On these openings, the sensitivity to Syzygy is high. It seems Syzygy helps Texel significantly more than they help Stockfish:

Code: Select all

Score of SF7 No TB vs Texel 1.06 No TB: 291 - 74 - 353  [0.651] 718
ELO difference: 108
Finished match

Score of SF7 Syzygy vs Texel 1.06 Syzygy: 257 - 166 - 295  [0.563] 718
ELO difference: 44
Finished match

Do you have a reason for Texel implementation of Syzygy being better than Stockfish one?

I would like Ronald guess that this is mainly caused by texel being weaker to begin with.

Another difference is that texel has "TB swindle mode" permanently enabled, which means that it does not give a 0 score for a TB draw. Cursed wins are scored between 0.35 and 0.70 depending on how far away they are from a real win, and "normal draws" are scored between 0 and 0.34 depending on how good texel's normal search+eval think the position is. (And correspondingly for negative scores between 0 and -0.70).

I don't know if this swindle mode gives any elo advantage in your setup. I can prepare a special version that lets you disable swindle mode if you want to test this theory.

Yes, I will test it in the evening.