7-men Syzygy attempt

[Isaac]

OK, what you meant is that playing with a time control as opposed to fixed depth lets both engines search deeper in endgame positions. That is indeed true and fixing depth indeed artificially lowers the level of endgame play of both engines.

I think you're again missing Ronald's point.

In addition, it seems to me that you set this fixed depth limit to have a clear picture of the benefits of TB. If that was your goal, the method you employed to achieve it is flawed. Ronald pointed this out several times.

I suggest rereading what he wrote in his last answers here.

[syzygy]

OK, what you meant is that playing with a time control as opposed to fixed depth lets both engines search deeper in endgame positions. That is indeed true and fixing depth indeed artificially lowers the level of endgame play of both engines.

I think you're again missing Ronald's point.

But now you're quoting my text

But indeed my point is that increased search depth is one of the advantages of TBs.

Ozymandias is right that using a time control instead of fixed depth increases depth in endgames whether you use TBs or not. But that was not my point.

Ozymandias' point is also an important reason why testing at fixed depth is flawed. If you fix depth to just 15 ply or so all top engines will be absolutely terrible at playing endgames. So I should have included this in my initial list.

So there are at least three reasons why fixed-depth tests are not useful for estimating the Elo gain from TBs:
- fixed depth hides the speed penalty (lower nps) incurred by probing TBs;
- fixed depth hides the benefit of reaching larger depths in late middlegame and endgame when using TBs; and
- fixed depth artificially cripples endgame play of top engines, which gives too much weight to the advantage of having perfect knowledge close to the root.

[Isaac]

I indeed messed up with the quoting part.
Now your points are, I hope, crystal clear.

[syzygy]

I just noticed a problem:
https://lichess.org/analysis/standard/6 ... /1B6/KQ6_w_-_-

The best move is Qg6+ with DTZ 604. Then Kf8 (only move) with DTZ 603. But then Qg7+ has DTZ 104??

Interestingly, after Ke8 the best move is Nc7+ with DTZ 600, which seems correct again. But something is wrong in between.

Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.

[Ozymandias]

Testing TBs at fixed depth will not give a complete picture. On the one hand, they take time to probe. On the other hand, they allow the search to go deeper. By fixing the depth you eliminate these two important components from the equation.

What do you mean by "time to probe"? CPU throttle? There isn't.

Time to probe is simply time to probe. It takes many cpu cycles to probe a position, and data may have to be read from SSD.

Not shooting for the obvious, but everything takes time, aren't those CPU cycles equivalent to the ones needed to reach the same depth without TBs?

As the test shows, more depth equals less of a gap, if you increase said depth you'll hardly register any differences at all.

No, your test artificially limits the depth reached by the engine playing with TBs. At the same time, it artificially removes the speed penalty of probing.

Limiting depth is less of an intrusion than limiting time (more on that later), and both the engines playing with TBs, as well as the one without, were depth limited. I think that's fair testing.

What's "the speed penalty of probing"? When the HDD doesn't get in the way, I've noticed none.

Just because it's the more chosen time control, it doesn't mean you should jump onboard blindly.

Playing with fixed depth is simply not a time control at all.

Only in the sense that the TC isn't fixed, total time per game is variable because the engine requires more or less time, depending on how difficult the positions involved turn out to be. For engine testing, it makes sense to externally influence the engine, imposing a deadline, but looking at TBs performance is different. Let me explain with an example, I played two quick matches between asmFish 9 at depth 12, and asmFish 9 with a TC of 25 moves per second (equivalent total time on that PC); in one of them, asmFish 9 25/1 used 6-men and in the other, no TBs. The difference in engine's strength was between the margin of error, does that mean that TBs only add a few Elo points? No, it means that when the engine has to manage time allocation, it saves more than enough for the "ending" (past move 53) and that doubling time or probing TBs will have little effect.

[noobpwnftw]

I just noticed a problem:
https://lichess.org/analysis/standard/6 ... /1B6/KQ6_w_-_-

The best move is Qg6+ with DTZ 604. Then Kf8 (only move) with DTZ 603. But then Qg7+ has DTZ 104??

Interestingly, after Ke8 the best move is Nc7+ with DTZ 600, which seems correct again. But something is wrong in between.

Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.

Just checked that Cfish gives the same results on my local table, rebuilding to see what happens.

[syzygy]

Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.

It does correct itself if you play Qb3+:
Be1 (562), Qc4 (561), Kb1 (222 ??), Qb3+ (103 ??), Kb2 (558).

The problem is the position after Qb3+ probing as having DTZ=103.
The position after Kb1 is not in the half-sided DTZ table, so probing that position causes a 1-ply round of probing, which would have given the correct result if the position after Qb3+ had returned the correct value.

[noobpwnftw]

Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.

It does correct itself if you play Qb3+:
Be1 (562), Qc4 (561), Kb1 (222 ??), Qb3+ (103 ??), Kb2 (558).

The problem is the position after Qb3+ probing as having DTZ=103.
The position after Kb1 is not in the half-sided DTZ table, so probing that position causes a 1-ply round of probing, which would have given the correct result if the position after Qb3+ had returned the correct value.

Then it looks like a probing problem of the site?

[syzygy]

The problem is the position after Qb3+ probing as having DTZ=103.

DTZ=102 to be precise.

The first problem is caused by the position after Ke8 incorrectly returning DTZ=102.

[syzygy]

Some moves later the problem repeats itself:
https://lichess.org/analysis/standard/8 ... 3N4/8/K7_w_-_-
But now it does not correct itself and a few moves later it starts to get a bit random.

It does correct itself if you play Qb3+:
Be1 (562), Qc4 (561), Kb1 (222 ??), Qb3+ (103 ??), Kb2 (558).

The problem is the position after Qb3+ probing as having DTZ=103.
The position after Kb1 is not in the half-sided DTZ table, so probing that position causes a 1-ply round of probing, which would have given the correct result if the position after Qb3+ had returned the correct value.

Then it looks like a probing problem of the site?

If Cfish showed the same problem, then the problem is probably in the table.

The WDL table seems to be OK and the DTZ table seems to be OK for most positions. But several positions have the wrong DTZ value.

There also problems with non-cursed wins:
https://lichess.org/analysis/standard/8 ... 5/2n5/8_b-_-
DTZ=17 for Qe3, then DTZ=7