Cache over-writing and PV's

[gladius]

Note that the design goal of Stockfish is not to be a good or useful engine. Its goal is to optimize beating a small set of opponents in bullet games under conditions where hash replacement virtually does not occur. Any patches that would improve its hashing, or solve the problems that you are experiencing, would be instantly rejected, as they would not cause a higher win rate in the bulet games.

This is just not true. The goal is indeed to be a "good and useful" engine. The *means* to achieve this is indeed by self-testing in bullet matches most of the time. This has proven to be a very effective strategy for improving strength. Changes that affect hash strategy are certainly tested at hash levels where replacement plays a major role (4mb hash at 60s).

You are also wrong about such a fix being "instantly rejected". Where does this extremely pessimistic view of SF development come from?

Yes, the main goal is definitely to increase ELO, but usability of the engine is also a big goal. For example, the issue with SF having a lot of trouble finding mates in SF5 was fixed recently, and was an ELO neutral fix at best.

[gladius]

All opinions and thoughts on the subject are welcome.

I took a look at the search output, and I think everything looks okay actually. SF is not actually losing the mate, it's finding shorter and shorter mates as it goes deeper. You should ignore the fail high/low output of SF along the way (ie. looks like 62/75- or 62/75+ instead of just 62/75), if we remove that your search log looks like this:

The fail high/low output is the result of an incomplete search, it's just meant to give an idea of what the engine is working on - not what it's current best score/line is. Once the fail high/low has been resolved, you get the actual output, and SF sticks to the mate in it's main lines.

...
31/47 00:03.460 24,213k 6,998k +59.75 43. ... Kc6 44.Be6 Na6 45.f5 Kc7 46.fxg6 hxg6 47.Rg4 Nc5 48.Rc4 Kd6 49.Rxc5 Kxc5 50.h4 Kd6 51.Bf7 g5 52.hxg5 Ke7 53.g6 Kf8 54.a5 Kg7 55.Kb4 Kf8 56.a6 Kg7 57.a7 Kf6 58.a8Q Ke7 59.g7 Kxf7 60.g8Q+ Kf6 61.Qh8+ Kg6
32/47 00:03.517 24,595k 6,993k +59.75 43. ... Kc6 44.Be6 Na6 45.f5 Kc7 46.fxg6 hxg6 47.Rg4 Nc5 48.Rc4 Kd6 49.Rxc5 Kxc5 50.h4 Kd6 51.Bf7 g5 52.hxg5 Ke7 53.g6 Kf8 54.a5 Kg7 55.Kb4 Kf8 56.a6 Kg7 57.a7 Kf6 58.a8Q Ke7 59.g7 Kxf7 60.g8Q+ Kf6 61.Qh8+ Kg6
33/47 00:03.668 25,642k 6,991k +59.75 43. ... Kc6 44.Be6 Na6 45.f5 Kc7 46.fxg6 hxg6 47.Rg4 Nc5 48.Rc4 Kd6 49.Rxc5 Kxc5 50.h4 Kd6 51.Bf7 g5 52.hxg5 Ke7 53.g6 Kf8 54.a5 Kg7 55.Kb4 Kf8 56.a6 Kg7 57.a7 Kf6 58.a8Q Ke7 59.g7 Kxf7 60.g8Q+ Kf6 61.Qh8+ Kg6
34/47 00:03.728 26,085k 6,997k +59.75 43. ... Kc6 44.Be6 Na6 45.f5 Kc7 46.fxg6 hxg6 47.Rg4 Nc5 48.Rc4 Kd6 49.Rxc5 Kxc5 50.h4 Kd6 51.Bf7 g5 52.hxg5 Ke7 53.g6 Kf8 54.a5 Kg7 55.Kb4 Kf8 56.a6 Kg7 57.a7 Kf6 58.a8Q Ke7 59.g7 Kxf7 60.g8Q+ Kf6 61.Qh8+ Kg6 62.Qc6+
35/51 00:13.651 101,418k 7,429k +M29 43. ... h6 44.Be6 Be8 45.f5 Nc6 46.f6 Ne5 47.Re4 Ng6 48.Bf5 Bf7 49.Bxg6 Bxg6 50.Rg4 Bh5 51.Rh4 Bf7 52.Rxh6 Kd6 53.Rh7 Bg6 54.Ra7 Ke6
36/51 00:30.541 227,743k 7,457k +M28 43. ... h5 44.Rd5 Be8 45.Bb5 Bf7 46.Rf5 Be6 47.Rxh5 Bg4 48.Rh7+ Kb6 49.Rh6+ Ka5 50.Rh8 Nd7 51.Bxd7 Bxd7 52.Rh7 Be6 53.f5 Bg8 54.Re7 Kxa4 55.Rg7 Bd5 56.f6 Kb5 57.Rg5 Kc6 58.Rxd5
37/51 00:31.176 232,078k 7,444k +M28 43. ... h5 44.Rd5 Be8 45.Bb5 Bf7 46.Rf5 Be6 47.Rxh5 Bg4 48.Rh7+ Kb6 49.Rh6+ Ka5 50.Rh8 Nd7 51.Bxd7 Bxd7 52.Rh7 Be6 53.f5 Bg8 54.Re7 Kxa4 55.Rg7 Bd5 56.f6 Kb5 57.Rg5 Kc6 58.Rxd5
38/51 00:35.573 262,931k 7,391k +M28 43. ... h5 44.Rd5 Be8 45.Bb5 Bf7 46.Rf5 Be6 47.Rxh5 Bg4 48.Rh7+ Kd8 49.a5 Bf5 50.Rh8+ Kc7 51.Rh6 Kb7 52.Rf6 Be4 53.a6+ Nxa6 54.Bxa6+ Kc7 55.Bd3 Bf3 56.f5 Kd8 57.Rh6 Kc7 58.f6 Bd5 59.Bc4 Bxc4 60.Kxc4 Kd6 61.Rh7 Ke6 62.f7 Ke7 63.Kd5 Kf8 64.Kc5 Ke7 65.f8R+ Kxf8 66.Ra7
39/51 00:41.484 323,672k 7,802k +M28 43. ... h5 44.Rd5 Be8 45.Bb5 Bf7 46.Rf5 Be6 47.Rxh5 Bg4 48.Rh7+ Kd8 49.a5 Bf5 50.Rh8+ Kc7 51.Rh6 Kb7 52.Rf6 Be4 53.a6+ Nxa6 54.Bxa6+ Kc7 55.Bd3 Bf3 56.f5 Kd8 57.Rh6 Kc7 58.f6 Bd5 59.Bc4 Bxc4 60.Kxc4 Kd6 61.Rh7 Ke6 62.f7 Ke7 63.Kd5 Kf8 64.Kc5 Ke7 65.f8R+ Kxf8
40/51 00:57.943 482,098k 8,320k +M28 43. ... h5 44.Rd5 Be4 45.Rxh5 Nd7 46.a5 Kb7 47.Rh6 Nf8 48.Rf6 Nd7 49.a6+ Ka8 50.Re6 Bh1 51.Re8+ Ka7 52.Re7 Bc6 53.f5 Kb6 54.Rxd7 Bxd7 55.f6 Be8
41/54 01:01.566 518,168k 8,416k +M28 43. ... h5 44.Rd5 Be4 45.Rxh5 Nd7 46.a5 Kb7 47.Rh6 Nf8 48.Rf6 Nd7 49.a6+ Ka8 50.Re6 Bf3 51.Re8+ Ka7 52.Re7 Bc6 53.f5 Kb6 54.Rxd7 Bxd7 55.f6 Be8
42/56 01:16.226 664,758k 8,721k +M28 43. ... h5 44.Rd5 Be4 45.Rxh5 Nd7 46.a5 Kb7 47.Rh6 Nf8 48.Rf6 Nd7 49.a6+ Ka8 50.Re6 Bf3 51.Re8+ Ka7 52.Re7 Bc6 53.f5 Kb6 54.Rxd7 Bxd7 55.f6 Be8
43/56 01:55.052 1,054,333k 9,164k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
44/57 02:03.174 1,142,733k 9,277k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
45/57 02:06.098 1,173,265k 9,304k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
46/57 02:34.998 1,454,168k 9,382k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
47/57 03:03.619 1,748,551k 9,523k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
48/57 03:12.095 1,840,669k 9,582k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
49/57 04:05.105 2,391,383k 9,757k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
50/57 04:46.988 2,829,157k 9,858k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
51/57 06:50.326 4,074,104k 9,929k +M28 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2
52/67 48:36.682 29,443,642k 10,095k +M26 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2 Bf3 47.Re7+ Kd6 48.Rxh7 Nd7 49.a5 Be4 50.Rh6+ Kc5 51.Rh5+ Kd6 52.Kd4 Bf3 53.Rh6+ Kc7
53/71 5:06:52.470 185,058,793k 10,051k +M26 43. ... Bf5 44.Rd5 Be4 45.Re5 Bg2 46.Re2 Bf3 47.Re7+ Kd6 48.Rxh7 Nd7 49.a5 Be4 50.Rh6+ Kc5 51.Rh5+ Kd6
54/71 7:04:25.286 256,362,361k 10,067k +M24 43. ... Bf5 44.Rd5 Bg6 45.f5 Bh5 46.f6 Bg6 47.Rd4 Nc6 48.f7 Bxf7 49.Bxf7 Ne5 50.Bh5 Kb6 51.Rd5 Nc6 52.Rd6 Kc7 53.Rxc6+ Kxc6
55/71 7:33:53.106 274,534,343k 10,081k +M24 43. ... Bf5 44.Rd5 Bg6 45.f5 Bh5 46.f6 Bg6 47.Rd4 Nc6 48.f7 Bxf7 49.Bxf7 Ne5 50.Bh5 Kb6 51.Rd5 Nc6 52.Rd6 Kc7 53.Rxc6+ Kxc6 54.Kd4 Kc7 55.Bf7
56/75 25:07:30.658 899,477,435k 9,944k +M24 43. ... Bf5 44.Rd5 Bg6 45.f5 Bh5 46.f6 Bg6 47.Rd4
57/75 30:12:54.512 1,086,334,318k 9,987k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
58/75 35:11:25.737 1,269,825,516k 10,023k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
59/75 38:14:36.866 1,386,206,840k 10,069k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
60/75 43:51:45.495 1,598,624,748k 10,124k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
61/75 51:01:08.022 1,866,180,440k 10,161k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5

[syzygy]

Note that the design goal of Stockfish is not to be a good or useful engine. Its goal is to optimize beating a small set of opponents in bullet games under conditions where hash replacement virtually does not occur. Any patches that would improve its hashing, or solve the problems that you are experiencing, would be instantly rejected, as they would not cause a higher win rate in the bulet games.

This is just not true. The goal is indeed to be a "good and useful" engine. The *means* to achieve this is indeed by self-testing in bullet matches most of the time. This has proven to be a very effective strategy for improving strength. Changes that affect hash strategy are certainly tested at hash levels where replacement plays a major role (4mb hash at 60s).

I'm still waiting for any of the critics to explain how they are succesfully using 10-minute searches to improve their engines. Those engines must be really excellent at playing long time controls! I'm sure they'd blow SF away.

Oh wait...

[hgm]

Well, Fairy-Max would of course beat Stockfish by something like 36-2 at any TC. It would only lose the Chess and Chess960 games.

And it was mostly tuned using 40/5' games.

[xmas79]

I took a look at the search output, and I think everything looks okay actually...

57/75 30:12:54.512 1,086,334,318k 9,987k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
58/75 35:11:25.737 1,269,825,516k 10,023k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
59/75 38:14:36.866 1,386,206,840k 10,069k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
60/75 43:51:45.495 1,598,624,748k 10,124k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5
61/75 51:01:08.022 1,866,180,440k 10,161k +M24 43. ... Nc6 44.Rd5 Ne7 45.Re5

Oh boy, a search at depth 61 with only... one..two..three...mmmhh four... wow.. four moves, which should have 48 moves to show the +M24 REALLY looks OK to you?? Please... As a programmer I can understand all the whole output (including fail high and fail low which are a fundamental part of the search) and agree with you that everything actually IS ok (since the engine will play the correct move anyway), but from a pure user point of view (which seems to me the OP POV), this is a HUGE problem IMHO.

Just my 2 cents,
Natale.

[hgm]

Indeed, I also remarked upon that before: reporting fail lows is a curse rather than a boon. It might be very useful to engine developers, but for the average user it is a disaster.

[syzygy]

Oh boy, a search at depth 61 with only... one..two..three...mmmhh four... wow.. four moves, which should have 48 moves to show the +M24 REALLY looks OK to you??

Yes, it does. Have you never used a chess engine before?

[xmas79]

Oh boy, a search at depth 61 with only... one..two..three...mmmhh four... wow.. four moves, which should have 48 moves to show the +M24 REALLY looks OK to you??

Yes, it does. Have you never used a chess engine before?

Really? Have you any other arguments to support this in addition to "Have you never used a chess engine before?" ?

Where are the missing moves? The missing 44 moves...

[hgm]

Printing a complete PV as supposed to badly clipping it is another feature that has a huge impact on the usefulness of an engine. There do exist good solutions for that (e.g. the method used in Crafty). But of course they don't provide any Elo. The overhead to bother with saving the PV might even cost 0.1 Elo. So forget about it...

[xmas79]

Printing a complete PV as supposed to badly clipping it is another feature that has a huge impact on the usefulness of an engine. There do exist good solutions for that (e.g. the method used in Crafty). But of course they don't provide any Elo. The overhead to bother with saving the PV might even cost 0.1 Elo. So forget about it...

I have no problem with that at all, I know it is ok for a developer, but it is "crystal clear" that it is NOT for a user. And that should also be crystal clear to people that take up the cudgels on stockfish. I do not like people who start "offending" other people when they have no arguments, because there are absolutely none in this particular situation...