Stockfish depth vs. others; challenge

[lkaufman]

As anyone who follows TCEC surely knows. Stockfish almost invariably outsearches all other engines by many plies, sometimes in endgames even reaching twice the depth of the opposing engine! Since Stockfish is open source we know that this is not due to deliberate subterfuge or strange counting of plies.

Let us limit the discussion to comparing Stockfish with Ivanhoe or any of the other open-source Ippolit derivates. Stockfish achieves greater depths than Ivanhoe partly due to doing less extensions, and partly to more aggressive LMR. Movecount pruning, score-based pruning, and the like are not more aggressive (in terms of adding plies) in Stockfish than in Ivanhoe, as best as I can determine. But based on my experience with Komodo, no amount of LMR and cutting out extensions can explain the huge gap in search depth without resorting to silly things that would cripple the engine. But of course there must be some explanation.

So I propose this challenge: Starting with Ivanhoe or any other open-source Ippo derivative, modify LMR (without reducing the usual suspects like hash move, killers, captures, and checks), extensions (but preserve at least safe check extension), and anything else in search with the goal of creating a version that matches Stockfish's depth on average, especially in endgames at fairly slow time controls, without weakening the engine too drastically, let's say by fifty elo. I think you will find this quite hard to do.

Obviously, Stockfish pays for this great depth, else it would be hundreds of elo points above Houdini and Komodo, but at least we would know how these amazing depths are even possible in a strong program.

[jdart]

One of the things Stockfish does differently from other programs is aggressive LMR combined with quite a lot of logic that excludes certain moves from reduction. I think you probably have to have both these components. Deep LMR alone will weaken the program unless you are smart about it.

On the other hand, high depths in the endgame are not unusual nowadays. The latest Crafty release increased the max ply to 128 (64 was not enough).

My own engine Arasan despite being well below Stockfish in rating can still get above 40 ply in endgames.

--Jon

[lkaufman]

One of the things Stockfish does differently from other programs is aggressive LMR combined with quite a lot of logic that excludes certain moves from reduction. I think you probably have to have both these components. Deep LMR alone will weaken the program unless you are smart about it.

On the other hand, high depths in the endgame are not unusual nowadays. The latest Crafty release increased the max ply to 128 (64 was not enough).

My own engine Arasan despite being well below Stockfish in rating can still get above 40 ply in endgames.

--Jon

Well, first of all I don't think that Stockfish excludes many more moves from reduction than other top engines, but anyway if they do this would not account for the high depths reached. In other words, I'll bet that if you just copy the Stockfish LMR formula into Ivanhoe with no other changes Ivanhoe still won't approach Stockfish depths. Maybe somebody can try this?
If Arasan typical outsearches Ivanhoe (or any other Ippo cousin) in the endgame by a substantial number of plies, to what do you attribute this?

[Ryan Benitez]

Try doing almost no extensions in non PV nodes and few in PV. It is consistent with the LMR spirit to have an avoidReduction case in place of most extensions anyway.

[lkaufman]

Try doing almost no extensions in non PV nodes and few in PV. It is consistent with the LMR spirit to have an avoidReduction case in place of most extensions anyway.

Well, I believe that Stockfish still does both (safe) check extensions and Singular extensions in non-PV, which isn't so drastically different from most top engines. It does explain some of the difference, but not very much I think.

[mar]

non-pv nodes aren't junk, they help to find better moves

[Eelco de Groot]

But based on my experience with Komodo, no amount of LMR and cutting out extensions can explain the huge gap in search depth without resorting to silly things that would cripple the engine. But of course there must be some explanation.

Hello Larry! I think this observation is, likely, mostly an illusion. The number of iterations by itself says very little about the actual depth as you know well; it says nothing about the reductions taking place in every new generation of child nodes. You can have very strong reductions, in practice, even if the actual depth increase from a new iteration is zero, if you have a hash hit in most nodes, you still achieve at least the same depth in a node as in the previous iteration, but the move sorting will hopefully have improved a bit. Even if you don't deepen the search much in the following iteration; the better move sorting by iterative deepening is what you are after. If increasing the number of iterations in this way hurts the engine, it probably means the engine has some problem retracing its path from memory from the previous iteration. The risk for this is greatest in the PV I think. Any number of re-searches in the PV for instance will reduce the efficiency of retracing. Re-searches in the PV are not bad per se, but the effect of doing the same thing by increasing the iteration counter is that re-searches take place across the whole search tree, not simply the PV alone.

Regards, Eelco

So I propose this challenge: Starting with Ivanhoe or any other open-source Ippo derivative, modify LMR (without reducing the usual suspects like hash move, killers, captures, and checks), extensions (but preserve at least safe check extension), and anything else in search with the goal of creating a version that matches Stockfish's depth on average, especially in endgames at fairly slow time controls, without weakening the engine too drastically, let's say by fifty elo. I think you will find this quite hard to do.

Obviously, Stockfish pays for this great depth, else it would be hundreds of elo points above Houdini and Komodo, but at least we would know how these amazing depths are even possible in a strong program.

[Uri Blass]

Eelco,There is no illusion and it is clear that Larry means the number that stockfish prints as depth by using the word depth.

For some reason this number is bigger than the number for other programs and the question is what is the reason.


I also think that stockfish see deeper because in many games stockfish see the win faster than the opponent(including games that stockfish lose)
and often the scores of stockfish is more than twice bigger than the scores of komodo and I think that it cannot be explained only by difference in the evaluation function.

[Eelco de Groot]

Eelco,There is no illusion and it is clear that Larry means the number that stockfish prints as depth by using the word depth.

For some reason this number is bigger than the number for other programs and the question is what is the reason.


I also think that stockfish see deeper because in many games stockfish see the win faster than the opponent(including games that stockfish lose)
and often the scores of stockfish is more than twice bigger than the scores of komodo and I think that it cannot be explained only by difference in the evaluation function.

The illusion is that Larry thinks that you could not achieve the same number of iterations, in another program, without losing strength and that you have, or Larry has, to introduce silly reductions that make another program, like Komodo, lose strength. Then I think you are looking in your program in the wrong place. It are not the increased reductions by itself that would make the program weaker. Of course, if your move sorting improves, in any way, that also helps finishing an iteration faster and that will increase the search efficiency, increase the average real search depths, but that is not really the point here I think.

Eelco

[Uri Blass]

Eelco,There is no illusion and it is clear that Larry means the number that stockfish prints as depth by using the word depth.

For some reason this number is bigger than the number for other programs and the question is what is the reason.


I also think that stockfish see deeper because in many games stockfish see the win faster than the opponent(including games that stockfish lose)
and often the scores of stockfish is more than twice bigger than the scores of komodo and I think that it cannot be explained only by difference in the evaluation function.

The illusion is that Larry thinks that you could not achieve the same number of iterations, in another program, without losing strength and that you have, or Larry has, to introduce silly reductions that make another program, like Komodo, lose strength. Then I think you are looking in your program in the wrong place. It are not the increased reductions by itself that would make the program weaker. Of course, if your move sorting improves, in any way, that also helps finishing an iteration faster and that will increase the search efficiency, increase the average real search depths, but that is not really the point here I think.

Eelco

The point is that larry does not understand how to achieve the same depth in Ivanhoe without losing strength(depth of course means number of iteration in all of this post).

The point is not that it is impossible but that if people understand why stockfish achieve higher depths than other programs then it may help to improve other programs.