Elo points gain from doubling time

[lkaufman]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

Mystery solved

I wonder whether Stockfish has a similar spike? SF starts doing singular extensions at different depths for PV and non-PV nodes:

Code: Select all

  // Minimum depth for use of singular extension
  const Depth SingularExtensionDepth[] = { 8 * ONE_PLY, 6 * ONE_PLY };

  /* ... */

    singularExtensionNode =   !RootNode
                           && !SpNode
                           && depth >= SingularExtensionDepth[PvNode]
                           && ttMove != MOVE_NONE
                           && !excludedMove // Recursive singular search is not allowed
                           && (tte->type() & VALUE_TYPE_LOWER)
                           && tte->depth() >= depth - 3 * ONE_PLY; 

Okay, I ran the test for SF 2.3. Here are the results (3000 games minimum for each entry):
Elo gap
6 ply vs. 5: 175.7
7 ply vs. 6: 205.2
8 ply vs. 7: 175.4
9 ply vs. 8: 148.2
10 ply vs 9: 128.5

So the expected jump at 7 ply did materialize, but not the one at 9 ply. Based on this data, it appears that the SF Singular Extension works well enough at PV nodes but is pretty much worthless at non-PV nodes. I realize that the details of SF non-PV singular extension are rather different from Critter and from Ivanhoe (and therefore presumably also at least early Houdini versions), but I would still expect it to have a fairly significant effect. Does anyone have any idea from looking at the SF code why Singular extension at nonPV nodes would apparently do next to nothing?

[michiguel]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

Mystery solved

I wonder whether Stockfish has a similar spike? SF starts doing singular extensions at different depths for PV and non-PV nodes:

Code: Select all

  // Minimum depth for use of singular extension
  const Depth SingularExtensionDepth[] = { 8 * ONE_PLY, 6 * ONE_PLY };

  /* ... */

    singularExtensionNode =   !RootNode
                           && !SpNode
                           && depth >= SingularExtensionDepth[PvNode]
                           && ttMove != MOVE_NONE
                           && !excludedMove // Recursive singular search is not allowed
                           && (tte->type() & VALUE_TYPE_LOWER)
                           && tte->depth() >= depth - 3 * ONE_PLY; 

Okay, I ran the test for SF 2.3. Here are the results (3000 games minimum for each entry):
Elo gap
6 ply vs. 5: 175.7
7 ply vs. 6: 205.2
8 ply vs. 7: 175.4
9 ply vs. 8: 148.2
10 ply vs 9: 128.5

So the expected jump at 7 ply did materialize, but not the one at 9 ply. Based on this data, it appears that the SF Singular Extension works well enough at PV nodes but is pretty much worthless at non-PV nodes. I realize that the details of SF non-PV singular extension are rather different from Critter and from Ivanhoe (and therefore presumably also at least early Houdini versions), but I would still expect it to have a fairly significant effect. Does anyone have any idea from looking at the SF code why Singular extension at nonPV nodes would apparently do next to nothing?

Not easy to know if things are "working" or not because this is at fixed depth.

Miguel

[lkaufman]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

Mystery solved

I wonder whether Stockfish has a similar spike? SF starts doing singular extensions at different depths for PV and non-PV nodes:

Code: Select all

  // Minimum depth for use of singular extension
  const Depth SingularExtensionDepth[] = { 8 * ONE_PLY, 6 * ONE_PLY };

  /* ... */

    singularExtensionNode =   !RootNode
                           && !SpNode
                           && depth >= SingularExtensionDepth[PvNode]
                           && ttMove != MOVE_NONE
                           && !excludedMove // Recursive singular search is not allowed
                           && (tte->type() & VALUE_TYPE_LOWER)
                           && tte->depth() >= depth - 3 * ONE_PLY; 

Okay, I ran the test for SF 2.3. Here are the results (3000 games minimum for each entry):
Elo gap
6 ply vs. 5: 175.7
7 ply vs. 6: 205.2
8 ply vs. 7: 175.4
9 ply vs. 8: 148.2
10 ply vs 9: 128.5

So the expected jump at 7 ply did materialize, but not the one at 9 ply. Based on this data, it appears that the SF Singular Extension works well enough at PV nodes but is pretty much worthless at non-PV nodes. I realize that the details of SF non-PV singular extension are rather different from Critter and from Ivanhoe (and therefore presumably also at least early Houdini versions), but I would still expect it to have a fairly significant effect. Does anyone have any idea from looking at the SF code why Singular extension at nonPV nodes would apparently do next to nothing?

Not easy to know if things are "working" or not because this is at fixed depth.

Miguel

I find that comment rather strange here, because any sensible extension that kicks in at a certain depth should definitely help the elo at that depth. Of course it could help the elo and still be a bad idea if it slowed the search too much, but the point here is that I'm not even detecting any help to the Elo, it appears to drop off just as would be expected without the extension. We know it dramatically helps the elo at 9 ply in Critter and Houdini, so why no detectible help with SF?
Testing at a time limit would not isolate the effect of the singular extension algorithm the way fixed depth does.

[michiguel]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

Mystery solved

I wonder whether Stockfish has a similar spike? SF starts doing singular extensions at different depths for PV and non-PV nodes:

Code: Select all

  // Minimum depth for use of singular extension
  const Depth SingularExtensionDepth[] = { 8 * ONE_PLY, 6 * ONE_PLY };

  /* ... */

    singularExtensionNode =   !RootNode
                           && !SpNode
                           && depth >= SingularExtensionDepth[PvNode]
                           && ttMove != MOVE_NONE
                           && !excludedMove // Recursive singular search is not allowed
                           && (tte->type() & VALUE_TYPE_LOWER)
                           && tte->depth() >= depth - 3 * ONE_PLY; 

Okay, I ran the test for SF 2.3. Here are the results (3000 games minimum for each entry):
Elo gap
6 ply vs. 5: 175.7
7 ply vs. 6: 205.2
8 ply vs. 7: 175.4
9 ply vs. 8: 148.2
10 ply vs 9: 128.5

So the expected jump at 7 ply did materialize, but not the one at 9 ply. Based on this data, it appears that the SF Singular Extension works well enough at PV nodes but is pretty much worthless at non-PV nodes. I realize that the details of SF non-PV singular extension are rather different from Critter and from Ivanhoe (and therefore presumably also at least early Houdini versions), but I would still expect it to have a fairly significant effect. Does anyone have any idea from looking at the SF code why Singular extension at nonPV nodes would apparently do next to nothing?

Not easy to know if things are "working" or not because this is at fixed depth.

Miguel

I find that comment rather strange here, because any sensible extension that kicks in at a certain depth should definitely help the elo at that depth. Of course it could help the elo and still be a bad idea if it slowed the search too much, but the point here is that I'm not even detecting any help to the Elo, it appears to drop off just as would be expected without the extension. We know it dramatically helps the elo at 9 ply in Critter and Houdini, so why no detectible help with SF?

No, with the data shown here you don't even know if it helps Critter or Houdini at depth = 9! It may even hurt.

You are testing at fixed depth, so you have no way to know if that helps in a game. Any extension, even if it hurts, will look good at fixed depth just because you search more nodes. As an example, let's say an extension at fixed depth increases 10 elo (depth 10 vs depth 10+extension). If the extension ends up doubling the number of nodes needed to search the same depth, in a real game, the extension will hurt badly rather than helping with 10 elo.

Let's do the opposite (reductions). Assume nullmove is triggered at depth 9. We know that nullmove is good, however, in the same experiment you run, you will see going from depth 8 to 9 a ditch, rather than a bump.

Miguel

Testing at a time limit would not isolate the effect of the singular extension algorithm the way fixed depth does.

[lkaufman]

I did a similar partial study for Critter 1.2 and 1.6, to see if the pattern is much different.

As far as I know, Critter 1.2 does not have any search feature other than Singular Extension that kicks in at 9 ply, so this would appear to fully account for the observed jump in Critter

Easy to verify. Retry the 8 vs 9 ply match with singular extensions disabled and compare the results.

Okay, I did it. This time I got 157.2 elo (after 7k games), a dramatic reduction from 236.9. As a control, I reran 7 vs 8 ply, and got 180.0, up a bit from 173.6. I also reran 10 vs 9 ply (which also benefits from SE), and got 138.9 elo, down sharply from 175.2.

Conclusion: Singular Extension accounts for all the anomalies observed with the Critter matches. Without it the elo gain decays as it should from 180.0 to 157.2 to 138.9 at these depths. So there is no reason to think that this would not also fully explain the spike in Houdini 2 at depth 9, since the behavior was so similar.

Mystery solved

I wonder whether Stockfish has a similar spike? SF starts doing singular extensions at different depths for PV and non-PV nodes:

Code: Select all

  // Minimum depth for use of singular extension
  const Depth SingularExtensionDepth[] = { 8 * ONE_PLY, 6 * ONE_PLY };

  /* ... */

    singularExtensionNode =   !RootNode
                           && !SpNode
                           && depth >= SingularExtensionDepth[PvNode]
                           && ttMove != MOVE_NONE
                           && !excludedMove // Recursive singular search is not allowed
                           && (tte->type() & VALUE_TYPE_LOWER)
                           && tte->depth() >= depth - 3 * ONE_PLY; 

Okay, I ran the test for SF 2.3. Here are the results (3000 games minimum for each entry):
Elo gap
6 ply vs. 5: 175.7
7 ply vs. 6: 205.2
8 ply vs. 7: 175.4
9 ply vs. 8: 148.2
10 ply vs 9: 128.5

So the expected jump at 7 ply did materialize, but not the one at 9 ply. Based on this data, it appears that the SF Singular Extension works well enough at PV nodes but is pretty much worthless at non-PV nodes. I realize that the details of SF non-PV singular extension are rather different from Critter and from Ivanhoe (and therefore presumably also at least early Houdini versions), but I would still expect it to have a fairly significant effect. Does anyone have any idea from looking at the SF code why Singular extension at nonPV nodes would apparently do next to nothing?

Not easy to know if things are "working" or not because this is at fixed depth.

Miguel

I find that comment rather strange here, because any sensible extension that kicks in at a certain depth should definitely help the elo at that depth. Of course it could help the elo and still be a bad idea if it slowed the search too much, but the point here is that I'm not even detecting any help to the Elo, it appears to drop off just as would be expected without the extension. We know it dramatically helps the elo at 9 ply in Critter and Houdini, so why no detectible help with SF?

No, with the data shown here you don't even know if it helps Critter or Houdini at depth = 9! It may even hurt.

You are testing at fixed depth, so you have no way to know if that helps in a game. Any extension, even if it hurts, will look good at fixed depth just because you search more nodes. As an example, let's say an extension at fixed depth increases 10 elo (depth 10 vs depth 10+extension). If the extension ends up doubling the number of nodes needed to search the same depth, in a real game, the extension will hurt badly rather than helping with 10 elo.

Let's do the opposite (reductions). Assume nullmove is triggered at depth 9. We know that nullmove is good, however, in the same experiment you run, you will see going from depth 8 to 9 a ditch, rather than a bump.

Miguel

Testing at a time limit would not isolate the effect of the singular extension algorithm the way fixed depth does.

Of course, what you say is true, and obvious. This experiment does not confirm or refute a claim that singular extension helps Critter or Houdini. The point is, however, that the fixed depth test shows that the extension has a major effect on Critter and Houdini, but no apparent effect on Stockfish. If extensions don't even help at fixed depth, they can hardly help in timed play. The question is, why is Stockfish apparently not generating any significant number of nonPV singular extensions at depth nine, although the code as well as the experience with Critter and Houdini would predict that it would generate many such extensions? Perhaps we are missing something about the code in SF that would explain it.

[Eelco de Groot]

Of course, what you say is true, and obvious. This experiment does not confirm or refute a claim that singular extension helps Critter or Houdini. The point is, however, that the fixed depth test shows that the extension has a major effect on Critter and Houdini, but no apparent effect on Stockfish. If extensions don't even help at fixed depth, they can hardly help in timed play. The question is, why is Stockfish apparently not generating any significant number of nonPV singular extensions at depth nine, although the code as well as the experience with Critter and Houdini would predict that it would generate many such extensions? Perhaps we are missing something about the code in SF that would explain it.

I could not find an easy explanation for it just from the code. The better way to get an answer is to repeat the experiment you already did Larry, but now to switch off singular extensions in non PV nodes, or letting them kick in later and see if that makes especially the 7 ply search perform worse. Maybe the spike is really there but very low.

One thing that you could say about the code of Stockfish here is that several other things are happening, just below 6 ply. IID is switched on, verification search is switched on, threatdepth is already switched off at four plies so that would not have much negative impact. In the singular extension search, a six ply search is cut in half to only a three ply exclusion search. It is possible that Fails High so often that there are very few singular extensions awarded. I found that the most reasonable assumption to make. Because a three ply search with an optimistic eval like Stockfish is really likely to Fail High (over a rbeta margin of beta - 12 in this case).

I find it most interesting in fact from your experiment Larry, if the singular extension would really have this positive effect in PV nodes in Stockfish. I did not think it very important yet. I have no idea what actually the difference of SE in Stockfish is with singular extensions in Critter (or Houdini). But the way it works in PV nodes is as far as I can see very different from that in non PV nodes. In non PV nodes, the ttmove in a CUT node when singular extended has to fail low when the not extended search would not, for the singular extension to have an effect on effective search depth. Maybe this does not happen very often. In PV nodes however, the singular extension would also be applied in "ALL" nodes or what would be ALL nodes in a nullwindow search, and so if one move there is significantly better, the search will be extended. The search can be extended both in PV "CUT" and PV "ALL" nodes and the effect of singular extension is doubled, as far as I can see. That would be my explanation why singular extensions do have this for me a bit unexpected benefit in PV nodes (in Stockfish). But it is speculating without the numbers to back it up...

Regards, Eelco

[Eelco de Groot]

Correction: Where I am talking about six plies singular extension searches that should really be eight plies in non PV nodes, sorry, with an exclusion search of four plies in that eight plies nullwindow search for instance. But without the repeat experiment, it is also a possibility that the observed spike for Stockfish at 7 plies, is not really due to the PV node singular extensions but to the other changes in non PV search.

Regards, Eelco

[lkaufman]

Of course, what you say is true, and obvious. This experiment does not confirm or refute a claim that singular extension helps Critter or Houdini. The point is, however, that the fixed depth test shows that the extension has a major effect on Critter and Houdini, but no apparent effect on Stockfish. If extensions don't even help at fixed depth, they can hardly help in timed play. The question is, why is Stockfish apparently not generating any significant number of nonPV singular extensions at depth nine, although the code as well as the experience with Critter and Houdini would predict that it would generate many such extensions? Perhaps we are missing something about the code in SF that would explain it.

I could not find an easy explanation for it just from the code. The better way to get an answer is to repeat the experiment you already did Larry, but now to switch off singular extensions in non PV nodes, or letting them kick in later and see if that makes especially the 7 ply search perform worse. Maybe the spike is really there but very low.

One thing that you could say about the code of Stockfish here is that several other things are happening, just below 6 ply. IID is switched on, verification search is switched on, threatdepth is already switched off at four plies so that would not have much negative impact. In the singular extension search, a six ply search is cut in half to only a three ply exclusion search. It is possible that Fails High so often that there are very few singular extensions awarded. I found that the most reasonable assumption to make. Because a three ply search with an optimistic eval like Stockfish is really likely to Fail High (over a rbeta margin of beta - 12 in this case).

I find it most interesting in fact from your experiment Larry, if the singular extension would really have this positive effect in PV nodes in Stockfish. I did not think it very important yet. I have no idea what actually the difference of SE in Stockfish is with singular extensions in Critter (or Houdini). But the way it works in PV nodes is as far as I can see very different from that in non PV nodes. In non PV nodes, the ttmove in a CUT node when singular extended has to fail low when the not extended search would not, for the singular extension to have an effect on effective search depth. Maybe this does not happen very often. In PV nodes however, the singular extension would also be applied in "ALL" nodes or what would be ALL nodes in a nullwindow search, and so if one move there is significantly better, the search will be extended. The search can be extended both in PV "CUT" and PV "ALL" nodes and the effect of singular extension is doubled, as far as I can see. That would be my explanation why singular extensions do have this for me a bit unexpected benefit in PV nodes (in Stockfish). But it is speculating without the numbers to back it up...

Regards, Eelco

I ran SF normal against SF without SE at nonPV nodes. At nine ply the normal version won by ten elo (and was 3% slower). So the algorithm does work at nine ply, but it seems the ten elo was lost in the error margins of the previous test. At ten ply the gap grew to 25 elo (and 5% slower). So perhaps there is no real mystery, it's just that SE is much more important at PV nodes than at nonPV nodes, especially at the lower depths.

[Eelco de Groot]

Thanks for doing the repeat experiments Larry! Interesting data. You predicted that in PV nodes, the spike would be in the PV "CUT" node at seven plies and not already at 6 plies. I did try once to get singular extensions working in ALL nodes in a nullwindow search but failed. For the overall extending effect in PV nodes you would think it (extending ALL nodes) to be of some importantance, but it does not show a spike. And singular extensions in both Critter and Houdini seem to have more of an extending effect than in Stockfish. Not entirely unexpected but it is nice that you have actually measured it.

Eelco