Stockfish - single evasion extensions

[Ralph Stoesser]

Apropos ELO. How is it going in the actual SF branch. Do you still fight against ELO regression or have you found a way to increase ELO significantly over SF 1.7.1?

We have tried very hard to find the regression but without success, actually we are thinking to release the new version with only few features in although we have done a lot of cleanup and code changes.

So we don't expect big ELO gains this time, there are some tests ongoing right now so probably at the end of this week we will know a bit better how is with this version.

The main reason to release is to give people like you an up to date codebase to play with.

Nice!

I think the biggest potential lies in reductions, extensions a more accurate eval.

One example. Recently I have tried a tiny change in eval suggested by Ed Schroeder on his Rebel page:
Calculate RookOpenFileBonus and RookHalfOpenFileBonus dependent on the number of enemy pawns.

a) decrease basic bonuses slightly

Code: Select all

const Score RookOpenFileBonus = make_score(43-3, 43-3);
const Score RookHalfOpenFileBonus = make_score(19-3, 19-3);

b) calculate the bonuses depenent on number of enemy pawns

Code: Select all

if (ei.pi->file_is_half_open(Us, f))
{
    if (ei.pi->file_is_half_open(Them, f))
        ei.value += Sign[Us] * (RookOpenFileBonus + make_score(pos.piece_count(Them, PAWN), 0));
    else
        ei.value += Sign[Us] * (RookHalfOpenFileBonus + make_score(pos.piece_count(Them, PAWN)/5, 0));
}

Result compared with SF 1.7.1 is smaller search trees (tested with SF bench) and - Hell yeah! - an ELO increase (tested with 1 sec games).

[zamar]

One example. Recently I have tried a tiny change in eval suggested by Ed Schroeder on his Rebel page:
Calculate RookOpenFileBonus and RookHalfOpenFileBonus dependent on the number of enemy pawns.

a) decrease basic bonuses slightly

Code: Select all

const Score RookOpenFileBonus = make_score(43-3, 43-3);
const Score RookHalfOpenFileBonus = make_score(19-3, 19-3);

b) calculate the bonuses depenent on number of enemy pawns

Code: Select all

if (ei.pi->file_is_half_open(Us, f))
{
    if (ei.pi->file_is_half_open(Them, f))
        ei.value += Sign[Us] * (RookOpenFileBonus + make_score(pos.piece_count(Them, PAWN), 0));
    else
        ei.value += Sign[Us] * (RookHalfOpenFileBonus + make_score(pos.piece_count(Them, PAWN)/5, 0));
}

Result compared with SF 1.7.1 is smaller search trees (tested with SF bench) and - Hell yeah! - an ELO increase (tested with 1 sec games).

Are you absolutely sure about this? For me it looks impossible that change could have a measurable effect.

Let me explain: Rook on open-file usually gets bonus of 43 = 21cp. With your change the bonus can vary between 20cp, 21cp, 22cp or 23cp. Now remembering that GrainSize (the accurary of rounding) = 8 = 4cp, for me it looks like that magnitude of the changes you have made in your patch is so small that difference is doomed to be lost in rounding.

Then of course I could be wrong

Did I miss something?

[mcostalba]

Result compared with SF 1.7.1 is smaller search trees (tested with SF bench) and - Hell yeah! - an ELO increase (tested with 1 sec games).

Not to rain on your party, but for evaluation tweaks _normally_ the good patches increase the search tree.

This is for me not clear why, but I have seen this to happen many times. Instead search tweaks, like better move ordering or something like that shrink the search tree.

Care to test at 5" and report results (included number of games done) ?

[Ralph Stoesser]

Result compared with SF 1.7.1 is smaller search trees (tested with SF bench) and - Hell yeah! - an ELO increase (tested with 1 sec games).

Not to rain on your party, but for evaluation tweaks _normally_ the good patches increase the search tree.

This is for me not clear why, but I have seen this to happen many times. Instead search tweaks, like better move ordering or something like that shrink the search tree.

Care to test at 5" and report results (included number of games done) ?

This sounds strange. At least some eval tweaks should be able to improve the move ordering and shrink the search tree, because the move ordering is based on history and gain tables for non-capture moves and those table values finally come from static eval. Maybe it's simply wrong to count open files bonuses for rooks as constants?

OK I will test it over night, although I think it does not make much sense to test in a detailed manner against SF 1.7.1. It should rather be tested against the actual SF branch. Or in my case against my toyFish branch.

At the moment I'm fiddling with qsearch extensions based on accurate single evasion detection. The performance impact on pre-computing legal moves when in check seem to get outweigthed by the more accurate single evasion extensions, especially if I use those extensions also in qsearch. But SF 1.7.1 plays well. I think I have not more than +15 Elo all in all at the moment in fast games (15 sec).

[Ralph Stoesser]

One example. Recently I have tried a tiny change in eval suggested by Ed Schroeder on his Rebel page:
Calculate RookOpenFileBonus and RookHalfOpenFileBonus dependent on the number of enemy pawns.

a) decrease basic bonuses slightly

Code: Select all

const Score RookOpenFileBonus = make_score(43-3, 43-3);
const Score RookHalfOpenFileBonus = make_score(19-3, 19-3);

b) calculate the bonuses depenent on number of enemy pawns

Code: Select all

if (ei.pi->file_is_half_open(Us, f))
{
    if (ei.pi->file_is_half_open(Them, f))
        ei.value += Sign[Us] * (RookOpenFileBonus + make_score(pos.piece_count(Them, PAWN), 0));
    else
        ei.value += Sign[Us] * (RookHalfOpenFileBonus + make_score(pos.piece_count(Them, PAWN)/5, 0));
}

Result compared with SF 1.7.1 is smaller search trees (tested with SF bench) and - Hell yeah! - an ELO increase (tested with 1 sec games).

Are you absolutely sure about this? For me it looks impossible that change could have a measurable effect.

Let me explain: Rook on open-file usually gets bonus of 43 = 21cp. With your change the bonus can vary between 20cp, 21cp, 22cp or 23cp. Now remembering that GrainSize (the accurary of rounding) = 8 = 4cp, for me it looks like that magnitude of the changes you have made in your patch is so small that difference is doomed to be lost in rounding.

Then of course I could be wrong

Did I miss something?

Yes, you miss that every eval change, be it ever so small, has an effect. That's what the SF bench numbers are telling me. And that's why you automatically tune the eval parameters, isn't it?

It's well possible that the effect of such a change is not more measurable with longer time controls with a reasonable number of games. But what should we conclude if a tiny eval change works in very fast games, but seems to not have an effect at longer time controls, tested with a much fewer number of games? Discard the change? Why?

For sure the formula I tried is far from tuned. It was a random shot from 5 min lasting trials. But this random shot shows me that it is probably a good idea to include the number of enemy pawns in the formula.

Because of my very limited hardware resources, it's not possible for me to statistically prove that it works. I would assume it is an +2 ELO patch. The error margins are much too high, but here's my result from testing over night.

+830, -782, =1388 (+5 Elo)

Tested with an old Athlon dual core. 1 Thread. Linux 64bit. Cutechess-cli. GM2001.pgn, cut after 24 half moves. Every position repeated with switched colors.

[mcostalba]

This sounds strange.

Yes I agree on this.

I have not an explanation, I wonder it is because a good evaluation change makes some positions, previously discarded, to look interesting and so being search further....and perhaps after a bit more search that positions turn out to be _really_ interesting and so you get both a wider tree and also a stronger engine, because picks up good moves previously ignored.

For instance let's consider an engine that evaluates _only_ material gains, so that the evaluation is just the sum of piece values. This engine would be very weak, but I would guess the search tree to be very small too because when it finds a material gain it will discard almost everything else.

[Ralph Stoesser]

Marco,

since you will release the next version soon, here's another tiny code change proposal:

evaluate.cpp, line 988

Code: Select all

if (   pos.non_pawn_material(Them) <= KnightValueMidgame
                && pos.piece_count(Them, KNIGHT) <= 1)

If I do not miss something I would say it's not possible to have non_pawn_material <= KnightValueMidgame and at the same time to have more than one knight.

Therefore this condition

if (pos.non_pawn_material(Them) <= KnightValueMidgame)

should be enough.

[mcostalba]

Therefore this condition

if (pos.non_pawn_material(Them) <= KnightValueMidgame)

should be enough.

Thanks. Fix applied.

[Ralph Stoesser]

Instead your observation on ttove is correct, if we have only a TT move as possible move the extension never fires, this is the only case where we get things wrong, but it is so rare and very probably with almost zero impact on ELO that if the fix is not very small and nice (and I didn't found such a fix) better leave things as they are.

I'm not yet sure about which case does hurt more.

In case we have a TT move we can't fire the single evasion extension, but in some cases we extend anyway because of the TT move extension (Singular extension).

If we have no TT move, we get the single evasion extension wrong if number of pseudo legal evasions > number of legal evasions == 1.

I should do some measurement.

---

Regarding eval: I like your explanation about why good eval changes usually increase the search tree.

Now if we would remove one by one

1) material
2) mobility
3) king safety

from eval and would look at the tree sizes, they should be much smaller in all three cases compared to default eval tree size.

Should be easy to test.

[Uri Blass]

Apropos ELO. How is it going in the actual SF branch. Do you still fight against ELO regression or have you found a way to increase ELO significantly over SF 1.7.1?

We have tried very hard to find the regression but without success, actually we are thinking to release the new version with only few features in although we have done a lot of cleanup and code changes.

You can also release not only the final version.

I guess that you have something like sources of
Stockfish_01_05
Stockfish_09_05
Stockfish_18_05
Stockfish_27_05
Stockfish_05_06

When you know Stockfish_01_05 is stronger than Stockfish_05_06 but you do not know the source of the regression.

In this case you can release all these versions so people who are interested in the source of the regression may do more tests in order to find the source of the regression.

I think that a possible problem may be that the playing strength is not transitive with small differences.
In other words it may be possible that you get based on direct match that A is stronger than B and B is stronger than C when C is stronger than A in direct matches.
If it is the case then it may be interesting to have the source of A and B and C including this information.

Uri