Tuning

[Lyudmil Tsvetkov]

And the same goes true about implementation of closed positions.

[d]6k1/8/4p3/3pP3/3P4/8/8/6K1 w - - 0 1
With this pawn constellation, give penalty for the short-castled black king. Well, it makes some sense, as the chain is pointed towards the enemy black king, however what if:

1. there is a black pawn on f5. In this case the position on the king side is already sufficiently closed for the black king to bother
2. black has good counterplay on the queen side, in this case the pawn structure is already not that efficient

The conclusion would be that this is not a real closed position that white could frequently enough use to its advantage.

[d]6k1/5p2/4p3/3pP3/2pP4/2P5/8/6K1 w - - 0 1
Adding one more file of blocked pawns along the c file, plus an f7 pawn. Well, this is already a real closed position. The advantages of this implementation over the previous one:

1. the f7 black pawn ensures that black still has not played f5, this pawn is vulnerable, and white could use this situation to organize an efficient king side attack

2. the further pair of blocked pawns on the c file ensures that, even if black has some counterplay on the queen side, it will develop much less rapidly, as a larger portion of the board is blocked. This gives white sufficient time to organize king side attack.

Well, this is really a closed position, a closed position engines should learn to play more satisfactorily.

[d]6k1/6p1/5p2/4pP2/3pP3/3P4/8/6K1 w - - 0 1
And another closed position with the same characteristics, only shifted one file to the right. Those, with the 2 mirror black cases, are the 2 main closed position types. It is enough to code just those 2 types so that an engine has sufficient knowledge of closed positions. The second posted position from black's point of view is actually the KID itself.

[d]6k1/8/3p4/3Pp3/4Pp2/5P2/6P1/6K1 w - - 0 1

But of course, the KID is everywhere, you can enter KID structures from a very wide range of positions. And also, there are many other features and eval terms associated with KID structures, but the above simple rule is the most necessary and basic rule defining closed game with attacking opportunities for one of the sides. (to be distinguished from an entirely blocked position, or a fortress, which only ensures a draw, but gives no real winning chances for one of the sides, as in this case)

[mcostalba]

the correct definition would be that a position is closed when on at least 3 of the central c,d,e or f files there are blocked pawns. I.e., a structure like white pawns on c3,d4,e5, black pawns on c4,d5,e6. The at least 3 central blocked pawns rule is the minimal requirement for a closed position, but piece values are even better adjusted with more blocked pawns.

I can write a closed position detector to be used by people to write tests on it, but I need a correct definition. From what you write pawns blocked is not enough, it seems they have to be part of a chain, is it correct?

I mean that a structure like white pawns on c2, d6, e3 and black pawns on c3,d7,e4 respect the "blocked" constrain, but I don't think is a closed position.

[mcostalba]

I can write a closed position detector to be used by people to write tests on it

Ok I have pushed a branch called closed_positions:

https://github.com/mcostalba/Stockfish/ ... d_position

with the code that implements your definition. Now I am able to detect positions like this one:

[d]r1q2rk1/2p3pp/2Pp4/p2P1p2/Q3p3/4P3/PP1R2PP/1K5R b - - 0 23

[Lyudmil Tsvetkov]

the correct definition would be that a position is closed when on at least 3 of the central c,d,e or f files there are blocked pawns. I.e., a structure like white pawns on c3,d4,e5, black pawns on c4,d5,e6. The at least 3 central blocked pawns rule is the minimal requirement for a closed position, but piece values are even better adjusted with more blocked pawns.

I can write a closed position detector to be used by people to write tests on it, but I need a correct definition. From what you write pawns blocked is not enough, it seems they have to be part of a chain, is it correct?

I mean that a structure like white pawns on c2, d6, e3 and black pawns on c3,d7,e4 respect the "blocked" constrain, but I don't think is a closed position.

Marco, such blocked pawns as you describe never happen in actual games - the probability is less than 1 in a million. So that, yes, closed positions consist of blocked chain pawns, except for the base pawn of the chain, if you do not consider it as connected.

There could be an even better definition that would take into account other files also, but this is the most simple minimal requirement.

I think that, in terms of elo, teaching SF to understand better KID structures of the type d3,e4,f5 white pawns and black pawns on d4,e5,f6,g6, with the g7 black pawn being an important feature, is the most practical side to it.

If SF would be able to understand and play it however, I do not know.

[mcostalba]

Marco, such blocked pawns as you describe never happen in actual games

See my previous post, position I show is one that I have actually found testing this new code.

[Lyudmil Tsvetkov]

I hope very much you try this, Arjun.

I know what people would say about it:

- it is unorthogonal
- it is wasting resources
- let us move on

But, please take note that specific suggestions are very much orthogonal, non-specific suggestions are those that lack any orthogonality. Actually, the more specific a suggestion is, the more it obeys the orthogonality rule. Specific suggestions avoid randomness, non-specific suggestions, no matter how well tuned, fall easy prey to randomness.

While you are there Arjun, with very nice closed ideas, like giving bonus for storming pawns and some piece types with closed positions, why do not you apply a correct definition for a closed position? What concerns adjusting knights and bishop values in closed positions, the correct definition would be that a position is closed when on at least 3 of the central c,d,e or f files there are blocked pawns. I.e., a structure like white pawns on c3,d4,e5, black pawns on c4,d5,e6. The at least 3 central blocked pawns rule is the minimal requirement for a closed position, but piece values are even better adjusted with more blocked pawns.

A similar rule could be used concerning storms, but there the problem might be bigger, as this would depend also on other factors.

I suggested a while ago a very nice closed idea, improving a bit on Joona Kiiski's closed patch. Very simple: white pawns on c3,d4,e5, black pawns on c4,d5,e6, f7, and black king castled short - then give some 20cps penalty to the short castled black king. Why do not you try this Arjun, just once precisely the way I have suggested. Just this one very simple idea. It will help SF learn for the first time what a closed position really is and how to behave in such positions. SF needs this code, it simply needs it.

That is way too specific, IMHO. A human can dissect the pattern, but the computer cannot deal so easily with the exceptions. In other words, the whole concept is fine (do not castle towards position with negative pawn chains), but it is not simple at all.

Miguel

[d]6k1/8/3p4/3Pp3/4Pp2/5P2/6P1/6K1 w - - 0 1

Is not this the KID, Miguel?
What more or less do you need to define better the KID, so that it is easy for engines to implement? You can not do without the backward g2 pawn, for sure.

Otherwise, I agree, there are probably too many particular evaluation and search features in engines that prevent them to play the KID and related structures in a satisfactory fashion. I can think of, at least, a whole array of eval terms that have a bearing on closed play, like adjusting N and B values, different use of storming pawns, need to find moves that gradually increase the pressure instead of looking for the best attacking moves, i.e. need for a positional manoeuvering based on gradual improvement of psqt, mobility, etc. So that, when you consider a position closed, you should take into account, at the same time, many other eval and search features. I do not understand search very much, but quite probably, the way it is written now, the general pattern, would suit primarily play in open positions.

It takes much for engines to be able to learn to play better those position types, it is true, but it should not be impossible.

[mcostalba]

Ok I have pushed a branch called closed_positions:

https://github.com/mcostalba/Stockfish/ ... d_position

Following Lyudmil indications I have refined the detector requiring that pawns are on a diagonal chain. Now I don't detect anymore positions as the previous one posted, but only these kind of positions:

[d]rn1qk2r/pp3ppp/2p2n2/2Pp1b2/3Pp3/P1P1P3/3B1PPP/R2QKBNR w KQkq - 1 9

[Lyudmil Tsvetkov]

I can write a closed position detector to be used by people to write tests on it

Ok I have pushed a branch called closed_positions:

https://github.com/mcostalba/Stockfish/ ... d_position

with the code that implements your definition. Now I am able to detect positions like this one:

[d]r1q2rk1/2p3pp/2Pp4/p2P1p2/Q3p3/4P3/PP1R2PP/1K5R b - - 0 23

It is still closed, but would have practical repercussions only on piece values and possible detection of more blocked pawns and fortress. There are degrees of closeness, and the more blocked pawns you find, for example 5 or 6 also on non-central files, the more closed a position is. But I understand that might be too much effort for an engine to consider all blocked files, so the central definition should be good enough.

In terms of practical play, however, what matters in 80% and more of middlegame cases, is to have understanding of the KID-like structures for black and white, of the type:

1. white pawns on c3,d4,e5, black pawns on c4,d5,e6,f7, black king castled short, then give 20cps or so penalty to the black king
2. white pawns on d3,e4,f5, black pawns on d4,e5,f6, g7, black king castled short, then give penalty to the black king

with the respective mirrored positions from black's point of view.

Almost everything of practical importance in closed middlegame positions is concentrated into the above 2 main position types. So that, I think the basic definition/detector should be the above 2 main position types.

[Lyudmil Tsvetkov]

Ok I have pushed a branch called closed_positions:

https://github.com/mcostalba/Stockfish/ ... d_position

Following Lyudmil indications I have refined the detector requiring that pawns are on a diagonal chain. Now I don't detect anymore positions as the previous one posted, but only these kind of positions:

[d]rn1qk2r/pp3ppp/2p2n2/2Pp1b2/3Pp3/P1P1P3/3B1PPP/R2QKBNR w KQkq - 1 9

Perfect, that is precisely a KID-type closed position.

Taking into account that an f2 backward pawn is also present, this is precisely the type of position for which the white king should receive some nice penalty, as the general pattern favours black very much, especially what concerns possible king attacks.

I see SF starts understanding the KID now. Carl will be very happy.

[arjuntemurnikar]

I am certain Arjun Temurnikar reads this forum, as he often replies here.

Many thanks to Arjun for trying to implement another idea of mine. I have never been so famous in my life, so I feel a bit uneasy.

Arjun, did you really try to implement bonus for minor piece behind chain pawn on top of the already existing bonus for minor piece behind pawn? I think you realise at first glance adding another minor behind pawn is very redundant. SF tuned for a long time this term in the summer and the value they finally got seems to be an optimum, attempts to increase it further failed. So that adding still further bonus is obviously bad.

I never said minor behind chain pawn will perform better in SF than minor behind pawn, but I think it will, as this is what chess knowledge suggests. So that my suggestion to Arjun would be to retest the patch, removing beforehand the existing bonus for minor behind pawn. As this will hugely simplify the eval, as instead of checking for all pawns, now the engine will check only for chain pawns, you could test it with SPRT -4;0.

OK, I give up on SPRT benefits. Just retest the patch of minor behind chain pawn, when the existing minor behind pawn bonus is previously removed. So that you will test SF performance of minor behind chain pawn instead of minor behind any pawn. I think it is better to specify different values for B and N, as well as for ranks, but you can very well simply leave absolutely the same values as for minor behind pawn, I do not care.

So please, Arjun, thank you very much for testing, but, if you would like to see if real knowledge actually works, please retest the patch as I suggested.

Many thanks in advance for your understanding.

Wow, you need not give me so much praise. The real praise should go to you as you have consistently provided valuable insight into SF improvement. It is always great to have someone who really understands chess very well and is so enthusiastic about SF and chess programming, despite not really being able to contribute directly towards it. I find your posts most intriguing, though I haven't read them all as that would probably take a long long time!

I actually do come on talkchess sometimes, but I not as many times as you think. I just casually stumbled across your minor behind pawn idea, and as it was pretty easy to implement, I thought I would give it a try! Pardon me if I misunderstood your idea. I will send another patch with the specific changes you mentioned! I would say though that I am not too sure it will pass because as you said and know, SF eval has been tuned to death, and it will be difficult to replace something like that with a better parameter without spending some time tuning it. If I have time I will try messing around with CLOP for it in the future.

I hope very much you try this, Arjun.

I know what people would say about it:

- it is unorthogonal
- it is wasting resources
- let us move on

But, please take note that specific suggestions are very much orthogonal, non-specific suggestions are those that lack any orthogonality. Actually, the more specific a suggestion is, the more it obeys the orthogonality rule. Specific suggestions avoid randomness, non-specific suggestions, no matter how well tuned, fall easy prey to randomness.

While you are there Arjun, with very nice closed ideas, like giving bonus for storming pawns and some piece types with closed positions, why do not you apply a correct definition for a closed position? What concerns adjusting knights and bishop values in closed positions, the correct definition would be that a position is closed when on at least 3 of the central c,d,e or f files there are blocked pawns. I.e., a structure like white pawns on c3,d4,e5, black pawns on c4,d5,e6. The at least 3 central blocked pawns rule is the minimal requirement for a closed position, but piece values are even better adjusted with more blocked pawns.

A similar rule could be used concerning storms, but there the problem might be bigger, as this would depend also on other factors.

I suggested a while ago a very nice closed idea, improving a bit on Joona Kiiski's closed patch. Very simple: white pawns on c3,d4,e5, black pawns on c4,d5,e6, f7, and black king castled short - then give some 20cps penalty to the short castled black king. Why do not you try this Arjun, just once precisely the way I have suggested. Just this one very simple idea. It will help SF learn for the first time what a closed position really is and how to behave in such positions. SF needs this code, it simply needs it.

As I told you, the pawn on f7 is very much necessary for the closed trick to work, you can not do without it.

Please Arjun, listen just once to what real chess knowledge, as specific as possible, tells us.

As for my closed position patch, it was a very experimental. I saw joona's earlier patch (which was based on your idea of 3/4 central pawns c,d,e,f) and a few other related patches but they hadn't worked so well, so I thought I will try something different. My idea was simply that I defined a closed position more generally as a function of the number of blocked pawns in a position (giving 2x more value to central pawns on d, e files). I also later tried using an array to give fixed (predetermined "tuned") bonuses and penalties to knights and bishops based on the number of blocked pawns in the position (this time not giving special preference to central pawns). This was all just experimental and I was pretty sure it wouldn't work because:

1. Such a feature in SF would have to be tuned to a decent amount of precision in order to show any elo gain.
2. SF eval is already tuned to death, and additional bonuses/penalties might actually be redundant. Somehow the combination of various pawn structure and minor piece elements in the evaluation function are already taking care of closed positions. SF is not that bad in closed positions, really!

Anyway, I saw marco created a new branch based on your advice. It is very interesting. Hopefully we will see more people experiment with different approaches to closed positions in SF.

To conclude, I think that the approach that will finally work will be one which will remove some code from SF before adding specific code regarding closed positions. Somehow I feel there is already something there that is contradicting any attempts at trying to find a good closed position implementation. Aaah! That's just my naive guess. There is still lot of work to do...

Thanks for your great insight. I hope it never stops coming!