pointy chains redux (part 1)

[Lyudmil Tsvetkov]

I think that in the second game Rodent_chain tried to form a chain when it had no time to do so. As for downloading, I think I have to fix a couple of things before release, so it will probably take some time.

OK, thanks Pawel, but please report back if a chain patch succeeds, yielding even 1 elo.

That would be tantamount to a breakthrough.

[PK]

So far it is a very slight regression, well within error bar (chain version scored 49,2%). I still hope that a gain can be produced out of this heuristic. Even if it is not, I will keep it as an option.

[carldaman]

I think you probably know more engines than I do, Lyudmil, but we'll help each other out with insider tips every now and then

After Black's g5, White's best plan is g3/Kh2 as you indicated, but Black does not have to trade on d4 at all - instead, playing Kh8/Rg8 and then double on the g-file, followed by gxf4 leaves Black with a solid position with no weaknesses, and a slight but lasting initiative. White will still have e3 and a4 to defend, and if both pairs of Rooks get traded on the g-file, a trade on d4 won't hurt Black. However, White should probably hold with best play.

That was long long time ago when I used to play every conceivable engine any author on this planet had created.

Now I mostly concentrate on the top and some interesting ones, though weaker.

So you are much more competent here.

However, concerning the position from the Andscacs game, I think the initiative is with white, as black needs time to develop and double along g, just as white, but white constantly attacks e6 with the knight, so this ties the black bishop with its defence, and at some point black will probably have to capture on d4, leaving white with a pointed chain.

a4 is not such a weakness in white's camp, as there is simply no way to attack it, on the other hand, it makes b7 backward.

Probably a draw with best play, but the initiative lies with white. I have played so many games along these lines...

I don't doubt you have experience with such positions, Lyudmil, but your comments appear based on a cursory look at the position I gave. I tend to take a deep look before I post something - I'm a stickler for that, ever since I've been playing cyborg correspondence chess.

From the earlier diagram, after 17. g3 Kh8 18. Kh2 Rg8 19. Bd2 Bd7 20. Rg1 Rg7, it is clear that Black gets to double on the g-file, and his Bd7 not only defends e6, which can be further overprotected, but also indirectly eyes a4, which will need a piece to guard it. It is Black that gets to decide when and where to exchange, and White that has the latent weaknesses.

Anyway, I'm a big fan of Stockfish, but I always keep a keen eye on improving second/third tier engines. What I like to see is when an engine gains significant Elo, but its play is visibly enhanced as well. There's a lot of potential for even greater things among such engines.

I hope to post more on the pointy chain later tonight.

[Lyudmil Tsvetkov]

So far it is a very slight regression, well within error bar (chain version scored 49,2%). I still hope that a gain can be produced out of this heuristic. Even if it is not, I will keep it as an option.

Great.

Did you try:

- smaller/larger bonus
- not counting where the kings are to save computations
- only mg bonus

Do not be angry at me, Pawel, but we all expect (at least me and Carl) to see at last even a single engine that knows even 5% of the rule.

My experience is, that when an engine changes for the good some of its terms, and that brings 2 elo, it performs better in some 10-20% of all cases related to this factor, and in the rest of cases the so called improvement usually does not help.

So it seems to be a matter of statistics with abundant room for improvement.

Thumbs up.

[carldaman]

Extending the chain, part III

In Part 3, I'd like to return to the basics of what makes a Kingside-directed pointed chain effective. Such chains don't drop out of the sky, but are formed, most of the time, by extending the chain - pushing a pawn to the foremost position in the chain, that is, forming its point or spearhead.

Let's look at the first diagram, where Black has the option and opportunity to correctly extend the chain:


[D]r1bq1rk1/pppnn1bp/3p2p1/3Ppp2/2P1P3/2N1BP2/PP2B1PP/R2QNRK1 b - - 0 11

This is a typical position from the King's Indian Defense (KID), where every human player worth his or her salt will push the pawn to f4 on the next move. Most engines, even the strongest ones, will hesitate here, and defer the pawn advance, until some unnecessary further play on the Queenside, such as b6 and a5/Bb7. This happens despite the fact the f5-f4 comes favorably with a tempo in this main variation of the Mar Del Plata or Classical KID. Sometimes, f5-f4 never arrives at all, and we may even see the lousy f5xe4 pawn trade that only helps White, unfortunately.

This inexplicable behavior suggests there is something really wrong and rotten about the eval factors of most engines, that rears its ugly head in such situations. As mentioned elsewhere, even if f5-f4 were to be played, the typical eval score is unfavorable to Black by some 60-70 cps, which is grossly lopsided, even to the point of deterring some testers from using the Classical KID in their tests, thus further masking the serious hole in the eval (!)

Now let's assume f5-f4 was played, and we now get this commonly seen mainline position (in human games, at least):

[D]r1bq1rk1/pppnn1bp/3p4/3Pp1p1/2P1Pp2/2N2P2/PP2BBPP/R2QNRK1 w - - 0 13

It would behoove us to have a very close look at the features of this position, as all the elements of a healthy attacking chain are present.

-Black has more space on the Kingside
-a further Kingside expansion (pawn storm) is readily achievable by Black
-the Black King is safe even if its pawn shelter storms forward
-most of the Black pieces can quickly reach more advanced positions on the Kingside and join in the attack
-the White King is castled Kingside, right into the teeth of the attack, in the direction the Black pawn chain is pointing [very important]
-the White King has limited mobility and cannot easily run away due to being hemmed in by his own pieces and pawns [this is a critical aspect, and I refer to it as congestion]
-the White pawn chain (g2-f3-e4-d5) is aimed at the Queenside, and together with the attacking Black pawn chain (c7-d6-e5-f4) forms an impenetrable barrier that the White pieces cannot cross in order to counterattack or to come to the aid of the White King [an essential feature again]

Most people, programmers included, will readily perceive the first three salient features listed above, but I very rarely hear anything about the next three, except for the directional pawn chain Lyudmil and myself have been discussing for the past year.

Taken together, the above factors are formidable, and should further convince us that the lopsided evaluation in favor of White cannot be correct and fixing it should well be worth the effort. We should remember that such positions are not necessarily rare in human-engine tests, if the engine is playing with a limited book and has to play on its own from the earliest moves. A skilled human WILL definitely be able to maneuver the game into such chain-dominated closed positions from either side of the board, as has been demonstrated by Lyudmil Tsvetkov (and others) right in this forum, on many occasions. It is a major exploitable weakness even top engines have, and anybody who denies it is an obfuscator.

It is my hope that a better understanding of the latter three factors will lead to progress in this difficult area.


In my assessment, there are a few more conditions that have to apply before we can consider such a chain directed at the Kingside as favorable:

-the (non-pawn) material present on the board for the attacking side should be above 20-22 points, or else a lack of firepower greatly diminishes the value of the directional chain
-in particular, having the Queens on the board is nearly mandatory, or a successful attack will be much harder to achieve
-the next most essential piece is the good Bishop, that sits on the color opposite to that of the chain; for Black it is the Bc8; when attacking from the White side, it is the Bc1
-as a rule-of-thumb minimum, I prefer that the attacking side not trade down to less than Q+B+N+R; of course, we also need enough pieces for potential thematic sacrifices, such as Bxh3 or Nxh3, etc.
-another reason not to trade too many pieces off is that it can relieve the congestion around the defending King

-the defensive pawn chain has to be present as well, as it fills out the barrier together with the attacking chain, adding to the congestion the defending side suffers from; holes in the defensive chain can help the defender by giving him counterplay, through a path that crosses into the attacker's camp; it is important that the defensive chain contain a pawn adjacent to the attacking spearhead, what I call the (main) 'congestion point' of the defensive chain
-when the defensive chain is not present, an advantage for the attacking side is no longer guaranteed, but may still be there - a lot will also depend on the search, rather than static factors at that point [see later example]

-it may well be necessary to integrate the pawn chain barrier (containing both the attacking and defensive chain portions) into an expanded concept of King Safety if all the favorable conditions are in place
-when an attacking chain is extended, and the other main conditions apply, as seen in the above example, a logical 'green light' function should be switched on in the code, where the attacker will be free to storm on the Kingside, without having to worry about its own King's shelter, since it cannot be easily attacked
-if all goes well here, even the search should now favor storming moves and Kingside attacking maneuvers over Queenside ones, if the right conditions are in place and the 'green light' is on

I have a feeling many programmers will be terrified of the king safety changes I'm proposing here, but they seem logical, and programming is all about logic. A lot of what I've mentioned in this thread, I have not seen elsewhere. If others have also toyed with such concepts, please join in and let's make this a richer discussion.


In closing part 3, as a contrast, and to show why the right conditions have to be present to make the chain favorable, I'll post an extended chain that has no opposing defensive counterpart:

[D]r1bqk1nr/1pp3bp/n2p4/p2Pp3/2P2p2/2NB2N1/PP3PPP/R1BQK2R w KQkq - 0 10

The last move played (by Morozevich) was f5-f4, "extending" the proverbial chain. Great set-up for Black, right, after reading this, wouldn't you say?! Well, not quite -- Black is in fact much worse here, IF White plays correctly. White can simply try Qh5+, or even the cheeky 0-0 (!!?) and have an immediate advantage. Since this was a blitz game, White continued with the lesser Nge4, went wrong again later on and eventually lost. The source is Balashov, Yuri - Morozevich, Alex. Moscow-ch blitz 2005.

The main problem with Black's chain is that there are no congestion points, allowing the White pieces great freedom. The chain is simply prematurely formed, and even if White does castle Kingside right away, (offering a piece sac) he still has the better position whether Black declines or not, due to his superior mobility and shaky position of the Black King. Hmm, but why do I sense somehow that Lyudmil would still prefer to be Black here?

(to be continued)

Regards,
CL

I'd like to go over the pointy chains issue again, this time attempting to break it down into finer details, which will hopefully assist others in their understanding, and (why not?) perhaps clear the path for improved coding implementations down the line.

This is not aimed at any particular engine or programmer, but an open-ended discussion that others should chime into with further on-topic comments or observations.

<snip>


In part 2, later on, I'd like to go over pointy chains directed at the Queenside, and further elaborate on congestion points.

(End of part 1 here)

Regards,
CL

Alright, now I would like to take a look at the world of pawn chains facing towards the Queenside (more precisely - towards the side of the board where the opposing King is not located).


I'll actually show a couple of notable examples from recent engine games.

[D]r1b2rk1/ppq1bpp1/2n1pn1p/3pN3/2pP1P2/2PBPQ2/PP1N2PP/R1B2RK1 w - - 0 11

In the above diagram, Fizbo 1.31, playing black, has just extended its chain (with tempo, no less) by playing c5-c4. This sort of move is probably as old as computer chess, and is typically seen in Stonewall and French formations. [I remember witnessing old forum discussions from long ago that touched on the need of keeping the engines from making such moves.] Not much has changed in this regard over the years.

Most engines, even very strong ones will routinely play like this when it comes to extending a pawn chain on the Queenside, but will balk at (correctly) doing the same thing on the Kingside! Obviously, something is really wrong, but the solution to this problem is by no means obvious.

Before a solution can be tackled in earnest, a better understanding of the elements of such positions can be very useful. So what do we have here?

Black has just extended his chain, but once the Bishop retreats to c2, it has not achieved anything except maybe gaining some space, away from the opposing King, however. What about the White pawn chain (b2-c3-d4)? It does not look all that impressive, but it can still be extended forward if e3-e4-e5 can be played. Even without the White e-pawn advance, the Black chain (f7-e6-d5-c4) is acting as a barrier that restricts most of his pieces to the Queenside, while White has more freedom to act and better chances on the Kingside, which should count for more.

So, a penalty for extending the pawn chain in the wrong direction is owed to Black, even though there is no White "pointy" chain spearheaded on Black's side of the board! I purposely picked a Stonewall position because it illustrates this very fact. I would suggest a 20 cps penalty to Black for playing c5-c4, which could be increased to 30 cps if White succeeds in getting a pawn to e5, AND thereby extending his chain, aimed at the Kingside.

Now, let's take a look at the next position, arrived at just a few moves later:

[D]r1b2rk1/p3bpp1/2q1pn1p/3pP3/1ppP1P2/2P2Q2/PPBN2PP/R1B2RK1 b - - 0 14

Lo and behold, White (represented by the wonderfully improved engine Andscacs 0.72) has gotten his e-pawn to e5, extending his chain in the right direction. We reward him with an additional bonus, bringing it to at least 30 cps. [Earlier we'd already given a penalty to Black for c5-c4 of about 20 cps, which translated into an equal bonus to White. Now we increase this bonus based on the e-pawn advancing to e5.] Andscacs went on to win a nice game from this position.


Next, I'd like to discuss congestion points from the perspective of a pawn chain pointed at the Queenside. We'll again use a game played by Andscacs, this time with Black.

[D]r1b2rk1/1pq1b1pp/2n1p3/p1ppPp2/P4P2/2PBP2P/1PQN2P1/R1B2RK1 b - - 0 13

In the above diagram, a piece exchange has taken place on e5, and Black still has the option to extend his chain by playing c5-c4 (and again, with tempo). Would such a move incur a similar penalty as before? If not, why not?

Some significant differences are present. There is no longer a congestion point on d4, where it would normally be after c5-c4, since White has already recaptured dxe5. Moreover, White will need help from Black if he is ever going to build a meaningful (of 3 pawns or more) chain directed at Kingside, by being able to somehow make an exd4 capture. This is not likely to happen here; instead, the a7-g1 diagonal will be used by Black to create counterplay.

Under the circumstances, I propose that no penalty be given to c5-c4, for the reasons specified, in particular because of the lack of a congestion point on d4.


[D]r1b2rk1/1p4pp/1qn1p3/p1bpPp2/P1pN1P2/2P1P2P/1PQ1B1P1/R1B2RK1 b - - 0 16

Now, let's take a look at the position a few moves later. Black (Andscacs) has indeed developed counterplay on the a7-g1 diagonal and despite the Knight on d4, White will see his central pawn structure totally shattered after Black's g7-g5!

Andscacs again won convincingly over Fizbo (time control was 25 min + 15 sec inc on an i3.)


To conclude this second part, I'll reiterate the importance of congestion points to directional pawn chains. If a congestion point is not present, both the positive and negative effects of pointy (or pointed) pawn chains are reduced or outright eliminated. In the case of Kingside-aiming chains, I will partially agree with Lyudmil and still award half the normal bonus (half of 30-40 cps), because of the practical attacking chances offered by the pointy chain even under less optimal circumstances.

I've changed the title of the thread a little bit [basically reverting to the original title I had in mind], to reflect the action of extending the pawn chain as these chains don't appear out of thin air, and a real choice has to be made whether to extend a chain or not, and since I'm a believer in concepts more so than the names we actually come up with to describe them, which are highly subjective anyhow. Pointy or pointed, who cares?

(end of part 2, more to follow when I have time)

Regards,
CL

Completely agree with the diagrams and conclusions from the first Andscacs game, where the pointed chain (I am a stickler too ) plays big role and wins the game for white.

However, I fully disagree with what you conclude from the second game.

[d]r1b2rk1/1p4pp/1qn1p3/p1bpPp2/P1pN1P2/2P1P2P/1PQ1B1P1/R1B2RK1 b - - 0 16

White is still better here, much better I would say, even having winning chances.

The reason is the same: the pointed white c3-d4-e5 chain. Now, there is a knight on d4, not a pawn, but that knight is defended twice by pawns and not attackable by enemy pawns, e6 is blocked, so the knight kind of acts like a linking point between c3 and e5 pawns. The best black can do to neutralise the powerful white d4 knight blocking the position and preventing black counterplay, is to capture that knight, when the c3,d4,e5 chain will be recreated.

So when you have such a blocking knight, unattackable by enemy pawns, and linking 2 pawns that would otherwise as a triade constitute a pointed chain, I would still give the pointed chain bonus.

On g5, supposed black's best move, white simply replies g3, followed by Kh2, retaining a fair share of advantage.

Eagerly waiting for your next contributions, Carl.

There is no doubt you are the best engine knower of our times.

I think you probably know more engines than I do, Lyudmil, but we'll help each other out with insider tips every now and then

After Black's g5, White's best plan is g3/Kh2 as you indicated, but Black does not have to trade on d4 at all - instead, playing Kh8/Rg8 and then double on the g-file, followed by gxf4 leaves Black with a solid position with no weaknesses, and a slight but lasting initiative. White will still have e3 and a4 to defend, and if both pairs of Rooks get traded on the g-file, a trade on d4 won't hurt Black. However, White should probably hold with best play.

[Lyudmil Tsvetkov]

[d][D]r1bqk1nr/1pp3bp/n2p4/p2Pp3/2P2p2/2NB2N1/PP3PPP/R1BQK2R w KQkq - 0 10

Very briefly from me.

I still prefer black here.

A pointed chain/triade is a triade, and other factors are other factors.

Above, the black pointed chain/d6,e5,f4 triade of pawns alone compensates for the big lag in development of black, and the unsafe position of the king.
So really a major factor here, even though most of the pawns of the chain are not blocked.

I also do not quite agree with the necessity to requite the presence of a bishop sitting on a colour opposite the colour of the triade pawns: bishop pawns would already take this into account.
Besides, a triade is useful in all situations in the middlegame, regardless of bishops, presence of queens, etc.; the important hing is that you are still into the middlegame, so a lot of material present.

Another hint might be to possibly make a distinction in terms of the different triades: but maybe in some engines a c3,d4,e5 triade is worth more than a d3,e4,f5 triade, or vice-versa?
Maybe it might make sense to distinguish in terms of the size of the bonus, being more specific and avoiding possible redundancies. Who knows?

[PK]

OK, there is something that actually helps Rodent a bit. Chains of three pawns pointing towards king side, as specified by Lyudmil, get a bonus of 20 centipawns provided that the enemy king is there. Furthermore, that bonus climbs to 28 cp if there is an nemy pawn on the "congestion point" as defined by Carl. I have made another adjustement that makes the code a bit messy, namely the chains whose spearhead is on the 4th rank need two opposing pawns to obtain the second bonus.

To make this work, I needed to reduce speed overhead by hashing pawn chains eval. Rodent already had king and pawn hashtable that became a bit bloated as a result. Right now it keeps passer eval (midgame and endgame), pawn structure eval (ditto), king shield (midgame only) and separately chain eval (midgame only).

There are many things to tweak (increasing pawn storm bonus comes to mind, as in the KID positions with the chain in place Rodent sometimes attacks with the "g" pawn, sometimes not), but I'm glad that the basic formula finally brought some benefit.

[Lyudmil Tsvetkov]

OK, there is something that actually helps Rodent a bit. Chains of three pawns pointing towards king side, as specified by Lyudmil, get a bonus of 20 centipawns provided that the enemy king is there. Furthermore, that bonus climbs to 28 cp if there is an nemy pawn on the "congestion point" as defined by Carl. I have made another adjustement that makes the code a bit messy, namely the chains whose spearhead is on the 4th rank need two opposing pawns to obtain the second bonus.

To make this work, I needed to reduce speed overhead by hashing pawn chains eval. Rodent already had king and pawn hashtable that became a bit bloated as a result. Right now it keeps passer eval (midgame and endgame), pawn structure eval (ditto), king shield (midgame only) and separately chain eval (midgame only).

There are many things to tweak (increasing pawn storm bonus comes to mind, as in the KID positions with the chain in place Rodent sometimes attacks with the "g" pawn, sometimes not), but I'm glad that the basic formula finally brought some benefit.

Long live congestion points and Carl!

Please, provide further info, Pawel: results of SPRT test, TC, elo increase, etc.

[carldaman]

[d][D]r1bqk1nr/1pp3bp/n2p4/p2Pp3/2P2p2/2NB2N1/PP3PPP/R1BQK2R w KQkq - 0 10

Very briefly from me.

I still prefer black here.

A pointed chain/triade is a triade, and other factors are other factors.

Above, the black pointed chain/d6,e5,f4 triade of pawns alone compensates for the big lag in development of black, and the unsafe position of the king.
So really a major factor here, even though most of the pawns of the chain are not blocked.

I also do not quite agree with the necessity to requite the presence of a bishop sitting on a colour opposite the colour of the triade pawns: bishop pawns would already take this into account.
Besides, a triade is useful in all situations in the middlegame, regardless of bishops, presence of queens, etc.; the important hing is that you are still into the middlegame, so a lot of material present.

Another hint might be to possibly make a distinction in terms of the different triades: but maybe in some engines a c3,d4,e5 triade is worth more than a d3,e4,f5 triade, or vice-versa?
Maybe it might make sense to distinguish in terms of the size of the bonus, being more specific and avoiding possible redundancies. Who knows?

No surprise there, Lyudmil, I know what to expect by now.

I'll say this, though - Black suffers in the above diagram for tactical reasons; f5-f4 was basically forced, and objectively, Qh5+ is now very annoying, but only IF White plays it (and he did not). Black was forced into this line because he had no better options. The position was by no means easy to play for White, either, and he in fact did go wrong allowing Black to win, meaning that Black did have some real practical chances as well.

However, in all fairness, I posted a rather extreme example, just to illustrate that when congestion points are not present, it's anybody's game, and the pointy chain is no guarantee of an advantage under those less-than-ideal circumstances.

In the next installment, I'm going to show more balanced examples of what happens when the congestion point is missing.

Objective regards,
CL

[carldaman]

OK, there is something that actually helps Rodent a bit. Chains of three pawns pointing towards king side, as specified by Lyudmil, get a bonus of 20 centipawns provided that the enemy king is there. Furthermore, that bonus climbs to 28 cp if there is an nemy pawn on the "congestion point" as defined by Carl. I have made another adjustement that makes the code a bit messy, namely the chains whose spearhead is on the 4th rank need two opposing pawns to obtain the second bonus.

To make this work, I needed to reduce speed overhead by hashing pawn chains eval. Rodent already had king and pawn hashtable that became a bit bloated as a result. Right now it keeps passer eval (midgame and endgame), pawn structure eval (ditto), king shield (midgame only) and separately chain eval (midgame only).

There are many things to tweak (increasing pawn storm bonus comes to mind, as in the KID positions with the chain in place Rodent sometimes attacks with the "g" pawn, sometimes not), but I'm glad that the basic formula finally brought some benefit.

Very interesting and encouraging developments, Pawel. I've always thought you'd be among the likeliest candidates to not only take on this challenge, but to also succeed.

My recommendation is to give an extra bonus for storming with the g and h pawns when the extended chain conditions are already in place.

Regards,
CL