Castling long

[BubbaTough]

There is always the possibility that Alekhine and Kasparov castled long more frequently than others not because they thought it was objectively best, but because the resulting positions were more unbalanced (more likely to lead to a decisive result for one side or the other).

-Sam

By the level of Alekhine and Kasparov I would doubt that very much. How often do you remember Kasparov playing an objectively inferior move just to bluff the opponent? I think their chess grandmaster's ethics would even almost completely exclude the possibility of playing inferior moves. They might have tried to experiment sometimes, but that would only happen extremely rare (5% of the games maybe), and only when the chosen option would not compromise the position beyond a point where this would already constitute a danger for the end result.

Lyudmil

I agree that neither was likely to purposely play a move they deemed inferior (well maybe Alekhine a bit but its not my main point). To me, its not a matter of "bluffing" an opponent, it is a matter of playing an opening to reach a double edged position. Given the choice between 2 positions that seem objectively equal, one complicated and double edged, the other relatively simple, both choose complication quite regularly. Both because they enjoyed it more, and because they thought it gave them a better chance to win. At least that is my impression.

-Sam

[Don]

Majority of games
come from books or positions where already done castling.

Search Statistics
appears unreliable.
Unless the effect has been searched.

I composed a suite with the king in the center
to test my engine, I almost always make castling,
I have to re-prepare it lost "castling rights".

My statistics are based on a 5 move opening book where about 9% of white side has castling and about 14% of the black openings have castling.

I just made a 3 move (6 ply) opening book that has no castling - I'll look to see what kind of statistics Komodo produces without 10% of the castling moves being forced on it. This gives me 5,174 openings.

Don

Your highly varied and shallow books are of great interest to me (for self-testing). Currently I'm using some 49 thousand FEN positions, that I extracted from Adam Hair's 8move GM PGN file, posted on the Stockfish developpement forum.

How do you create them ? Do you use your Multi-PV to make Komodo create the book ex-nihilo ? If so, that makes me want to implement multi-PV to generate my own search-based books with any depth/branching factor that I choose.

I have made books that way for other games where there is no known theory but I have not done it that way for chess. A future project I want to do it to create a book from scratch without the benefit of human theory - something to compare against known theory to note the differences and perhaps find problems in Komodo.

The books I made are for automated testing, not tournament performance. I am using the "TWIC" data, game from the week in chess database. I wrote my own scripts to process them along with pgn-extract to clean them up, remove the comments and such (to make it easier for my scripts to parse.)

I exact one line per game and then I sort them and pipe them into the "uniq" program to remove duplicates. I require that line has occurred at least N time, where N determines the quality of the book usually.

I did just recently make a variety book in polyglot format, that was a one-off for me. It takes into consideration the quality of the player and I cannot vouch for the quality - but it is also based on current theory, not Komodo generated.

Here are the books I have made for my tester - I basically compile the books into the jar file.

Autoplayer Version 2.0

Books available:

note: FOQ is minimum number of times it appears in twic

standard - 35,533 10 ply (default book)
gambit - 167 openings, variable depth
V10 - 28,936 openings, variable depth, FOQ 10, unique
V20 - 14,703 openings, variable depth, FOQ 20, unique
V50 - 7,229 openings, variable depth, FOQ 50
F10_50 - 2,017 openings, 10 ply, FOQ 50
F20_50 - 437 openings, 20 ply, FOQ 50
F20_25 - 1,100 openings, 20 ply, FOQ 25
short - 5,174 openings, 6 ply

ending - 1,339 openings, variable depth

[Lyudmil Tsvetkov]

Hi Don again.
Here just a small further message just to keep alive the discussion (this is a discussion board, is not it?). Please, do not take it as any kind of infringement upon your status, or other programmers' status, war on engines, etc. It is just to keep the discussion going (btw., thank you very much for your statistics, I appreciated them enormously, and they are quite an useful amount of information).

I have no idea what you are talking about here - I think you are misunderstanding me completely.

Here are the castle long statistics, I probably don't have enough Stockfish games to make this accurate though and I'm putting them from most to least order:

Stockfish 3.71 percent
Komodo 3.60 percent
Houdini 2.89 percent

Note that Stockfish will castle long much more often than Houdini and this is a stylistic difference. The similarity tester works so well because each program has its own style of play and preferences and biases.

Thanks, Don,
very interesting statistics, you went to awful pains to do that.

It seems that Komodo and Stockfish do not have to do muc more than lower their long casling rate in order to catch Houdini.

If you have the time to answer, do not you think that, by perfect play, the rate of castling long/short should be a fixed number (although difficult to ascertain)?

Also, how reliable is your statistics in relation to the strength of moves reproduced on the board? If short and very short time control games have been used predominantly, that would mean that the engine would have some 5 seconds to take a complicated decision, while Alekhine some 5 minutes - a ratio of 1 to 60! I think that things might be further complicated by the fact that the decision which way to castle (almost tantamount to where to cover your king) is probably one of the most complicated ones. You need time for that, to evaluate exactly pawn storm opportunities. Could it be true that more complicated decisions in general are avoided in short time control games?

Thanks again for the statistics.

My experiences with the similarity tester and move matching shows that the style of a program hardly changes with level - even though ELO does change enormously with the level. If you really get familiar enough with a program to understand how it plays it will be same at 5 minute chess and 5 hour chess. I believe that castling is more stylistic than anything else.

I don't think it's easy to say that you should castle a certain way a certain percentage of the time to be optimal as that depends greatly on the openings that are played.

I did get a surprise with the 3 move book - Komodo is castling long 4.41 percent of the time now - a pretty big increase. I'm not playing 5 minute chess, I'm playing 5 SECOND chess. Game in 3 seconds + 0.03 fischer increment! Average depth of search is only about 11 ply. I will run some much longer time control games to see if the statistics change - my guess is that they won't but I could be wrong.

Thanks Don.
I hope you will post the results of the longer TC test.

[Lyudmil Tsvetkov]

There is always the possibility that Alekhine and Kasparov castled long more frequently than others not because they thought it was objectively best, but because the resulting positions were more unbalanced (more likely to lead to a decisive result for one side or the other).

-Sam

By the level of Alekhine and Kasparov I would doubt that very much. How often do you remember Kasparov playing an objectively inferior move just to bluff the opponent? I think their chess grandmaster's ethics would even almost completely exclude the possibility of playing inferior moves. They might have tried to experiment sometimes, but that would only happen extremely rare (5% of the games maybe), and only when the chosen option would not compromise the position beyond a point where this would already constitute a danger for the end result.

Lyudmil

I agree that neither was likely to purposely play a move they deemed inferior (well maybe Alekhine a bit but its not my main point). To me, its not a matter of "bluffing" an opponent, it is a matter of playing an opening to reach a double edged position. Given the choice between 2 positions that seem objectively equal, one complicated and double edged, the other relatively simple, both choose complication quite regularly. Both because they enjoyed it more, and because they thought it gave them a better chance to win. At least that is my impression.

-Sam

But if there is not an objective difference between the 2 choices, then engines also should have chosen the long castling lines instead.

[Lyudmil Tsvetkov]

Dear CCC friends, you might want to crucify me, but I will post a last comment on the 3 positions that were picked up.

I think Alekhine chose the right continuation in all 3 games.


[d]r2qrnk1/p3bppp/1p3n2/2pp1BB1/3P3P/2N1P3/PPQ2PP1/R3K2R w KQ - 0 14
White plays 000 and many additional factors would favour this:
- white is ready to play g4 and subsequently h5
- white has excellent control of center, Bf5 and Nc3 control e4, Nc3 controls d5, the only black knight on f6 that controls e4 and d5 is attacked by Bg5
- black has difficulty playing c4, because e4 will follow (as does in the actual game), thanks to the superior white piece control of center; b5 is currently impossible, square controlled by Nc3; a5-a4 takes some time, besides, white has superior control of the a4 square in front of the storming pawn (Nc3 and Qc2 controlling it, so it can not advance)

If white castled short instead, a large portion of those beneficial factors would have evaporated.


[d]r2q1rk1/pp3pbp/2p3p1/7n/3N4/2P1B3/PP2QPPP/R3K2R w KQ - 0 16

White plays 000 and again many additional factors favour this:
- white is ready to play g4 and h2-h4
- black has difficulty playing c5, because of Nb3 and the pawn will be lost; b5, because c6 is attacked by Nd4; a5-a4-a3 takes some time; and of course, the black queen is along a pin (x-ray attack), it will need to lose a tempo in moving
- changing Bg7 for the d4 knight is bad for black, as the bishop is better on 2 sides

If white had played 00 instead, most of those beneficial factors would have ceased to exist and white would have lost a significant portion of its advantage.

[d]r3k2r/pppqnppp/2nb4/3p4/3P2b1/2PB1N2/PP1N1PPP/R1BQR1K1 b kq - 0 9
Black plays 000 and, in doing so, is favoured by the following factors:
- black is ready to play g5 (because the f3 knight is pinned) and h5
- white has difficulty playing c4, because d4 becomes weak (isolated); b4-b5 would result in a no-advance situation, as the b6 square will be controlled by 2 enemy pawns, while the black knight, going to a5, would block the white a pawn
- a4-a5 takes some time; besides, on a4, black would almost certainly have replied with a5, and b2-b4 would be impossible, because the b4 square would be controlled by the a5 pawn and 2 black minor pieces (obviously, apart from blocking, piece control in front of storming pawns has some influence on developments)

If black had castled short instead, most of those beneficial features would have been reduced to nothing; besides, with 00 black finds itself attacked by Bd3, and g7-g6 would weaken black coloured squares in the shelter (where Bc1 is waiting to make use of this).

Overall, if I am crucified or not, I would say that Alekhine was a wise man

Any comments very much appreciated.

Best, Lyudmil

[BubbaTough]

There is always the possibility that Alekhine and Kasparov castled long more frequently than others not because they thought it was objectively best, but because the resulting positions were more unbalanced (more likely to lead to a decisive result for one side or the other).

-Sam

By the level of Alekhine and Kasparov I would doubt that very much. How often do you remember Kasparov playing an objectively inferior move just to bluff the opponent? I think their chess grandmaster's ethics would even almost completely exclude the possibility of playing inferior moves. They might have tried to experiment sometimes, but that would only happen extremely rare (5% of the games maybe), and only when the chosen option would not compromise the position beyond a point where this would already constitute a danger for the end result.

Lyudmil

I agree that neither was likely to purposely play a move they deemed inferior (well maybe Alekhine a bit but its not my main point). To me, its not a matter of "bluffing" an opponent, it is a matter of playing an opening to reach a double edged position. Given the choice between 2 positions that seem objectively equal, one complicated and double edged, the other relatively simple, both choose complication quite regularly. Both because they enjoyed it more, and because they thought it gave them a better chance to win. At least that is my impression.

-Sam

But if there is not an objective difference between the 2 choices, then engines also should have chosen the long castling lines instead.

I probably should not have said "objectively equal", I should have said "roughly equal". Human's don't nitpick 0.01 degrees of difference the way engines do. If short castling is scored at 0.13, and long castling is scored at 0.05, the engine will always pick short castling. The human will consider them both equal, and the choice becomes stylistic.

My only point is we are interpreting the data slightly differently. The data is two world champions (Alekhine and Kasparov) liked to castle long more than many others. Your interpretation is that they thought this was objectively best (defined as maximizing expected result against an equally skilled opponent). My interpretation was they did it for other reasons (such as optimizing the chance to win against inferior opponents).

-Sam

[velmarin]

With a little time I've got the four positions views
a baseline of reference of 9,500,000 items,
the first position have left about 400 games played.

H4 not the most played, even Kasparov seems to prefer the removal of Bishop.
But the victory of White reaches a high percentage. 60%.
Games:
http://www.mediafire.com/download/16ll2 ... hineH4.zip

[d] r1bq1rk1/pp1nbpp1/2p2n1p/3p2B1/3P4/2NBP3/PPQ1NPPP/R3K2R w KQ - 0 10


Positions 2 and 3, there is only games played by Alekhine.
r2qrnk1/p3bppp/1p3n2/2pp1BB1/3P3P/2N1P3/PPQ2PP1/R3K2R w KQ - 0 14
r2q1rk1/pp3pbp/2p3p1/7n/3N4/2P1B3/PP2QPPP/R3K2R w KQ - 0 16


The last position is the one that offers more games and more diversity.
In the beginning you see that Black has a high performance in all variants. 40%.
"O O O" is clearly the most widely played.
http://www.mediafire.com/download/tj3f6 ... eBlack.zip
[d] r3k2r/pppqnppp/2nb4/3p4/3P2b1/2PB1N2/PP1N1PPP/R1BQR1K1 b kq - 0 9

[Lyudmil Tsvetkov]

With a little time I've got the four positions views
a baseline of reference of 9,500,000 items,
the first position have left about 400 games played.

H4 not the most played, even Kasparov seems to prefer the removal of Bishop.
But the victory of White reaches a high percentage. 60%.
Games:
http://www.mediafire.com/download/16ll2 ... hineH4.zip

[d] r1bq1rk1/pp1nbpp1/2p2n1p/3p2B1/3P4/2NBP3/PPQ1NPPP/R3K2R w KQ - 0 10


Positions 2 and 3, there is only games played by Alekhine.
r2qrnk1/p3bppp/1p3n2/2pp1BB1/3P3P/2N1P3/PPQ2PP1/R3K2R w KQ - 0 14
r2q1rk1/pp3pbp/2p3p1/7n/3N4/2P1B3/PP2QPPP/R3K2R w KQ - 0 16


The last position is the one that offers more games and more diversity.
In the beginning you see that Black has a high performance in all variants. 40%.
"O O O" is clearly the most widely played.
http://www.mediafire.com/download/tj3f6 ... eBlack.zip
[d] r3k2r/pppqnppp/2nb4/3p4/3P2b1/2PB1N2/PP1N1PPP/R1BQR1K1 b kq - 0 9

Thanks Jose!
Very useful information. I start liking more and more statistical data, it teaches me a lot, especially when there is someone else to prepare it

The data should suggest that in Pos1 00 is the somewhat better choice than 000, and in Pos4 00 is the slightly better choice. However, are the numbers fully statistically relevant (maybe some stats wizzard might help here)? With only 20 games against 300 played in Pos1, and with couple-of-fold more games (maybe still few) played in Pos4 with 000 than 00.

Btw., do you have a clue why, if 00 seems to be the better choice in Pos4, significantly less people have played it than 000? Seems strange to me.

If I could use this message to ask Don (if he is reading it) if he had finished his Komodo test with the shorter book and longer TC? He promised to post the results, would be interesting to know.

Thanks again, Jose, great job.

Best, Lyudmil