Stockfish underpromotes much more often than Komodo

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Laskos
Posts: 8963
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Stockfish underpromotes much more often than Komodo

Post by Laskos » Wed Oct 05, 2016 7:37 am

I tried to see initially which promotes where, but the most striking result was about underpromotions. I equalized the engines to have similar strength. 10,000 games between K and SF without adjudication at close to 10'' + 0.1'' time control (adjusted for strength)

Komodo 10.1 generally promotes more often than Stockfish dev, an important thing in itself, but underpromotes almost an order of magnitude less:

Code: Select all

Komodo                 SF dev
promotions             promotions
(under-pr)             (under-pr)  

2484                   1901
(18)                   (141)
Here are the details on which square promotions (and under-promotions) occur:

Code: Select all

      Komodo    SF dev
a8    482(0)    357(17)    
b8    320(4)    261(12)  
c8    279(3)    227(17)   
d8    220(3)    168(19)  
e8    216(3)    175(20) 
f8    286(3)    184(16)   
g8    322(2)    227(16)  
h8    359(0)    302(24)
And here to what pieces underpromotions occur:

Code: Select all

Komodo:  N=9,  B=4,  R=5
SF dev:  N=35, B=13, R=93
If I am not doing something wrong (I didn't go through the games manually), it seems Stockfish underpromotes a lot, especially to Rook.

Lyudmil Tsvetkov
Posts: 6037
Joined: Tue Jun 12, 2012 10:41 am

Re: Stockfish underpromotes much more often than Komodo

Post by Lyudmil Tsvetkov » Wed Oct 05, 2016 8:07 am

That makes sense.

SF is winning more games through king attacks, and Komodo endgames other engines more, hence Komodo tends to promote more frequently.

Considering underpromotions, I have seen those SF 'underpromotions', and yes, they are very funny, many other engines also do so, probably because of some bug in the search/SEE/MVV-LVA.

Almost all SF underpromotions to rook are in positions where a queen promotion is changes nothing, as the opponent is obliged to capture on the very next move, either the promoted queen, or the underpromoted rook.

SF just prefers that a smaller-value piece(the rook) is captured(attacked) :)

Nothing changes, but I find SF approach aesthetically uglier for a human to look at, though it basically changes nothing, just evoking a funny feeling...

Jouni
Posts: 1952
Joined: Wed Mar 08, 2006 7:15 pm

Re: Stockfish underpromotes much more often than Komodo

Post by Jouni » Wed Oct 05, 2016 2:40 pm

I am curious Kai: how much you gave handicap to Komodo?
Jouni

Norm Pollock
Posts: 1015
Joined: Thu Mar 09, 2006 3:15 pm
Location: Long Island, NY, USA
Contact:

Re: Stockfish underpromotes much more often than Komodo

Post by Norm Pollock » Wed Oct 05, 2016 3:00 pm

Lyudmil Tsvetkov wrote: SF just prefers that a smaller-value piece(the rook) is captured(attacked) :)

Nothing changes, but I find SF approach aesthetically uglier for a human to look at, though it basically changes nothing, just evoking a funny feeling...
It makes it easier to spot clones.

Dann Corbit
Posts: 9492
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Stockfish underpromotes much more often than Komodo

Post by Dann Corbit » Wed Oct 05, 2016 3:19 pm

Laskos wrote:I tried to see initially which promotes where, but the most striking result was about underpromotions. I equalized the engines to have similar strength. 10,000 games between K and SF without adjudication at close to 10'' + 0.1'' time control (adjusted for strength)

Komodo 10.1 generally promotes more often than Stockfish dev, an important thing in itself, but underpromotes almost an order of magnitude less:

Code: Select all

Komodo                 SF dev
promotions             promotions
(under-pr)             (under-pr)  

2484                   1901
(18)                   (141)
Here are the details on which square promotions (and under-promotions) occur:

Code: Select all

      Komodo    SF dev
a8    482(0)    357(17)    
b8    320(4)    261(12)  
c8    279(3)    227(17)   
d8    220(3)    168(19)  
e8    216(3)    175(20) 
f8    286(3)    184(16)   
g8    322(2)    227(16)  
h8    359(0)    302(24)
And here to what pieces underpromotions occur:

Code: Select all

Komodo:  N=9,  B=4,  R=5
SF dev:  N=35, B=13, R=93
If I am not doing something wrong (I didn't go through the games manually), it seems Stockfish underpromotes a lot, especially to Rook.
It would be very interesting to know what percentage of underpromotions are necessary, and what percentage are "I would rather lose a rook than a queen" sillyness.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

matthewlai
Posts: 789
Joined: Sun Aug 03, 2014 2:48 am
Location: London, UK
Contact:

Re: Stockfish underpromotes much more often than Komodo

Post by matthewlai » Wed Oct 05, 2016 6:21 pm

Dann Corbit wrote:
Laskos wrote:I tried to see initially which promotes where, but the most striking result was about underpromotions. I equalized the engines to have similar strength. 10,000 games between K and SF without adjudication at close to 10'' + 0.1'' time control (adjusted for strength)

Komodo 10.1 generally promotes more often than Stockfish dev, an important thing in itself, but underpromotes almost an order of magnitude less:

Code: Select all

Komodo                 SF dev
promotions             promotions
(under-pr)             (under-pr)  

2484                   1901
(18)                   (141)
Here are the details on which square promotions (and under-promotions) occur:

Code: Select all

      Komodo    SF dev
a8    482(0)    357(17)    
b8    320(4)    261(12)  
c8    279(3)    227(17)   
d8    220(3)    168(19)  
e8    216(3)    175(20) 
f8    286(3)    184(16)   
g8    322(2)    227(16)  
h8    359(0)    302(24)
And here to what pieces underpromotions occur:

Code: Select all

Komodo:  N=9,  B=4,  R=5
SF dev:  N=35, B=13, R=93
If I am not doing something wrong (I didn't go through the games manually), it seems Stockfish underpromotes a lot, especially to Rook.
It would be very interesting to know what percentage of underpromotions are necessary, and what percentage are "I would rather lose a rook than a queen" sillyness.
I don't think it's "I would rather lose a rook than a queen" silliness. It's "it doesn't matter what I promote to because I'll lose it anyways so I'll flip a coin".

In minimax the leaf values are propagated up, so those two options propagate up the exact same value and one is picked randomly, and there's really nothing wrong with that. Unless you just don't like it aesthetically.
Disclosure: I work for DeepMind on the AlphaZero project, but everything I say here is personal opinion and does not reflect the views of DeepMind / Alphabet.

User avatar
Laskos
Posts: 8963
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Stockfish underpromotes much more often than Komodo

Post by Laskos » Wed Oct 05, 2016 6:45 pm

Lyudmil Tsvetkov wrote:That makes sense.

SF is winning more games through king attacks, and Komodo endgames other engines more, hence Komodo tends to promote more frequently.

Considering underpromotions, I have seen those SF 'underpromotions', and yes, they are very funny, many other engines also do so, probably because of some bug in the search/SEE/MVV-LVA.

Almost all SF underpromotions to rook are in positions where a queen promotion is changes nothing, as the opponent is obliged to capture on the very next move, either the promoted queen, or the underpromoted rook.

SF just prefers that a smaller-value piece(the rook) is captured(attacked) :)

Nothing changes, but I find SF approach aesthetically uglier for a human to look at, though it basically changes nothing, just evoking a funny feeling...
Indeed, Stockfish seems to win more often before the the endgame, I just plotted for the same database the lengths of won games, Stockfish is a bit quicker to mate:

Image

For underpromotion, wasn't Stockfish code written with the goal to eliminate useless parts, which don't bring ELO? Isn't it easier to promote to Q unless forced by the position?

User avatar
Laskos
Posts: 8963
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Stockfish underpromotes much more often than Komodo

Post by Laskos » Wed Oct 05, 2016 6:46 pm

Dann Corbit wrote:
It would be very interesting to know what percentage of underpromotions are necessary, and what percentage are "I would rather lose a rook than a queen" sillyness.
Yes, but I am unable to filter that.

Lyudmil Tsvetkov
Posts: 6037
Joined: Tue Jun 12, 2012 10:41 am

Re: Stockfish underpromotes much more often than Komodo

Post by Lyudmil Tsvetkov » Thu Oct 06, 2016 7:44 am

It has long been known that all SF bugs are actually features. :)

Almost all such underpromotions (>95%) are the funny-looking-logically correct case. It is indeed very rare that an underpromotion to rook or bishop will be necessary because of avoiding study-position stalemates.

Only downside is that not all people run tests and compares stats, there are a handful that also look at what is played on the board, and unnecessary underpromotions somehow send a signal the engine is not able to make a mistake in a limited-depth exchange sequence, which is certainly true, though very weird to humans. We will get accustomed though, as soon as we start calculating as well as engines...

Jouni
Posts: 1952
Joined: Wed Mar 08, 2006 7:15 pm

Re: Stockfish underpromotes much more often than Komodo

Post by Jouni » Wed Oct 12, 2016 4:05 pm

Can somebody post position, where bishop promotion is THE ONLY winning or drawing move? It is not so easy to find any! This is nice
[pgn]
[Event "Dortmund op-A"]
[Site "Dortmund"]
[Date "1993.??.??"]
[Round "2"]
[White "Dragoslav Tomic"]
[Black "Frank Winzbeck"]
[Result "1-0"]
[ECO "B83"]
[PlyCount "87"]
[EventDate "1993.??.??"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 Nc6 6. Be2 e6 7. Be3 Be7 8.
Qd2 a6 9. O-O O-O 10. Rad1 Qc7 11. f4 Bd7 12. Nxc6 Bxc6 13. Bf3 Rad8 14. Qf2
Rd7 15. Bb6 Qb8 16. g4 d5 17. g5 Nxe4 18. Nxe4 dxe4 19. Rxd7 Bxd7 20. Bxe4 f5
21. Bd3 e5 22. Bc5 Qd8 23. h4 e4 24. Bc4+ Kh8 25. Bd4 Bc6 26. Rd1 Bc5 27. h5
Bxd4 28. Qxd4 Qxd4+ 29. Rxd4 g6 30. h6 a5 31. Kf2 Rc8 32. Ke3 Re8 33. Rd6 Rc8
34. Be6 Re8 35. c4 Rb8 36. b3 Be8 37. a3 b6 38. Bd5 b5 39. c5 b4 40. a4 Rc8 41.
c6 Rd8 42. c7 Rc8 43. Rd8 Rxd8 44. cxd8=B 1-0

[/pgn]
but unfortunately cxd8=N mates as well if longer.
Jouni

Post Reply