Shogi

[Evert]

It was intended to be a diverting sunday-afternoon, but it turned into a curses-filled stressful bug-hunting session instead. Either way, the end result is that the development version of Sjaak now plays Shogi (at least, it no longer crashes, makes illegal moves or false-claims illegal moves).

The only real gap is that it will not object to being mated by a pawn drop (but the GUI/referee should catch that), but during the search it will flag mates caused by pawn-drops as a loss rather than a win.

That's the good news.

A typical game of shogi, as played by Sjaak, seems to go something like this: one side gains an advantage (promoted piece, winning a minor piece), which it builds up until it reaches a very large material score (>+10 or +20). It sacrifices a few minor pieces to win more material and open up the position. At some point it starts to chase the king, which runs away and slips between a gap in the position until there is no immediate threat. The formely attacking king then gets swarmed by an army of promoted minor pieces and the score flips from +20 to -mate-in-two.
There are variations where the score flips sign a few times.

One thing that's obviously lacking is any useful concept of king-safety, but there is also a distinct drop in search depth shortly before the scores tip over. I guess more aggressive pruning/reduction of pointless drops can help here, but I haven't looked into it much yet.

So what are the experiences/tips from others here who have looked at shogi (or I guess general drop variants)?

[Evert]

Ok, I've actually added some shogi variants now too: sho shogi and tori shogi. However, I'm not entirely sure what to send to XBoard in the piece-to-char table to make the promotions work correctly; the text in the man page ("A '+' similarly indicates the piece is a shogi-style promoted piece, that should revert to its non-promoted version on capture (rather than to a Pawn). Note that promoted pieces are represented by pieces 11 further in the list.") isn't as obvious to me as I think it should be.

[hgm]

This is what Shokidoki prints in response to 'variant tori' (an engine-defined variant).

Code: Select all

setup (S.....FLR.C+.....+.PKs.....flr.c+.....+.pk) 7x7+6_shogi rpckcpl/3f3/sssssss/2s1S2/SSSSSSS/3F3/LPCKCPR [-] w 0 1
piece L& fRrbBlbF
piece R& fRlbBrbF
piece P& fDbF
piece C& FvW
piece +S& fAbD
piece +F& fBbRfsWbF2

The '+' 11 places behind the S indicates the (Swallow) Pawn 'Shogi-promotes' to a +S, that 11 places behind F indicates the Falcon promotes to +F. The others won't promote. As XBoard knows nothing of the Tori pieces other than K and S, they all have to be defined in 'piece' commands.

Other drop variants supported by Shokidoki are simpler, because they use standard Shogi pieces (and Shogi is the parent variant) ; e.g. in reply to 'variant mini' Shokidoki prints:

Code: Select all

setup (P.BR.S...G.+.++.+.Kp.br.s...g.+.++.+k) 5x5+5_shogi rbsgk/4p/5/P4/KGSBR [-] w 0 1

The 'second row' of the XBoard's 'periodic system of pieces' now contains plusses in the positions corresponding to P,B,R and S, indicating these promote to +P, +B, +R and +S, respectively, and are demoted on capture. (If they would have been indicated by a letter, the 'first-row' pieces would still promote to them, but they would not demote on capture. Although they might demote to Pawn if there is no room in the holdings for them.)

Note that when there are no holdings, like in Sho Shogi, the difference between a '+' and an ID letter is only expressed in the notation (FEN or SAN), as captured pieces are simply discarded.

[hgm]

So what are the experiences/tips from others here who have looked at shogi (or I guess general drop variants)?

Well, what you describe is rather typical. I have seen Shokidoki lose games after having been at +25, or even +40.

King Safety is essential. The conventional way in Chess, where you count the number of attacked squares next to the King, is inadequate in Shogi, as it does not distinguish captures to those squares from non-captures, while non-captures (i.e. drops) can in principle make pieces appear there any time (if the square was empty).

In Shokidoki I first used a King Safety based on friendly pieces only (Pawn Shield, surrounding by 3 Generals, having a flanking Lance or back-side Knight). But it got appreciably stronger when I pay much more attention to proximity of enemy pieces, at distance 2 or 3 for the King. And especially whether there are such attackers on both sides, which is almost always lethal. You must really make it fight any intrusion of its King space as hard as it can, because any enemy foothold there acts as 'nucleation site' for a large-scale invasion.

A common pitfall is trying to collect material bonus for promoting Pawns in parts of the zone where the defense collapsed, but in the other corner as the King. The Tokins are next to useless there. Yet it is a tricky thing, because if you can trade them for material there that gets in hand, you can get a real advantage from it. But usually it takes way too long (drop, promote, attack, capture), and initiative is extremely important.

In mini-Shogi all these considerations are of little importance. The board is so small that basically every piece is always involved in an attack on the King.

In Shokidoki I reduce (non-checking) drops 2 ply (LMR-wise), board non-captures 1 ply, and I use a conventional capture search for quiesence. (No checks.) This is inadequate for Shogi, but I haven't been able to find anything better so far. It seems Bonanza tries a 3-ply tsume search (i.e. check - evade - check) to see if there is a mate in every QS node. But I think it is much more important to have a very specific King-Safety term, that really judges in detail how dangerous enemy pieces of a given type are on specific squares near your King in combination with other pieces, and the defenders you have.

[Evert]

This is what Shokidoki prints in response to 'variant tori' (an engine-defined variant).

Code: Select all

setup (S.....FLR.C+.....+.PKs.....flr.c+.....+.pk) 7x7+6_shogi rpckcpl/3f3/sssssss/2s1S2/SSSSSSS/3F3/LPCKCPR [-] w 0 1
piece L& fRrbBlbF
piece R& fRlbBrbF
piece P& fDbF
piece C& FvW
piece +S& fAbD
piece +F& fBbRfsWbF2

The '+' 11 places behind the S indicates the (Swallow) Pawn 'Shogi-promotes' to a +S, that 11 places behind F indicates the Falcon promotes to +F. The others won't promote. As XBoard knows nothing of the Tori pieces other than K and S, they all have to be defined in 'piece' commands.

Ok, I've changed Sjaak's description of Tori to use the same piece-to-char mapping (does XBoard understand different mappings for different engines?). It doesn't send "piece" commands back yet, so for now it has to play with legality testing off.

Other drop variants supported by Shokidoki are simpler, because they use standard Shogi pieces (and Shogi is the parent variant) ; e.g. in reply to 'variant mini' Shokidoki prints:

Code: Select all

setup (P.BR.S...G.+.++.+.Kp.br.s...g.+.++.+k) 5x5+5_shogi rbsgk/4p/5/P4/KGSBR [-] w 0 1

The 'second row' of the XBoard's 'periodic system of pieces' now contains plusses in the positions corresponding to P,B,R and S, indicating these promote to +P, +B, +R and +S, respectively, and are demoted on capture. (If they would have been indicated by a letter, the 'first-row' pieces would still promote to them, but they would not demote on capture. Although they might demote to Pawn if there is no room in the holdings for them.)

Sjaak assumes that if a piece which appears in FEN as "P" has the promotion option "+", it will promote to a piece which appears in FEN as "+P". It will understand the promotion character '+' as long as the promotion option is unique. A promoted piece "+P" demotes to a piece "P" when captured. Seems to work well (but doesn't generalise nicely to crazyhouse, where there are more promotion options).
Anyway, what I wanted to say: Sjaak calls this variant "minisho" because I thought "mini" gave too little information abut the game. I really wanted to call it "minishogi" but that gets recognised as "shogi" by XBoard. Is that something that is on a list of things to fix? Otherwise I'll gladly submit it as a bug somewhere.

Note that when there are no holdings, like in Sho Shogi, the difference between a '+' and an ID letter is only expressed in the notation (FEN or SAN), as captured pieces are simply discarded.

Yes, I initially had pieces promote to G (for gold general), but that didn't work: xboard flagged an illegal move when Sjaak tried to play "a9a3g". It now uses the Shogi system, which unfortunately in Sjaak means that I have a large set of duplicate gold general definitions. Oh well, at least it works.

[Evert]

Well, Sjaak's king safety currently consists of a pawn shelter and nothing else. So I'm not surprised it's rather inadequate when it comes to shogi.

Actually, there was another thing that had me wondering. Most of the discussions I've found say that a promoted Pawn (+P) is "more valuable" than a Gold (G), by which is meant that trading G for +P is a bad trade. Since it's a bad trade, I would assign G a higher material value than +P, but that seems not to be the consensus (and it is indeed at odds with the intuitive notion that having two +Ps on the board puts you in a better position than if you had two Gs). So I'm having a minor issue with wrapping my head around piece values for promoted pieces. I have the feeling I'm probably missing something here, but I'm not sure what.

[hgm]

Ok, I've changed Sjaak's description of Tori to use the same piece-to-char mapping (does XBoard understand different mappings for different engines?).

No it doesn't, and this can be an intrinsic problem with engine-defined variants. It will pay attention only to the 'setup' command of the first engine. Now as far as mapping IDs to images it really doesn't matter much how that mapping is (but I suppose it would matter to the poor user), but when they would not indicate the same piece with the same ID this would wreck matters (as XBoard will always send a setboard or edit command to the second engine, and both of these use the piece IDs).

I am not sure how that can be fixed. Perhaps there should be a /second PieceToCharTable used only for the second engine, as a work-around option.

Sjaak assumes that if a piece which appears in FEN as "P" has the promotion option "+", it will promote to a piece which appears in FEN as "+P". It will understand the promotion character '+' as long as the promotion option is unique. A promoted piece "+P" demotes to a piece "P" when captured.

Yes, that is how it should be. In XBoard this used to work only in Shogi, but in 4.8 I tried to make it generally available, so that whenever a '+' would be defined in the pieceToChatTable for a 'second-row piece' it would enable this mechanism. There was a compatibility problem with SAN, however, where a '+' suffix outside Shogi means check. So in SAN Shogi promotions outside Shogi are written with an =+ suffix.

Seems to work well (but doesn't generalise nicely to crazyhouse, where there are more promotion options).

Indeed, for Crazyhouse there is a different mechanism: in FEN pieces obtained by promotion are denoted as the piece they became, with a ~ suffix. So Q~ is a Queen that demotes to P on capture. In the pieceToChar you have to define a ~ for the second-row piece corresponding to the first-row piece to which you promoted (so 11 places behind it), rather than to the piece from which you promoted (as with '+'). The distinction between B and B~ is ignored in SAN. (I sure hope that disambiguation works properly in that case...)

Anyway, what I wanted to say: Sjaak calls this variant "minisho" because I thought "mini" gave too little information abut the game. I really wanted to call it "minishogi" but that gets recognised as "shogi" by XBoard. Is that something that is on a list of things to fix? Otherwise I'll gladly submit it as a bug somewhere.

The sloppy identification of variants (originally it was just doing a strstr on the variants feature to see if it contained the requested variant) should have been fixed in 4.8.0. If it isn't, that is indeed a bug. For what purpose is it recognized as 'shogi'? For writing the labels on the buttons?

Yes, I initially had pieces promote to G (for gold general), but that didn't work: xboard flagged an illegal move when Sjaak tried to play "a9a3g".

Could be that I allow that only ICS moves.

It now uses the Shogi system, which unfortunately in Sjaak means that I have a large set of duplicate gold general definitions. Oh well, at least it works.

Well, they really are different pieces, as they demote differently. No one to blame for this other than Shogi itself.

[hgm]

Since it's a bad trade, I would assign G a higher material value than +P, ...

No, I don't get that. If the +P is more valuable, why would you assign a higher value to G?

Another thing to keep in mind is that you never have simple trades in Shogi, because there always is a repercussion in the hand. If you trade X for Y, you basically trade 2X for 2Y. But if pieces demote in the proces there is an extra loss for him who captured the promoted one.

As you also score the hand in the material balance, the largest part of this is taken into account automatically. So effectively in the G for +P trade the side that captures the +P loses G for P (both in hand) while before on the board it was effectively equal. So with equal values for G and +P it is already recognized as a significantly bad trade, and there is no reason to score that in the value of +P.

The reason that +P is worth more than G is more a second-order 'elephantiasis-related (3Q vs 7N)' effect: Because the trade is bad, you have to avoid it, and the trade-avoiding behavior decreases the usefulness of the pieces that have to exhibit it. In Chess only pieces that are objectively stronger suffer from this. But in Shogi even objectively weaker pieces need to shy away from Tokin, as long as they are stronger than the Tokin-Pawn average. That decreases their value, or, equivalently, increases the Tokin value.

[Evert]

Ok, I've changed Sjaak's description of Tori to use the same piece-to-char mapping (does XBoard understand different mappings for different engines?).

No it doesn't, and this can be an intrinsic problem with engine-defined variants. It will pay attention only to the 'setup' command of the first engine. Now as far as mapping IDs to images it really doesn't matter much how that mapping is (but I suppose it would matter to the poor user), but when they would not indicate the same piece with the same ID this would wreck matters (as XBoard will always send a setboard or edit command to the second engine, and both of these use the piece IDs).

I am not sure how that can be fixed. Perhaps there should be a /second PieceToCharTable used only for the second engine, as a work-around option.

Well, theoretically at least both engines should (for non-standard variants) send a pieceToCharTable as well as a startup FEN string. That should allow XBoard to do the translation.

Major hassle for XBoard though, for a feature that most people will never actually use...

There was a compatibility problem with SAN, however, where a '+' suffix outside Shogi means check. So in SAN Shogi promotions outside Shogi are written with an =+ suffix.

Hmm… not sure how that would work in Sjaak. I think it always ignores = as optional for promotions, but it may now get confused when it gets a + and assume that it indicates promotion. Chances are it'll be silently ignored though (Sjaak looks at the promotion character only if the other information is not enough to disambiguate the move).

Not an issue for the communication with XBoard though, since I don't use SAN there.

Indeed, for Crazyhouse there is a different mechanism: in FEN pieces obtained by promotion are denoted as the piece they became, with a ~ suffix. So Q~ is a Queen that demotes to P on capture. In the pieceToChar you have to define a ~ for the second-row piece corresponding to the first-row piece to which you promoted (so 11 places behind it), rather than to the piece from which you promoted (as with '+'). The distinction between B and B~ is ignored in SAN. (I sure hope that disambiguation works properly in that case…)

At the moment I'll need to duplicate the piece definitions for R,N,B and Q with different letters, and manually tell it to demote them to pawn. Not great, but it would work. I may think of something better when (if) I get round to adding Crazyhouse.

The sloppy identification of variants (originally it was just doing a strstr on the variants feature to see if it contained the requested variant) should have been fixed in 4.8.0. If it isn't, that is indeed a bug. For what purpose is it recognized as 'shogi'? For writing the labels on the buttons?

It's 4.8.0, compiled myself from the tarball.
Sjaak sends

Code: Select all

feature setboard=1 time=1 sigint=0 colors=0 ping=1 memory=1 analyze=1 pause=1 nps=1 myname="Sjaak II" variants="normal,spartan,pocketknight,losalamos,micro,capablanca,minisho,shoshogi,shogi,tori"

The label "shoshogi" does not appear on a button. The logfile contains many repetitions of

Code: Select all

recognized 'shoshogi' (-1) as variant shogi

If I try to select the variant from the command-line (with -variant shoshogi) the logfile again begins with

Code: Select all

recognized 'shoshogi' (-1) as variant shogi
recognized 'shoshogi' (-1) as variant shogi
recognized 'shoshogi' (-1) as variant shogi
shuffleOpenings = 0

and starting a game produces

Code: Select all

760 >first : new
random
760 >first : variant shogi

Well, they really are different pieces, as they demote differently. No one to blame for this other than Shogi itself.

True, for shogi. For Sho Shogi there is no demotion, of course.

[Evert]

Since it's a bad trade, I would assign G a higher material value than +P, ...

No, I don't get that. If the +P is more valuable, why would you assign a higher value to G?

Re-read the first part of my sentence.
I would do that because it's a bad trade, something you want to avoid. But if +P > G, then trading G for +P would be a "good trade". Which it isn't.

I guess where that logic becomes faulty is exactly because it doesn't take into account the material balance after the trade.
Before, I have G vs +P on the board.
Afterwards I have P vs G in hand. So I lose G for P and my opponent loses +P for G. This means that the trade is bad for me, even if +P > G, as long as (G - P) >> (G - +P), which is of course the case. The evaluation will get that right though, if I don't set the piece values too crazy. My SEE routine is another matter.

The reason that +P is worth more than G is more a second-order 'elephantiasis-related (3Q vs 7N)' effect: Because the trade is bad, you have to avoid it, and the trade-avoiding behavior decreases the usefulness of the pieces that have to exhibit it. In Chess only pieces that are objectively stronger suffer from this. But in Shogi even objectively weaker pieces need to shy away from Tokin, as long as they are stronger than the Tokin-Pawn average. That decreases their value, or, equivalently, increases the Tokin value.

Ah, that does make sense now.