Request for thoughts: SEE "Test Set"

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

Ferdy
Posts: 4840
Joined: Sun Aug 10, 2008 3:15 pm
Location: Philippines

Re: Request for thoughts: SEE "Test Set"

Post by Ferdy »

AndrewGrant wrote: ↑Tue May 24, 2022 4:41 am I would like to generate // mine from existing games, a list of a few million positions with an associated move. Each entry into this big list will have the following: <fen> <some_move> <expected_see_score>. (What about <fen> <square> <score>?) The score will follow the normal conventions, where the pieces are { +1, +3, +3, +5, +9 }. These SEE scores that get computed need to be "perfect". They should be the scores that any of our engines would return, if no one was concerned about the run-time cost of the SEE() functions.

A few topics:
1. How to generate such a list? How to select from existing PGNs?
2. What does "perfect" really mean for SEE? Blockers? Pinners? Checks?
3. What is the interface for this via UCI?
4. Engines don't use 1/3/3/5/9. What do do about that?

Would be nice to know "what % of SEE positions do you compute correctly", with various amounts of effort.
Perhaps a simple csv format will do.

Code: Select all

epd,square,score,pv

Code: Select all

epd: The position based on EPD format, we don't need the hmvc and fmvn.
square: The capture begins on this square, at least 1 capture should be done.
score: This is the material count based on piece values [p=1, n=3, b=3, r=5, q=9] after a move or sequence of moves.
       It starts at zero even if the initial position has material imbalance. The score point of view is the side to move
       from the initial position.
pv: The principal variation or best sequence of moves, can be a single move. Use uci move format.
Example:

Code: Select all

5rk1/5pp1/2r4p/5b2/2R5/6Q1/R1P1qPP1/5NK1 b - -,c2,-1,f5c2 a2c2 e2c2 c4c2 c6c2
[fen]5rk1/5pp1/2r4p/5b2/2R5/6Q1/R1P1qPP1/5NK1 b - -[/fen]

Code: Select all

Black gains: 1+5+5 = 11
White gains: 3+9 = 12
stm: black
score = 11 - 12 = -1
In testing, the engine can either just output the score for a quick test or output score and pv.

For uci engines:
position fen <fen>
go see c2

The engine is told to do SEE calculation on square c2.

Example:

Code: Select all

position fen 5rk1/5pp1/2r4p/5b2/2R5/6Q1/R1P1qPP1/5NK1 b - - 0 1
go see c2
The engine may reply with:

Code: Select all

score -1 pv f5c2 a2c2 e2c2 c4c2 c6c2
or just a score, specially in cases where there are two possible pv's with equal best score.

Code: Select all

score -1
1. How to generate such a list? How to select from existing PGNs?
The engine can be revised to output such a list, Source of positions can be anything. More variety is better, from a simple 1 or 2 plies solutions to a more deeper plies.
2. What does "perfect" really mean for SEE? Blockers? Pinners? Checks?
Moves should be legal.
4. Engines don't use 1/3/3/5/9. What to do about that?
A new test set can be created for other piece values.
Top
User avatar
lithander
Posts: 881
Joined: Sun Dec 27, 2020 2:40 am
Location: Bremen, Germany
Full name: Thomas Jahn

Re: Request for thoughts: SEE "Test Set"

Post by lithander »

I'm currently implementing SEE and remembered this thread. I have gathered the positions mentioned here and converted them all into the same text format. I've also added a few positions of my own. Currently my implementation returns the correct score on all 74 entries in the test set but I can't guarantee that the dataset is 100% correct. Maybe you can validate it with your own engines?

https://github.com/lithander/Leorik/blo ... st/see.epd

Code: Select all

6k1/1pp4p/p1pb4/6q1/3P1pRr/2P4P/PP1Br1P1/5RKN w - -; Rfxf4; -100; P - R + B
5rk1/1pp2q1p/p1pb4/8/3P1NP1/2P5/1P1BQ1P1/5RK1 b - -; Bxf4; 0; -N + B
4R3/2r3p1/5bk1/1p1r3p/p2PR1P1/P1BK1P2/1P6/8 b - -; hxg4; 0;
4R3/2r3p1/5bk1/1p1r1p1p/p2PR1P1/P1BK1P2/1P6/8 b - -; hxg4; 0;
4r1k1/5pp1/nbp4p/1p2p2q/1P2P1b1/1BP2N1P/1B2QPPK/3R4 b - -; Bxf3; 0;
2r1r1k1/pp1bppbp/3p1np1/q3P3/2P2P2/1P2B3/P1N1B1PP/2RQ1RK1 b - -; dxe5; 100; P
7r/5qpk/p1Qp1b1p/3r3n/BB3p2/5p2/P1P2P2/4RK1R w - -; Re8; 0;
6rr/6pk/p1Qp1b1p/2n5/1B3p2/5p2/P1P2P2/4RK1R w - -; Re8; -500; -R
7r/5qpk/2Qp1b1p/1N1r3n/BB3p2/5p2/P1P2P2/4RK1R w - -; Re8; -500; -R
6RR/4bP2/8/8/5r2/3K4/5p2/4k3 w - -; f8=Q; 200; B - P
6RR/4bP2/8/8/5r2/3K4/5p2/4k3 w - -; f8=N; 200; N - P
7R/5P2/8/8/6r1/3K4/5p2/4k3 w - -; f8=Q; 800; Q - P
7R/5P2/8/8/6r1/3K4/5p2/4k3 w - -; f8=B; 200; B - P
7R/4bP2/8/8/1q6/3K4/5p2/4k3 w - -; f8=R; -100; -P
8/4kp2/2npp3/1Nn5/1p2PQP1/7q/1PP1B3/4KR1r b - -; Rxf1+; 0;
8/4kp2/2npp3/1Nn5/1p2P1P1/7q/1PP1B3/4KR1r b - -; Rxf1+; 0;
2r2r1k/6bp/p7/2q2p1Q/3PpP2/1B6/P5PP/2RR3K b - -; Qxc1; 100; R - Q + R
r2qk1nr/pp2ppbp/2b3p1/2p1p3/8/2N2N2/PPPP1PPP/R1BQR1K1 w kq -; Nxe5; 100; P
6r1/4kq2/b2p1p2/p1pPb3/p1P2B1Q/2P4P/2B1R1P1/6K1 w - -; Bxe5; 0;
3q2nk/pb1r1p2/np6/3P2Pp/2p1P3/2R4B/PQ3P1P/3R2K1 w - h6; gxh6; 0;
3q2nk/pb1r1p2/np6/3P2Pp/2p1P3/2R1B2B/PQ3P1P/3R2K1 w - h6; gxh6; 100; P
2r4r/1P4pk/p2p1b1p/7n/BB3p2/2R2p2/P1P2P2/4RK2 w - -; Rxc8; 500; R
2r5/1P4pk/p2p1b1p/5b1n/BB3p2/2R2p2/P1P2P2/4RK2 w - -; Rxc8; 500; R
2r4k/2r4p/p7/2b2p1b/4pP2/1BR5/P1R3PP/2Q4K w - -; Rxc5; 300; B
8/pp6/2pkp3/4bp2/2R3b1/2P5/PP4B1/1K6 w - -; Bxc6; -200; P - B
4q3/1p1pr1k1/1B2rp2/6p1/p3PP2/P3R1P1/1P2R1K1/4Q3 b - -; Rxe4; -400; P - R
4q3/1p1pr1kb/1B2rp2/6p1/p3PP2/P3R1P1/1P2R1K1/4Q3 b - -; Bxe4; 100; P
3r3k/3r4/2n1n3/8/3p4/2PR4/1B1Q4/3R3K w - -; Rxd4; -100; P - R + N - P + N - B + R - Q + R
1k1r4/1ppn3p/p4b2/4n3/8/P2N2P1/1PP1R1BP/2K1Q3 w - -; Nxe5; 100; N - N + B - R + N
1k1r3q/1ppn3p/p4b2/4p3/8/P2N2P1/1PP1R1BP/2K1Q3 w - -; Nxe5; -200; P - N
rnb2b1r/ppp2kpp/5n2/4P3/q2P3B/5R2/PPP2PPP/RN1QKB2 w Q -; Bxf6; 100; N - B + P
r2q1rk1/2p1bppp/p2p1n2/1p2P3/4P1b1/1nP1BN2/PP3PPP/RN1QR1K1 b - -; Bxf3; 0; N - B
r1bqkb1r/2pp1ppp/p1n5/1p2p3/3Pn3/1B3N2/PPP2PPP/RNBQ1RK1 b kq -; Nxd4; 0; P - N + N - P
r1bq1r2/pp1ppkbp/4N1p1/n3P1B1/8/2N5/PPP2PPP/R2QK2R w KQ -; Nxg7; 0; B - N
r1bq1r2/pp1ppkbp/4N1pB/n3P3/8/2N5/PPP2PPP/R2QK2R w KQ -; Nxg7; 300; B
rnq1k2r/1b3ppp/p2bpn2/1p1p4/3N4/1BN1P3/PPP2PPP/R1BQR1K1 b kq -; Bxh2; -200; P - B
rn2k2r/1bq2ppp/p2bpn2/1p1p4/3N4/1BN1P3/PPP2PPP/R1BQR1K1 b kq -; Bxh2; 100; P
r2qkbn1/ppp1pp1p/3p1rp1/3Pn3/4P1b1/2N2N2/PPP2PPP/R1BQKB1R b KQq -; Bxf3; 100; N - B + P
rnbq1rk1/pppp1ppp/4pn2/8/1bPP4/P1N5/1PQ1PPPP/R1B1KBNR b KQ -; Bxc3; 0; N - B
r4rk1/3nppbp/bq1p1np1/2pP4/8/2N2NPP/PP2PPB1/R1BQR1K1 b - -; Qxb2; -800; P - Q
r4rk1/1q1nppbp/b2p1np1/2pP4/8/2N2NPP/PP2PPB1/R1BQR1K1 b - -; Nxd5; -200; P - N
1r3r2/5p2/4p2p/2k1n1P1/2PN1nP1/1P3P2/8/2KR1B1R b - -; Rxb3; -400; P - R
1r3r2/5p2/4p2p/4n1P1/kPPN1nP1/5P2/8/2KR1B1R b - -; Rxb4; 100; P
2r2rk1/5pp1/pp5p/q2p4/P3n3/1Q3NP1/1P2PP1P/2RR2K1 b - -; Rxc1; 0; R - R
5rk1/5pp1/2r4p/5b2/2R5/6Q1/R1P1qPP1/5NK1 b - -; Bxc2; -100; P - B + R - Q + R
1r3r1k/p4pp1/2p1p2p/qpQP3P/2P5/3R4/PP3PP1/1K1R4 b - -; Qxa2; -800; P - Q
1r5k/p4pp1/2p1p2p/qpQP3P/2P2P2/1P1R4/P4rP1/1K1R4 b - -; Qxa2; 100; P
r2q1rk1/1b2bppp/p2p1n2/1ppNp3/3nP3/P2P1N1P/BPP2PP1/R1BQR1K1 w - -; Nxe7; 0; B - N
rnbqrbn1/pp3ppp/3p4/2p2k2/4p3/3B1K2/PPP2PPP/RNB1Q1NR w - -; Bxe4; 100; P
rnb1k2r/p3p1pp/1p3p1b/7n/1N2N3/3P1PB1/PPP1P1PP/R2QKB1R w KQkq -; Nd6; -200; -N + P
r1b1k2r/p4npp/1pp2p1b/7n/1N2N3/3P1PB1/PPP1P1PP/R2QKB1R w KQkq -; Nd6; 0; -N + N
2r1k2r/pb4pp/5p1b/2KB3n/4N3/2NP1PB1/PPP1P1PP/R2Q3R w k -; Bc6; -300; -B
2r1k2r/pb4pp/5p1b/2KB3n/1N2N3/3P1PB1/PPP1P1PP/R2Q3R w k -; Bc6; 0; -B + B
2r1k3/pbr3pp/5p1b/2KB3n/1N2N3/3P1PB1/PPP1P1PP/R2Q3R w - -; Bc6; -300; -B + B - N
5k2/p2P2pp/8/1pb5/1Nn1P1n1/6Q1/PPP4P/R3K1NR w KQ -; d8=Q; 800; (Q - P)
r4k2/p2P2pp/8/1pb5/1Nn1P1n1/6Q1/PPP4P/R3K1NR w KQ -; d8=Q; -100; (Q - P) - Q
5k2/p2P2pp/1b6/1p6/1Nn1P1n1/8/PPP4P/R2QK1NR w KQ -; d8=Q; 200; (Q - P) - Q + B
4kbnr/p1P1pppp/b7/4q3/7n/8/PP1PPPPP/RNBQKBNR w KQk -; c8=Q; -100; (Q - P) - Q
4kbnr/p1P1pppp/b7/4q3/7n/8/PPQPPPPP/RNB1KBNR w KQk -; c8=Q; 200; (Q - P) - Q + B
4kbnr/p1P1pppp/b7/4q3/7n/8/PPQPPPPP/RNB1KBNR w KQk -; c8=Q; 200; (Q - P)
4kbnr/p1P4p/b1q5/5pP1/4n3/5Q2/PP1PPP1P/RNB1KBNR w KQk f6; gxf6; 0; P - P
4kbnr/p1P4p/b1q5/5pP1/4n3/5Q2/PP1PPP1P/RNB1KBNR w KQk f6; gxf6;	0; P - P
4kbnr/p1P4p/b1q5/5pP1/4n2Q/8/PP1PPP1P/RNB1KBNR w KQk f6; gxf6; 0; P - P
1n2kb1r/p1P4p/2qb4/5pP1/4n2Q/8/PP1PPP1P/RNB1KBNR w KQk -; cxb8=Q; 200; N + (Q - P) - Q		    
rnbqk2r/pp3ppp/2p1pn2/3p4/3P4/N1P1BN2/PPB1PPPb/R2Q1RK1 w kq -; Kxh2; 300; B
3N4/2K5/2n5/1k6/8/8/8/8 b - -; Nxd8; 0; N - N
3N4/2P5/2n5/1k6/8/8/8/4K3 b - -; Nxd8; -800; N - (N + Q - P) 
3n3r/2P5/8/1k6/8/8/3Q4/4K3 w - -; Qxd8; 300; N
3n3r/2P5/8/1k6/8/8/3Q4/4K3 w - -; cxd8=Q; 700; (N + Q - P) - Q + R
r2n3r/2P1P3/4N3/1k6/8/8/8/4K3 w - -; Nxd8; 300; N
8/8/8/1k6/6b1/4N3/2p3K1/3n4 w - -; Nxd1; 0; N - N
8/8/1k6/8/8/2N1N3/2p1p1K1/3n4 w - -; Nxd1; -800; N - (N + Q - P)
8/8/1k6/8/8/2N1N3/4p1K1/3n4 w - -; Ncxd1; 100; N - (N + Q - P) + Q 
r1bqk1nr/pppp1ppp/2n5/1B2p3/1b2P3/5N2/PPPP1PPP/RNBQK2R w KQkq -; O-O; 0;
The format first gives the FEN, then the move to evaluate, followed by the correct score the SEE(fen, move) function should return, followed by sequence of captures which is useful for debugging.
Note that I assume the value of a Knight and Bishop to be both 300.
Minimal Chess (simple, open source, C#) - Youtube & Github
Leorik (competitive, in active development, C#) - Github & Lichess
Top
pedrojdm2021
Posts: 157
Joined: Fri Apr 30, 2021 7:19 am
Full name: Pedro Duran

Re: Request for thoughts: SEE "Test Set"

Post by pedrojdm2021 »

Computing an effective SEE that is not too expensive has been something that i wanted to do, but is a challenging task to do that.
Good luck with your implementation
Top
User avatar
hgm
Posts: 27895
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Request for thoughts: SEE "Test Set"

Post by hgm »

I described a table-based SEE in the 'rollout engine' topic recently. Starting from the attack map it just required a few lookups in small tables and a subtraction. Like

SEE(sqr) = seeValue[board[sqr]] - seeTable[attackers[sqr][stm]][attackers[sqr][xstm]];

The attack map can be cheaply obtained as a side effect of move generation.
Top
User avatar
phhnguyen
Posts: 1454
Joined: Wed Apr 21, 2010 4:58 am
Location: Australia
Full name: Nguyen Hong Pham

Re: Request for thoughts: SEE "Test Set"

Post by phhnguyen »

lithander wrote: ↑Sat Jul 30, 2022 2:10 am I'm currently implementing SEE and remembered this thread. I have gathered the positions mentioned here and converted them all into the same text format. I've also added a few positions of my own. Currently my implementation returns the correct score on all 74 entries in the test set but I can't guarantee that the dataset is 100% correct. Maybe you can validate it with your own engines?

https://github.com/lithander/Leorik/blo ... st/see.epd

Code: Select all

6k1/1pp4p/p1pb4/6q1/3P1pRr/2P4P/PP1Br1P1/5RKN w - -; Rfxf4; -100; P - R + B
5rk1/1pp2q1p/p1pb4/8/3P1NP1/2P5/1P1BQ1P1/5RK1 b - -; Bxf4; 0; -N + B
4R3/2r3p1/5bk1/1p1r3p/p2PR1P1/P1BK1P2/1P6/8 b - -; hxg4; 0;
4R3/2r3p1/5bk1/1p1r1p1p/p2PR1P1/P1BK1P2/1P6/8 b - -; hxg4; 0;
4r1k1/5pp1/nbp4p/1p2p2q/1P2P1b1/1BP2N1P/1B2QPPK/3R4 b - -; Bxf3; 0;
2r1r1k1/pp1bppbp/3p1np1/q3P3/2P2P2/1P2B3/P1N1B1PP/2RQ1RK1 b - -; dxe5; 100; P
7r/5qpk/p1Qp1b1p/3r3n/BB3p2/5p2/P1P2P2/4RK1R w - -; Re8; 0;
6rr/6pk/p1Qp1b1p/2n5/1B3p2/5p2/P1P2P2/4RK1R w - -; Re8; -500; -R
7r/5qpk/2Qp1b1p/1N1r3n/BB3p2/5p2/P1P2P2/4RK1R w - -; Re8; -500; -R
6RR/4bP2/8/8/5r2/3K4/5p2/4k3 w - -; f8=Q; 200; B - P
6RR/4bP2/8/8/5r2/3K4/5p2/4k3 w - -; f8=N; 200; N - P
7R/5P2/8/8/6r1/3K4/5p2/4k3 w - -; f8=Q; 800; Q - P
7R/5P2/8/8/6r1/3K4/5p2/4k3 w - -; f8=B; 200; B - P
7R/4bP2/8/8/1q6/3K4/5p2/4k3 w - -; f8=R; -100; -P
8/4kp2/2npp3/1Nn5/1p2PQP1/7q/1PP1B3/4KR1r b - -; Rxf1+; 0;
8/4kp2/2npp3/1Nn5/1p2P1P1/7q/1PP1B3/4KR1r b - -; Rxf1+; 0;
2r2r1k/6bp/p7/2q2p1Q/3PpP2/1B6/P5PP/2RR3K b - -; Qxc1; 100; R - Q + R
r2qk1nr/pp2ppbp/2b3p1/2p1p3/8/2N2N2/PPPP1PPP/R1BQR1K1 w kq -; Nxe5; 100; P
6r1/4kq2/b2p1p2/p1pPb3/p1P2B1Q/2P4P/2B1R1P1/6K1 w - -; Bxe5; 0;
3q2nk/pb1r1p2/np6/3P2Pp/2p1P3/2R4B/PQ3P1P/3R2K1 w - h6; gxh6; 0;
3q2nk/pb1r1p2/np6/3P2Pp/2p1P3/2R1B2B/PQ3P1P/3R2K1 w - h6; gxh6; 100; P
2r4r/1P4pk/p2p1b1p/7n/BB3p2/2R2p2/P1P2P2/4RK2 w - -; Rxc8; 500; R
2r5/1P4pk/p2p1b1p/5b1n/BB3p2/2R2p2/P1P2P2/4RK2 w - -; Rxc8; 500; R
2r4k/2r4p/p7/2b2p1b/4pP2/1BR5/P1R3PP/2Q4K w - -; Rxc5; 300; B
8/pp6/2pkp3/4bp2/2R3b1/2P5/PP4B1/1K6 w - -; Bxc6; -200; P - B
4q3/1p1pr1k1/1B2rp2/6p1/p3PP2/P3R1P1/1P2R1K1/4Q3 b - -; Rxe4; -400; P - R
4q3/1p1pr1kb/1B2rp2/6p1/p3PP2/P3R1P1/1P2R1K1/4Q3 b - -; Bxe4; 100; P
3r3k/3r4/2n1n3/8/3p4/2PR4/1B1Q4/3R3K w - -; Rxd4; -100; P - R + N - P + N - B + R - Q + R
1k1r4/1ppn3p/p4b2/4n3/8/P2N2P1/1PP1R1BP/2K1Q3 w - -; Nxe5; 100; N - N + B - R + N
1k1r3q/1ppn3p/p4b2/4p3/8/P2N2P1/1PP1R1BP/2K1Q3 w - -; Nxe5; -200; P - N
rnb2b1r/ppp2kpp/5n2/4P3/q2P3B/5R2/PPP2PPP/RN1QKB2 w Q -; Bxf6; 100; N - B + P
r2q1rk1/2p1bppp/p2p1n2/1p2P3/4P1b1/1nP1BN2/PP3PPP/RN1QR1K1 b - -; Bxf3; 0; N - B
r1bqkb1r/2pp1ppp/p1n5/1p2p3/3Pn3/1B3N2/PPP2PPP/RNBQ1RK1 b kq -; Nxd4; 0; P - N + N - P
r1bq1r2/pp1ppkbp/4N1p1/n3P1B1/8/2N5/PPP2PPP/R2QK2R w KQ -; Nxg7; 0; B - N
r1bq1r2/pp1ppkbp/4N1pB/n3P3/8/2N5/PPP2PPP/R2QK2R w KQ -; Nxg7; 300; B
rnq1k2r/1b3ppp/p2bpn2/1p1p4/3N4/1BN1P3/PPP2PPP/R1BQR1K1 b kq -; Bxh2; -200; P - B
rn2k2r/1bq2ppp/p2bpn2/1p1p4/3N4/1BN1P3/PPP2PPP/R1BQR1K1 b kq -; Bxh2; 100; P
r2qkbn1/ppp1pp1p/3p1rp1/3Pn3/4P1b1/2N2N2/PPP2PPP/R1BQKB1R b KQq -; Bxf3; 100; N - B + P
rnbq1rk1/pppp1ppp/4pn2/8/1bPP4/P1N5/1PQ1PPPP/R1B1KBNR b KQ -; Bxc3; 0; N - B
r4rk1/3nppbp/bq1p1np1/2pP4/8/2N2NPP/PP2PPB1/R1BQR1K1 b - -; Qxb2; -800; P - Q
r4rk1/1q1nppbp/b2p1np1/2pP4/8/2N2NPP/PP2PPB1/R1BQR1K1 b - -; Nxd5; -200; P - N
1r3r2/5p2/4p2p/2k1n1P1/2PN1nP1/1P3P2/8/2KR1B1R b - -; Rxb3; -400; P - R
1r3r2/5p2/4p2p/4n1P1/kPPN1nP1/5P2/8/2KR1B1R b - -; Rxb4; 100; P
2r2rk1/5pp1/pp5p/q2p4/P3n3/1Q3NP1/1P2PP1P/2RR2K1 b - -; Rxc1; 0; R - R
5rk1/5pp1/2r4p/5b2/2R5/6Q1/R1P1qPP1/5NK1 b - -; Bxc2; -100; P - B + R - Q + R
1r3r1k/p4pp1/2p1p2p/qpQP3P/2P5/3R4/PP3PP1/1K1R4 b - -; Qxa2; -800; P - Q
1r5k/p4pp1/2p1p2p/qpQP3P/2P2P2/1P1R4/P4rP1/1K1R4 b - -; Qxa2; 100; P
r2q1rk1/1b2bppp/p2p1n2/1ppNp3/3nP3/P2P1N1P/BPP2PP1/R1BQR1K1 w - -; Nxe7; 0; B - N
rnbqrbn1/pp3ppp/3p4/2p2k2/4p3/3B1K2/PPP2PPP/RNB1Q1NR w - -; Bxe4; 100; P
rnb1k2r/p3p1pp/1p3p1b/7n/1N2N3/3P1PB1/PPP1P1PP/R2QKB1R w KQkq -; Nd6; -200; -N + P
r1b1k2r/p4npp/1pp2p1b/7n/1N2N3/3P1PB1/PPP1P1PP/R2QKB1R w KQkq -; Nd6; 0; -N + N
2r1k2r/pb4pp/5p1b/2KB3n/4N3/2NP1PB1/PPP1P1PP/R2Q3R w k -; Bc6; -300; -B
2r1k2r/pb4pp/5p1b/2KB3n/1N2N3/3P1PB1/PPP1P1PP/R2Q3R w k -; Bc6; 0; -B + B
2r1k3/pbr3pp/5p1b/2KB3n/1N2N3/3P1PB1/PPP1P1PP/R2Q3R w - -; Bc6; -300; -B + B - N
5k2/p2P2pp/8/1pb5/1Nn1P1n1/6Q1/PPP4P/R3K1NR w KQ -; d8=Q; 800; (Q - P)
r4k2/p2P2pp/8/1pb5/1Nn1P1n1/6Q1/PPP4P/R3K1NR w KQ -; d8=Q; -100; (Q - P) - Q
5k2/p2P2pp/1b6/1p6/1Nn1P1n1/8/PPP4P/R2QK1NR w KQ -; d8=Q; 200; (Q - P) - Q + B
4kbnr/p1P1pppp/b7/4q3/7n/8/PP1PPPPP/RNBQKBNR w KQk -; c8=Q; -100; (Q - P) - Q
4kbnr/p1P1pppp/b7/4q3/7n/8/PPQPPPPP/RNB1KBNR w KQk -; c8=Q; 200; (Q - P) - Q + B
4kbnr/p1P1pppp/b7/4q3/7n/8/PPQPPPPP/RNB1KBNR w KQk -; c8=Q; 200; (Q - P)
4kbnr/p1P4p/b1q5/5pP1/4n3/5Q2/PP1PPP1P/RNB1KBNR w KQk f6; gxf6; 0; P - P
4kbnr/p1P4p/b1q5/5pP1/4n3/5Q2/PP1PPP1P/RNB1KBNR w KQk f6; gxf6;	0; P - P
4kbnr/p1P4p/b1q5/5pP1/4n2Q/8/PP1PPP1P/RNB1KBNR w KQk f6; gxf6; 0; P - P
1n2kb1r/p1P4p/2qb4/5pP1/4n2Q/8/PP1PPP1P/RNB1KBNR w KQk -; cxb8=Q; 200; N + (Q - P) - Q		    
rnbqk2r/pp3ppp/2p1pn2/3p4/3P4/N1P1BN2/PPB1PPPb/R2Q1RK1 w kq -; Kxh2; 300; B
3N4/2K5/2n5/1k6/8/8/8/8 b - -; Nxd8; 0; N - N
3N4/2P5/2n5/1k6/8/8/8/4K3 b - -; Nxd8; -800; N - (N + Q - P) 
3n3r/2P5/8/1k6/8/8/3Q4/4K3 w - -; Qxd8; 300; N
3n3r/2P5/8/1k6/8/8/3Q4/4K3 w - -; cxd8=Q; 700; (N + Q - P) - Q + R
r2n3r/2P1P3/4N3/1k6/8/8/8/4K3 w - -; Nxd8; 300; N
8/8/8/1k6/6b1/4N3/2p3K1/3n4 w - -; Nxd1; 0; N - N
8/8/1k6/8/8/2N1N3/2p1p1K1/3n4 w - -; Nxd1; -800; N - (N + Q - P)
8/8/1k6/8/8/2N1N3/4p1K1/3n4 w - -; Ncxd1; 100; N - (N + Q - P) + Q 
r1bqk1nr/pppp1ppp/2n5/1B2p3/1b2P3/5N2/PPPP1PPP/RNBQK2R w KQkq -; O-O; 0;
The format first gives the FEN, then the move to evaluate, followed by the correct score the SEE(fen, move) function should return, followed by sequence of captures which is useful for debugging.
Note that I assume the value of a Knight and Bishop to be both 300.
Thanks for sharing. It is the very right time for me since I have just implemented a new one and it helped me to fix some errors!

Almost all tests are fine, but one. Because those tests have considered promotions thus the below one may be a problem:

Code: Select all

8/8/8/1k6/6b1/4N3/2p3K1/3n4 w - -; Nxd1; 0; N - N

[fen]8/8/8/1k6/6b1/4N3/2p3K1/3n4 w - -[/fen]


SEE of move Nxd1 should have a value of -800, not zero since Black could gain more from promoting black Pawn.

Another issue is that the format is not EPD, and cannot be parsed by EPD readers. A correct format one should be:

Code: Select all

6k1/1pp4p/p1pb4/6q1/3P1pRr/2P4P/PP1Br1P1/5RKN w - - id "Leorik SEE testsuit"; bm Rfxf4; ce -100; c0 "P - R + B";
https://banksiagui.com
The most features chess GUI, based on opensource Banksia - the chess tournament manager
Top
User avatar
lithander
Posts: 881
Joined: Sun Dec 27, 2020 2:40 am
Location: Bremen, Germany
Full name: Thomas Jahn

Re: Request for thoughts: SEE "Test Set"

Post by lithander »

phhnguyen wrote: ↑Sun Jul 31, 2022 3:43 pm Almost all tests are fine, but one. Because those tests have considered promotions thus the below one may be a problem:

Code: Select all

8/8/8/1k6/6b1/4N3/2p3K1/3n4 w - -; Nxd1; 0; N - N
[fen]8/8/8/1k6/6b1/4N3/2p3K1/3n4 w - -[/fen]

SEE of move Nxd1 should have a value of -800, not zero since Black could gain more from promoting black Pawn.
That's an interesting one but I think 0 is correct because usually you'd capture with the least valuable piece first and the most valuable piece last. A normal pawn capture is least valuable but a pawn capture that promotes I think should be valued between the rook and the queen with 800 cp. (Because if the promotion is not the last move in the static exchange than you will lose a queen and in that case losing only a bishop would be better)
phhnguyen wrote: ↑Sun Jul 31, 2022 3:43 pm Another issue is that the format is not EPD, and cannot be parsed by EPD readers. A correct format one should be:

Code: Select all

6k1/1pp4p/p1pb4/6q1/3P1pRr/2P4P/PP1Br1P1/5RKN w - - id "Leorik SEE testsuit"; bm Rfxf4; ce -100; c0 "P - R + B";
EPDs are usually talking about the entire position. Like bm is the best move in that position and ce it's evaluation. But to store a SEE test suite in that format would mean we repurpose the fields. bm is now not the best move but the move you have to pass into the SEE() function with the position and ce is not the evaluation of the position but the expected output from the SEE function. Isn't that confusing?
Minimal Chess (simple, open source, C#) - Youtube & Github
Leorik (competitive, in active development, C#) - Github & Lichess
Top
User avatar
hgm
Posts: 27895
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Request for thoughts: SEE "Test Set"

Post by hgm »

I would not call 0 correct, because it leads to wrong pruning decisions. The point is that you need a different SEE, using a different capture order, on a promotion square.
Top
User avatar
lithander
Posts: 881
Joined: Sun Dec 27, 2020 2:40 am
Location: Bremen, Germany
Full name: Thomas Jahn

Re: Request for thoughts: SEE "Test Set"

Post by lithander »

hgm wrote: ↑Mon Aug 01, 2022 3:38 pm I would not call 0 correct, because it leads to wrong pruning decisions. The point is that you need a different SEE, using a different capture order, on a promotion square.
So what would be correct? -800? I had the test-set I gathered (from multiple sources) guide my implementation because I have no good intuition what the correct implementation should be when promotions or en-passant are involved.

Frankly, I was considering to just not use SEE at all in my engine but call into Qsearch whenever I want to know the "value" of a move. SEE doesn't consider (as far as I can see) pinned pieces or giving check and so it seems to not really always give an accurate result, anyway. If I don't care about accuracy and only speed why not just check if on a negative MvvLva score the victim is protected or not? And if I do care about accuracy but not speed then my Qsearch implementation does that already. There seems to be little space inbetween for SEE to be of benefit. (But that's exactly why I wanted to implement it: so I could experiment with it and find out)
Minimal Chess (simple, open source, C#) - Youtube & Github
Leorik (competitive, in active development, C#) - Github & Lichess
Top
User avatar
hgm
Posts: 27895
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: Request for thoughts: SEE "Test Set"

Post by hgm »

Indeed, I think that perhaps as much as 90% of the beneficial effect of SEE is just from figuring out wheter the victim is protected or not. Deep SEEs just do not occur frequently, and are very unreliable estimates anyway. So I just postpone search of H x protected L captures (BLIND = Better or Lesser If Not Defended). Most of the work goes in figuring out wheter victims are protected, though. And in figuring that out you usually get to know what protects them as well. That makes it hardly nore expensive to use a better SEE approximation, if you use a table-based approach.

I think that what you would want to see is indeed -800, so that you would immediately abandon the idea of searching N x N.
Top
JVMerlino
Posts: 1357
Joined: Wed Mar 08, 2006 10:15 pm
Location: San Francisco, California

Re: Request for thoughts: SEE "Test Set"

Post by JVMerlino »

Thank you all for this discussion, and mostly for lithander for the test positions. I wasn't even halfway through the list when I discovered two bugs in my SEE implementation:
1) Captures by Kings were not included, and
2) If the first capture was of equal value, the calculation stopped and returned 0.

I wonder how much elo this will give? I'm guessing.....two? :D

Thank you!
Top

Return to “Computer Chess Club: Programming and Technical Discussions”