EPD destruction tests

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

chrisw
Posts: 4313
Joined: Tue Apr 03, 2012 4:28 pm

EPD destruction tests

Post by chrisw »

Some of these are fine, and some are broken.
What should engines do? What do engines do?
Disclaimer/admission: my C code fails because of not much integrity testing at all, but is on the todo list. While my test engine doesn't return in any sensible time in the case of 30 queens. Meanwhile, in Python my EPD checker is robust up to 2,000,000,000 epds thrown at it.

Code: Select all

rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; startpos
rnbqkbn1/ppppppppr/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; extra piece on rank
rnbqkbnr/ppppNNpp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; white 18 pieces
rnbqkbnr/pppppppp/8/8/8/8/PPPPPnPP/RNBQKBNR w KQkq - 0 1; black 17 pieces
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR K KQkq - 0 1; colour
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 400 500; movenums
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 10 5; movenums
4r1k1/p1qr1p2/2pb1Bp1/1p5p/3P1n1R/1B3P2/PP3PK1/2Q4R w - -; no movenums 
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQAb - 0 1; castle status
    rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR    w   KQkq  -   0    1 ; extra white space
rnbqkbnr.pppppppp.8.8.8.8.PPPPPPPP.RNBQKBNR w KQkq - 0 1; slash chars
nrbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; castle status
  rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; leading spaces
rnbqkbnr/ppppippp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; bad character
rnbqk0nr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; bad numeric
rnbqkbnr/pppppppp/8/6/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; square count too low
N7/8/2KQ2rp/6k1/4p3p/2p4P/4PP2/5N2 w - - 0 1; square count too high
rnkqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; 3 kings
rnbq1bnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1; 1 king
nnnnknnn/nnnnnnnn/8/8/8/8/NNNNNNNN/NNNNKNNN w - - 0 1; 30 knights
rnbqkbnr/nnnnnnnn/8/8/8/8/NNNNNNNN/RNBQKBNR w KQkq - 0 1; 18 knights
rnbqkbnr/bbbbbbbb/8/8/8/8/BBBBBBBB/RNBQKBNR w KQkq - 0 1; 18 bishops
rnbqkbnr/rrrrrrrr/8/8/8/8/RRRRRRRR/RNBQKBNR w KQkq - 0 1; 18 rooks
rnbqkbnr/qqqqqqqq/8/8/8/8/QQQQQQQQ/RNBQKBNR w KQkq - 0 1; 18 queens
qqqqkqqq/qqqqqqqq/8/8/8/8/QQQQQQQQ/QQQQKQQQ w - - 0 1; 30 queens
8/PPPPPPPP/rnbqkbnr/8/8/RNBQKBNR/pppppppp/8 w - - 0 1; lots of promotions
blablabl/blablabl/blablabl/blablabl/blablabl/blablabl/blablabl/blablabl w - - 0 1; garbage
hello have a nice day w - - 0 1; garbage


1R6/1brk2p1/4p2p/p1P1Pp2/P7/6P1/1P4P1/2R3K1 w - - 0 1 bm b8b7
4r1k1/p1qr1p2/2pb1Bp1/1p5p/3P1n1R/1B3P2/PP3PK1/2Q4R w - - 0 1 bm c1f4
r1b2rk1/ppq1bppp/2p1pn2/8/2NP4/2N1P3/PP2BPPP/2RQK2R w K - 0 1 bm e1g1
r3r1k1/1bppq2p/1p1n1bp1/3P1p2/p4N1P/P1PB1P2/1P1Q1BP1/1K1R3R w - - 0 1 bm g2g4
r1bqrk2/pp1n1n1p/3p1p2/P1pP1P1Q/2PpP1NP/6R1/2PB4/4RBK1 w - - 0 1 bm h5f7
rn1q1rk1/p2p1pb1/bp3np1/2pP2Bp/7P/P1N2N2/1PQ1PPP1/R3KB1R w KQ - 0 1 bm g2g4
N7/8/2KQ2rp/6k1/3p3p/2p4P/4PP2/5N2 w - - 0 1 bm f2f4
r1bqk2r/ppn1bppp/2n5/2p1p3/8/2NP1NP1/PP1BPPBP/R2Q1RK1 b kq - 0 1 bm e8g8
8/2p2kb1/np1p2p1/pN1Pp1Pp/2P1P2P/1P6/P4BK1/8 w - - 0 1 bm b5c7
4r1rk/p3qpp1/1pnp1n1p/5P2/P1PPP3/4Q2P/2BB2R1/6RK w - - 0 1 bm g2g7
rn1q1rk1/pp2bppp/1n2p1b1/8/2pPP3/1BN1BP2/PP2N1PP/R2Q1RK1 w - - 0 1 bm b3c2
r4r1k/1p4qp/b1pNRbp1/2P5/3p1P1P/PQ4P1/3R2BK/8 w - - 0 1 bm d2d1
2b2r1k/4q2p/3p2pQ/2pBp3/8/6P1/1PP2P1P/R5K1 w - - 0 1 bm a1a7
r2q1r2/1b2bpkp/n3p1p1/2ppP1P1/p6R/1PN1BQR1/NPP2P1P/4K3 w - - 0 1 bm f3f6
r1b1k2r/ppppqppp/8/2bP4/3p4/6P1/PPQPPPBP/R1B2RK1 b kq - 0 1 bm e8g8
rnbqkb1r/p3pppp/1p6/2ppP3/3N4/2P5/PPP1QPPP/R1B1KB1R w KQkq - 0 1 bm e5e6
mar
Posts: 2554
Joined: Fri Nov 26, 2010 2:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: EPD destruction tests

Post by mar »

chrisw wrote: Wed Feb 19, 2020 2:35 pmWhile my test engine doesn't return in any sensible time in the case of 30 queens.
qs explosion. the cure is simple: limit minimum depth (I never go deeper than -3*root_depth)
Martin Sedlak
Dann Corbit
Posts: 12538
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: EPD destruction tests

Post by Dann Corbit »

If you want some truly horrific EPD, go screen scrape (e.g. wget -m) Arves or download yacpdb. Problem composition sites are.loaded with bizarre EPD records. No kings, pawns in first and last ranks, ten pawns of one color, etc.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
chrisw
Posts: 4313
Joined: Tue Apr 03, 2012 4:28 pm

Re: EPD destruction tests

Post by chrisw »

Dann Corbit wrote: Wed Feb 19, 2020 6:16 pm If you want some truly horrific EPD, go screen scrape (e.g. wget -m) Arves or download yacpdb. Problem composition sites are.loaded with bizarre EPD records. No kings, pawns in first and last ranks, ten pawns of one color, etc.
What should chess engines do with bizarre EPDs is the question. I intend various levels of filter:
0. Will always disallow junky EPDs that decline reasonable parsing, lack of fields, wrong number squares and so on
1. Will always disallow chess illegal positions. Including pawn ranks, king count, opponent in check, impossible castling flags, dumb movenums, impossible ep square.
2. Will catch side to move being asked to search for a move when there is no move possible, game already over
Optional:
3. Will additionally disallow > 32 pieces but will allow things like 15 queens for one side
4. Will additionally disallow chess illegal piece counts (queens > N depending on pawn count etc)
User avatar
hgm
Posts: 27790
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: EPD destruction tests

Post by hgm »

It depends on what positions the engine can (or is willing) to handle. If it has no problem playing with 15 Queens, it would be a waste to prevent the user from exercising that capability. If its piece-list structure (say) would only allow for 3 Queens max, you will be forced to refuse the position. For UCI engines there is no formal way to report errors, though. You could use an 'info string' response in the hope that the GUI would show that to the user, but if that goes unnoticed, it is unavoidable that the GUI from that point on will assume a different position than the engine. UCI engines can also not resign. The only sure way to catch the user's attention is to make the engine quit. It isn't a bug, it is a (UCI) feature! :wink:

I must admit that even CECP has no formal way to complain against invalid positions loaded into the engine (so that the GUI could switch back to the position before it had before, as it would do on an illegal-move complaint). The specs recommend to just reject any subsequent move as illegal after loading an illegal position. And CECP offers more forceful ways to present a message to the user.
Dann Corbit
Posts: 12538
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: EPD destruction tests

Post by Dann Corbit »

The many components of a chess system all intertwine.
There is the chess GUI which may or may not handle a board with 31 white queens, one white king, 31 black queens and one black king.
There is the chess engine, which might pop his tiny little arrays and spew a 7 gig core dump.
And there is the wicked author of positions who composes things that simply are not possible in the game of chess.

Now, it is possible that GUIs and engines and interfaces can handle any sort of nonsense.
And it is good also if they recognize that they have bitten off more than they can chew and must punt the football.
In such a case, a friendly program will simply call the user dirty names and refuse to accept the position.

There are balky engines that won't even promote to bishop for me. But that is a design choice and I have to live with it or use another engine.
So it is possible for engines to reject even legal moves just because the designer cares only about Elo and not the art of the moves in a special problem.

As in just about everything in life, sometimes there are no simple answers. But we can try to come up with better answers if we attempt it.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Dann Corbit
Posts: 12538
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: EPD destruction tests

Post by Dann Corbit »

Sample bad data from Yacpdb, with the category of error and the EPD record.

"The board attachment limit has been reached."

See:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=585269
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
User avatar
hgm
Posts: 27790
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: EPD destruction tests

Post by hgm »

I think it is simple. Things are made for a purpose, and different purposes have often incompatible requirements. An tool for driving nails into the walls is quite unsuitable to sleep on. This is why we have developed a very wide range of tools. It is the same in chess. Having highest possible Elo and solving some quirky compositions (such as those requiring Bishop promotion or series of zugzwangs) are at odds with each other. You have to pick the tool (= engine) that was designed for the purpose you want to use it for.

You have no right to complain if you use a tool for the wrong purpose and it lets you down. Products should only be used within their specifications. If the specifications say you should feed it valid EPDs, because its purpose is to work under a GUI that uses a communication protocol that guarantees that, yfeeding it invalid EPDs is at your own risk. It might work, or might result in the proverbial nasal demons. You cannot dictate the developers to alter the purpose of their program to suit you.
Dann Corbit
Posts: 12538
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: EPD destruction tests

Post by Dann Corbit »

The thing is, I collect billions of EPD records from the internet (no, really, billions). So there are going to be errors.
What if the demon generators were handed to me, like this bunch (no, really, both kings in check, from Yacpdb):

Code: Select all

1B6/prpb2p1/2KPp3/qp1p4/Q1k5/nRP3p1/BRPP2Pp/BN6 w - -
1N4B1/2p2p2/2Q3k1/p2p1RPp/P4P1b/7P/2PPPrrK/8 w - -
1N6/p7/5R2/nkR1r3/rp1QB3/qNpp4/1p1n1p2/1K2b3 w - -
1Q2Rb2/1NB2kp1/4P2R/2K2P2/2p1P3/P7/n1bnPPP1/r2q4 w - -
1Q6/3P1R2/2N1kPpp/5BPK/1N1pp1pb/1n1b1Pp1/4P3/8 w - -
1b1knR1r/2pR3r/2b5/7K/4QBp1/1p1p3B/pn5p/7q w - -
1b6/pp6/1pN5/np4R1/1QNk2b1/R4K2/3r4/1n1r4 w - -
1bk1N3/1p1R2p1/P6p/r6p/r6p/2R1p2B/4Pp2/5Kqb w - -
1r1K1R2/2P1P3/2N4q/Bp4p1/6np/4k3/1bP1p1p1/1r1n1N1b w - -
2K5/3bBp1Q/5p2/5n2/2pN2q1/k5R1/n2rp3/8 w - -
2Q1NR2/5B2/3p2p1/p2RPk2/1p1p4/4BP1N/4pPpP/1rq3Kb w - -
2Q3k1/4b3/2K2q2/8/2B3r1/8/8/8 w - -
3N4/2p3p1/p1R4n/Rq2BkPb/2N1rPP1/2pQ3B/3nK3/2r1b3 w - -
3R4/Pp6/qK6/rp6/1R2p3/k3p2P/PQ2N3/BBN5 w - -
3RB3/1b4Bp/1b2p2p/r3kpK1/p2R2p1/n1qP2P1/1PPN2P1/2r5 w - -
3b3b/2ppRp1p/2p2KpQ/1P5P/5k1R/2P2PBq/4pPP1/6NN w - -
3n4/2Q1p3/p3p3/p2k1NP1/p7/2KP1B2/8/4b3 w - -
3nn3/2q1P3/3P2P1/B3P1b1/2Q2nP1/2q5/kn1P1qr1/2K5 w - -
4B1N1/1p3N1p/1Rn1k1pr/1K4pr/6Pp/1Q5P/1Pp1b3/4R3 w - -
4R3/1N3K2/pb4p1/1Pp2NP1/B1q1k3/1P5Q/bBP3P1/4n3 w - -
4q3/2p2K2/2N2N1B/r2R1kRr/3p4/2P3P1/p7/4Q3 w - -
5Q2/Q5P1/1q4KP/1P6/p3Q3/P4NpB/1P3ppp/RQ3kqn w - -
5b2/2Q5/2pBp3/1N1k1p2/5P2/2K1Pb2/2P1P1R1/1n1R3B w - -
5b2/2pQ3n/3kP2p/2pP4/b1pKP3/8/8/8 w - -
5b2/8/2p4q/1rB1nn2/pkR3r1/1NR5/1p4K1/1N4Q1 w - -
6B1/Q3n1n1/P2B1P1p/Rpp3pP/1p1p1rKp/1k1B3P/3p4/BrN1R2q w - -
6bn/1Np1Np2/4pPp1/Qp2P1Pp/kq2P2R/bp6/1p6/1Kr5 w - -
7Q/8/8/6p1/2N3Bk/pp3n1P/r2B1RRP/qK3n2 w - -
8/1K4Q1/3k3b/p6B/2N1bn2/3N3p/7p/6R1 w - -
8/1p6/5p2/2pP1ppn/P1R4k/2pK2rp/n1N1P3/8 w - -
8/1p6/K7/8/8/2B5/1k6/8 w - -
8/1qn5/5B2/8/2R5/k5n1/2R1P1pp/1KQ3rr w - -
8/2N5/1p6/bkp1p3/8/3pP3/Q1pK4/2N5 w - -
8/3N4/8/p7/1B6/k1Q5/1R2p1p1/rnN1K1rb w - -
8/3k1p2/4P3/8/8/6p1/3P1K2/4B3 w - -
8/3p1pq1/2p2b2/8/8/1B2B3/4pk1p/K1Rr1r2 w - -
8/3pp1bp/6pk/6Pp/1pp3rP/qK6/8/r7 w - -
8/4q1p1/4p3/5rKP/4kn1R/4bRNQ/4P1PB/5rNn w - -
8/8/1N2P2n/Q1NpB1p1/ppPk4/Kp2pPP1/1R2P3/8 w - -
8/8/4P1B1/8/4Qp2/3K1P2/1P3P2/3rk1R1 w - -
8/8/8/1k3q2/8/3K4/8/1Q6 w - -
8/8/8/5R1p/7p/6pK/5Ppq/5rQk w - -
8/8/8/8/RP5P/p6r/P5Qp/K5qk w - -
8/B2p1P1r/B7/7q/8/8/1pN1PPP1/2RQ1k1K w - -
8/P3p1Q1/4p2B/2bpPk2/3p2p1/R2b2N1/r1P1P3/KB1N4 w - -
8/R2k4/8/8/8/6Q1/K3r3/5q2 w - -
8/kq6/8/8/8/8/8/1K4Q1 w - -
Kq5r/5BPn/3p2R1/5k2/2pP1p1R/5P2/5N2/1Q6 w - -
Q4Q2/qk2P2n/NN4P1/1K4pb/Q5q1/qq6/4p3/2n5 w - -
Q4k2/2N1Bp1P/4Kp1P/5P2/8/1P6/8/n1Rq4 w - -
RBRqBb2/nQr2pp1/2pK1p2/1k6/bPpp2p1/1P1P2P1/3P4/8 w - -
Rb3b2/p1Bp1Rp1/P1pK1p2/nb6/p1k4P/2PP2P1/B1P1P3/rN6 w - -
bn2rb2/1rP4R/1KBkQP2/3P1n2/1N6/3NR3/8/6q1 w - -
k5Rq/4RNP1/5PPK/6N1/5B1r/7B/8/8 w - -
n5K1/7R/1pk2PN1/4N2R/PP2B3/1bP1P3/7B/8 w - -
q1KNNn1n/8/8/8/8/2brrb2/1RR5/3QQkB1 w - -
r2R1kb1/2R5/4P1B1/3pP3/1p1N4/1p2K1pN/1PP3Q1/n5br w - -
r6k/1K2q1Q1/8/8/8/8/8/8 w - -
r7/2N5/1bk5/p2P4/p1np4/B7/2K4r/8 w - -
rnb1k2r/pppp1ppp/8/4Q3/4P3/4q3/P1P2PPP/4KB1R w - -
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
User avatar
hgm
Posts: 27790
Joined: Fri Mar 10, 2006 10:06 am
Location: Amsterdam
Full name: H G Muller

Re: EPD destruction tests

Post by hgm »

Well, if a tool to filter those for a certain degree of legality doesn't exist, you would have to make it yourself. You cannot expect engines to do that; it is not their task. Which makes sense: engines are not stand-alone programs, but plugins for a GUI. Making the engine do the checking would require the check in many more places than having the GUI do it. There exist far more engines than GUIs. So requiring the engine does it would just be very inefficient division of labor. The program that feeds them to the engine must do it. If you use your own program for that, better make sure it does it.

UCI is more dogmatic in this than CECP. At least CECP allows engines to object to their input, through a formal Error command, or through 'Illegal Move'. So you can have the best of both worlds there: if you design an engine for some weird variant that no other engine plays, the ratio GUIs:engines reverses, and it does make sense to delegate things like legality checking to the engine, instead of requiring all GUIs to do it.