how to create a labeled epd from pgn?

[zenpawn]

I'd like to create my own set of positions with game result labels for Texel tuning, ala the quiet-labeled.epd file shared on this forum, which contains lines such as:

rn2kb1r/ppp1pp1p/2q3p1/3nN3/3P4/2N1P3/PPb2PPP/R1B1KB1R b KQkq - c9 "0-1";

The first two steps in the process are easy enough: 1) generate a file of quiet position FENs, 2) run fast games starting from these positions to get results. Is there a script that will take the PGN from #2 and create a labeled epd file combining the starting positions with their results (either in a comment or just appended to the end of the line)?

Thanks,
-Erin

[AlvaroBegue]

I'd like to create my own set of positions with game result labels for Texel tuning, ala the quiet-labeled.epd file shared on this forum, which contains lines such as:

rn2kb1r/ppp1pp1p/2q3p1/3nN3/3P4/2N1P3/PPb2PPP/R1B1KB1R b KQkq - c9 "0-1";

The first two steps in the process are easy enough: 1) generate a file of quiet position FENs, 2) run fast games starting from these positions to get results. Is there a script that will take the PGN from #2 and create a labeled epd file combining the starting positions with their results (either in a comment or just appended to the end of the line)?

Thanks,
-Erin

Since the FEN and the result are both described in the header of the PGN file, it should be very easy to parse. I would use Perl to do it. If you post an example game from your database, I'll give you a tiny piece of Perl that will do what you want.

[zenpawn]

That would be great. Here's an example:

Code: Select all

[Event "SF8-SF8-STS-10sec"]
[Site "ROOK"]
[Date "2017.12.02"]
[Round "21"]
[White "Stockfish_8_x64"]
[Black "Stockfish_8_x64 [001]"]
[Result "1/2-1/2"]
[BlackElo "3456"]
[Time "05:21:30"]
[WhiteElo "3456"]
[TimeControl "10+0"]
[SetUp "1"]
[FEN "1rb3k1/2pn2pp/p2p4/4p3/1pP4q/1P1PBP1P/1PQ2P2/R3R1K1 w - - 0 1"]
[Termination "normal"]
[PlyCount "14"]
[WhiteType "program"]
[BlackType "program"]

1. c5 {(c4-c5 d6-d5 Be3-c1 c7-c6 d3-d4 Qh4xh3 d4xe5 Qh3xf3 e5-e6 Qf3-g4+
Kg1-f1 Qg4-h3+ Kf1-g1 Qh3-g4+) 0.00/14 0} d5 {(d6-d5 Be3-c1 c7-c6 d3-d4
Qh4xh3 Qc2-e2 Qh3-h5 Qe2-e3 h7-h6 Bc1-d2 e5xd4 Qe3xd4 Nd7xc5 Qd4xc5 Qh5-g6+
Kg1-f1 Qg6-d3+ Kf1-g1 Qd3xd2 Re1-e8+ Kg8-h7 Qc5-f8 Qd2-g5+ Kg1-h1 Qg5-h5+
Kh1-g1 Qh5-g5+) 0.00/16 0} 2. Bc1 {(Be3-c1 c7-c6 d3-d4 Qh4xd4 Bc1-g5 Qd4xc5
Qc2xc5 Nd7xc5 Re1xe5 h7-h6 Bg5-e7 Nc5xb3 Ra1-d1 Kg8-f7 Be7-d6 Rb8-b7 Kg1-h2
Nb3-a5 Rd1-g1 b4-b3 Rg1-g3) +0.22/15 0} Qf6 {(Qh4-f6 Qc2-e2 Nd7xc5 Qe2xe5
Qf6xe5 Re1xe5 Nc5xb3 Ra1-b1 Bc8xh3 Kg1-h2 Bh3-d7 Re5-e7 Bd7-b5 Bc1-e3 d5-d4
Rb1-g1 g7-g6 Be3-f4 Rb8-f8 Bf4xc7 Bb5xd3) +0.12/17 1} 3. f4 {(f3-f4 Qf6-g6+
Kg1-h2 Qg6-f5 Kh2-g1 Qf5-g6+) 0.00/17 1} Qg6+ {(Qf6-g6+ Kg1-h2 Qg6-f5
Kh2-g1 Qf5-g6+) 0.00/18 0} 4. Kh2 {(Kg1-h2 Qg6-f5 Kh2-g1 Qf5-g6+) 0.00/19
0} Qf5 {(Qg6-f5 Kh2-g1 Qf5-g6+) 0.00/19 0} 5. Kg1 {(Kh2-g1 Qf5-g6+) 0.00/19
0} Qg6+ {(Qf5-g6+) 0.00/19 0} 6. Kh2 {(Kg1-h2) 0.00/26 0} Qf5 {(Qg6-f5)
0.00/23 0} 7. Kg1 {(Kh2-g1) 0.00/23 0} Qg6+ {(Qf5-g6+) 0.00/19 0 3-fold
repetition} 1/2-1/2

[brtzsnr]

Given a set of PGNs I use the following command (works on my Debian)

Code: Select all

cat ok/pgn-split.* | grep "FEN\|Result\|ECO" | tac | paste - - -d' ' | grep -v ECO | sed 's/\[FEN "//' | sed 's/0 1"\] \[Result/c9/' | sed 's/\]/;/' > quiet-labeled.epd

Let's split:

Code: Select all

cat ok/pgn-split.*

just concatenates everything

Code: Select all

grep "FEN\|Result\|ECO"

keeps only the lines containing FEN, ECO or RESULT

Code: Select all

tac

reverses the lines

Code: Select all

paste - - -d' ' 

merges the lines two by two

Code: Select all

grep -v ECO

removes ECO lines. Some cutechess versions output ECO instead of FEN

Code: Select all

sed 's/\[FEN "//' | sed 's/0 1"\] \[Result/c9/' | sed 's/\]/;/'

removes extra characters and adds c9.

[jdart]

I have two tools for this in my source tree (https://github.com/jdart1/arasan-chess):

1. A program called pgnselect that takes a PGN file and produces a sampled set of EPD positions (unlabeled). If you run it with the -q switch, it will select quiet positions. This is built from my Makefile using "make utils".

2. A python3 script named "label_positions.py" in the tools directory that takes the unlabeled EPD positions and add labels, using cutechess-cli. Cutechess-cli should be set up with an engines.json file containing the engine configuration. This script will probably require a few edits to adapt to your environment.

--Jon

[zenpawn]

Thanks, everyone. These ought to do it.

[AlvaroBegue]

Code: Select all

cat something.pgn | perl -ne '$result = $1 if /Result (\".*\")/; print "$1 $result\n" if /FEN \"(.*)\"/'

The invocation `perl -ane' means "for each line of input, execute the following piece of Perl code". What the code does is remember the result if it finds a line that matches some pattern, and print out a line if it matches a different pattern. `$1' is the part of the pattern captured between the parentheses.

[zenpawn]

Good stuff. Thanks.