PGN standard, its improvement and standardization

[bob]

Steven and I had many discussions about PGN over the years. His required 7 tags was OK back "when". Neither he nor anybody else ever envisioned what has happened today with capturing and encoding far more information than ever imagined back then.

My main complaint back then was with the EP field in the FEN standard he developed. Specifically that the EP square in the FEN string always shows the square behind a pawn that just advanced two squares, even if there was no enemy pawn on either adjacent file that could capture that pawn en passant. This was a little messy to deal with when a program saves a FEN position, as we don't consider an EP flag unless an EP capture can actually be made.

As far as PGN goes, it has been taken to the point of absurdity. I once had a PGN file that would break almost every engine at the time, mine included until I modified the code. Had lots of deep nested () and {} stuff that was really difficult to parse. Took a stack based parser to handle it. PGN was supposed to be readable by both humans and computers. But as ChessBase started to use these nested commands and suggestions, it became much harder for me to read. Fortunately Crafty could read 'em just fine and strip out the nested stuff. Then I could do a savepgn command and save a clean PGN file with just the actual game moves so that I could actually read and understand the move sequence...

Modifying it is always possible. But it needs to be done carefully so that it doesn't have to be then modified again, leading to a half-dozen different versions of the standard.

[sovaz1997]

Thank you for the suggestions!
Initially, the question was about creating a TCEC game viewer with their evals and other info for being able to view offline. But, since TCEC is not the only one to show ecals, a uniform standard is needed.

[lucasart]

Due to the fact that we have a huge number of different chess shells, I believe that it is necessary to create a new format for PGN files, which will standardize computer position evals, its analysis and additional data. With a single format, it will be much more convenient to use one PGN file in various chess shells.

For example we have TCEC comments format:

Code: Select all

{d=16, sd=51, mt=161000, tl=7049000, s=219832, n=35024512, pv=Nb3 Be7 h4 Rb8 h5 b5 Bd3 f6 exf6 Nxf6 Nd4 Na5 Qe2 Bb4 Kb1 Rb7 Nb3 Nc4 Bd4 Bd6 h6 g6 g3 Rff7 Bxc4, tb=0, h=86.3, ph=0.0, wv=1.08, R50=49, Rd=-11, Rr=-11, mb=+0+0+0+0+0,}

And tags:

Code: Select all

[Event "TCEC Season 16 - Superfinal"]
[Site "https://tcec-chess.com"]
[Date "2019.10.02"]
[Round "39.1"]
[White "AllieStein v0.5-dev_7b41f8c-n11"]
[Black "Stockfish 19092522"]
[Result "1-0"]
[BlackElo "3892"]
[ECO "C11"]
[GameDuration "03:50:43"]
[GameEndTime "2019-10-02T22:50:54.482 W. Europe Standard Time"]
[GameStartTime "2019-10-02T19:00:10.807 W. Europe Standard Time"]
[Opening "French"]
[PlyCount "120"]
[Termination "adjudication"]
[TerminationDetails "TCEC win rule"]
[TimeControl "7200+10"]
[Variation "Steinitz, Boleslavsky variation"]
[WhiteElo "3823"]

This is not standard. Different shells even in tags use different types of time recording (7200 + 10, 7200s + 10s).

Octagon GUI:

Code: Select all

[OS "Win10 x64"]
[CPU "i9-7960X CPU 32 Threads"]
[GPU "2 x GeForce RTX 2070"]
[Memory "64Gb"]
[TournamentFormat "Gauntlet"]
[WhiteTimeControl "7200s+10s"]
[BlackTimeControl "7200s+10s"]
[TimeOdds "1"]
[OptionsWhite ""]
[OptionsBlack ""]
[OpeningOptions ""]
[Tablebases ""]
[WinRule "1 moves, evals >=10.0 for 5 moves"]
[DrawRule "35 moves, evals <= 0.08 for 5 moves"]
[PlyCount "232"]
[Time "04:40:14433"]
[Termination "normal"]
[End Reason "White wins by adjudication rule"]

I suggest discussing on this topic: which tags really need to be standardized, which ones do not. In any case, this will be an extension of the PGN format, and not its replacement. I plan to write a standard, develop translators from different formats in PGN2 and write a test GUI that will show files of a new type.

The PGN format is an abortion…

But your proposals fail to adress the real problem of PGN which is its (un)parsability, and its use of SAN (another abortion), which requires to code full chess rules and board state to just parse moves.

In any case, you will not change anything by posting in an internet forum. Ever heard the phrase "he who codes decides" ? That's how open source progresses. Code talks, talk walks.

So if you want to move the needle *you* need to do the work of writing the code to parse you new format, and writing converter tools that are easy to use to convert to and from PGN. Even then, people won't use it because they won't get any practical benefit from it. There should be useful tools (eg. GUI) that *directly* input and out from/to this format. And *you* will have to do the work here, again.

[Ovyron]

Mandatory xkcd comic:

[hgm]

WinBoard/XBoard (and also Arena, I believe) use the convention that in a comment starting with <signed floating-point number>/<integer-number> these numbers indicate the score and search depth used by an engine to obtain the move before the comment. It can be optionally followed by a space and time indicator (encoding the time in seconds or <min>:<sec>) then tells how long the engine thought about that move.

This allows a simple and compact notation, which does not destroy readablity. WinBoard's implemetation does violate the origiall PGN standard in a minor way, though: this standard requires black moves after a comment to start with a move number plus three periods. This is a sensible requirement for long comments, but rather detrimental to readability in the case of the short score/depth comments. Therefore Winboard omits these black move numbers after comments that only contain the score/depth+time info.

Code: Select all

[Event "Computer Chess Game"]
[Site "ONTWIKKELLAPTOP"]
[Date "2019.10.07"]
[Round "-"]
[White "Fairy-Max 5.0b3"]
[Black "Fruit 2.1"]
[Result "0-1"]
[TimeControl "60"]

1. Nc3 {+0.28/7} d5 {-0.20/11 2.6} 2. Nf3 {+0.08/6 0.8} Nf6 {-0.23/10 1.4}
3. d4 {+0.02/6 1.7} Nc6 {-0.08/10 2.4} 4. Bf4 {+0.02/6 1.8} Nh5
{-0.21/10 2.3} 5. Qd2 {-0.02/5 0.6} Bf5 {+0.16/9 1.7} 6. Nb5 {+0.10/6 2.1}
Rc8 {+0.36/10 0.2} 7. O-O-O {+0.13/5 0.6} e6 {+0.83/9 1.7} 8. Be3
{-0.11/4 0.6} Be7 {+0.92/9 1.9} 9. Ne5 {-0.09/6 1.3} Nxe5 {+0.64/9 2.1} 10.
dxe5 {-0.13/6 1.0} Qd7 {+0.07/10 7} 11. Nxa7 {+0.41/6 1.4} Ra8
{+0.68/9 1.3} 12. Qc3 {+0.21/6 2.6} c5 {+1.18/10 0.1} 13. g4 {+0.17/6 0.9}
Be4 {+0.58/10 2.6} 14. f3 {+0.81/7 0.8} d4 {+0.23/9 0.3} 15. Qa3
{+0.74/6 0.6} Bd5 {+1.06/9 1.0} 16. c4 {+0.65/6 0.5} Bxc4 {+1.93/9 1.1} 17.
gxh5 {-0.09/6 0.6} b6 {+2.34/10 1.2} 18. b3 {-0.26/7 0.8} Rxa7
{+1.72/10 0.1} 19. Qb2 {-0.22/7 0.5} Bd5 {+1.87/9 0.6} 20. Bf4
{-0.23/6 0.5} O-O {+1.54/9 1.4} 21. h6 {+0.22/6 0.9} c4 {+1.61/8 1.1} 22.
Qxd4 {+0.77/5 1.2} cxb3 {+2.62/7 0.2} 23. axb3 {+0.41/5 0.8} Rc8+
{+3.94/7 0.3} 24. Kb1 {+0.19/5 1.1} Qb5 {+5.98/8 0.1} 25. Qd3 {+0.20/5 1.0}
Qa5 {+99.95/38 0.7} 26. Qa6 {-1000.02/19 0.4} Qxa6 {+99.97/35 0.2} 27. e4
{-1000.01/28 0.3} Qa1# {+99.99/35 0.6}
{Xboard adjudication: Checkmate} 0-1

I also want to point out that annotating a game with engine PVs hidden in a comment (as some GUIs do) is a very bad idea. The original standard provides 'recursive variations' (enclosed in parentheses) to encode alterative move sequences, and many GUIs (including WinBoard) have mechanisms that allow the user to view these move sequences with the same ease you can step through the main game. Hiding such a PV in a normal comment (within braces) hides these from the normal display mechanisms, and thusshould be considered a very bad practice.

[niklasf]

Hiding such a PV in a normal comment (within braces) hides these from the normal display mechanisms, and thusshould be considered a very bad practice.

Unfortunately the PV often coincides with the following moves of the game, and many GUIs don't support having distinct variations for the same moves. (And for a good reason, since most of the time that's not what users want.)

[hgm]

Hiding such a PV in a normal comment (within braces) hides these from the normal display mechanisms, and thusshould be considered a very bad practice.

Unfortunately the PV often coincides with the following moves of the game, and many GUIs don't support having distinct variations for the same moves. (And for a good reason, since most of the time that's not what users want.)

I am not completely sure what you mean. But the PGN standard doesn't forbid any sequence of legal moves to be included as variation. So if GUIs do not support that, they are simply not PGN compliant, and should be fixed.

I don't understand the remark that "users do not want that". If they don't want to step through an included variation they can simply refrain from doing it. Annotated PVs being present as recursive variations doesn't clutter up the game text any more than those being present inside comments. Code that allows people to step through variations would not in itself care whether these variations were (partial) duplicats or not; it would require dedicated code to break this functionality and make the software PGN non-compliant.

[niklasf]

it would require dedicated code to break this functionality and make the software PGN non-compliant.

All it takes is choosing Node := OrderedMap<Move, Node> as the datastructure for the game tree, which would otherwise be a natural choice.

[Fulvio]

The PGN format is an abortion…

But your proposals fail to adress the real problem of PGN which is its (un)parsability

Some time ago I would have agreed, but then I rewrote the code for SCID and changed my mind.
The trick is to avoid a state-machine that reads the characters one at a time: https://sourceforge.net/p/chessx/code/H ... e.cpp#l746
Instead using a lexer that identifies the next token is surprisingly easy:
https://sourceforge.net/p/scid/code/ci/ ... xer.h#l161

[jhaglund2]

Code: Select all

[Event "FIDE World Cup 2019"]
[Site "Khanty-Mansiysk RUS"]
[Date "2019.09.24"]
[Round "5.2"]
[White "Ding Liren"]
[Black "Grischuk,A"]
[Result "1-0"]
[WhiteElo "2811"]
[BlackElo "2759"]
[ECO "A20"]
[Opening "English opening"]
[TimeControl "40/240"] //e.g.
[Annotator "1. +0.53   1... +0.05"] //e.g.

1. c4 e5 2. g3 Nf6 3. Bg2 Bc5 4. d3 d5 5. cxd5 Nxd5 6. Nc3 Nb6 7. Nf3 Nc6 8. O-O
O-O 9. a3 a5 10. Na4 Nxa4 11. Qxa4 Nd4 12. Nxd4 Bxd4 13. Bd2 c6 14. e3 Bb6 15.
Bc3 Re8 16. Rfd1 Bd7 17. Rac1 h6 18. h3 Rb8 19. Rd2 Bc7 20. d4 c5 21. Qc2 exd4
22. exd4 c4 23. a4 Bd6 24. Rdd1 b6 25. Re1 Rxe1+ 26. Rxe1 Qc7 27. h4 Re8 28. Bd5
Rxe1+ 29. Bxe1 Be6 30. Bxe6 fxe6 31. Qe4 Kf7 32. Bc3 Bf8 33. d5 Qd6 34. dxe6+
Qxe6 35. Qb7+ Kg8 36. Bd4 Qf5 37. Kh2 Qc2 38. Qd5+ Kh7 39. Qf7 Qd3 40. Bc3 Qd6
41. Qxc4 Qg6 42. Bd4 1-0

This is what I think a PGN headers should include at the most. I don't think anything else is really good to know.

Adding nested evaluations should require you to make it another format,(e.g., .PGN2 .CB-PGN) coming from the source, which added the evaluations.