PGN standard, its improvement and standardization

[Ferdy]

Due to the fact that we have a huge number of different chess shells, I believe that it is necessary to create a new format for PGN files, which will standardize computer position evals, its analysis and additional data. With a single format, it will be much more convenient to use one PGN file in various chess shells.

For example we have TCEC comments format:

Code: Select all

{d=16, sd=51, mt=161000, tl=7049000, s=219832, n=35024512, pv=Nb3 Be7 h4 Rb8 h5 b5 Bd3 f6 exf6 Nxf6 Nd4 Na5 Qe2 Bb4 Kb1 Rb7 Nb3 Nc4 Bd4 Bd6 h6 g6 g3 Rff7 Bxc4, tb=0, h=86.3, ph=0.0, wv=1.08, R50=49, Rd=-11, Rr=-11, mb=+0+0+0+0+0,}

There was a pgn enhancement proposal in 2001 from the people in chessbase and chess assistant and others and is already practiced today.
https://www.enpassant.dk/chess/palview/enhancedpgn.htm

Move comments like { [%clk 1:20:34] } or even { [%eval 25] } are from this enhancements.
The commands clk and eval and others are not intended to be fixed as specified by this enhancements but with more usage this can eventually be considered as standard.

That d=16 can be converted to:
[%acd 16]
acd=analysis count depth from epd standard opcode.

That mt=161000 can be:
[%emt 0:02:41]
that is h:mm:ss
and emt=elapsed movetime

That s=219832 or speed or nps can be:
[%nps 219832]

Most people are now using [%eval 25] but
[%ce 25] is also good because ce is from epd standard meaning centipawn evaluation.

There is also:
[%eval 25,18]
where 18 is the depth, this is still valid according to the enhancement, but to make it more clearer:
[%ce 25] [%acd 18]
1. e4 { [%ce 25] [%acd 18] [%emt 0:0:10] }

But with computer chess engines on fast TC, emt format can be extended to include ms or milliseconds.
[%emt 0:00:00:560]
h:mm:ss:mls


References:

pgn standard:
https://opensource.apple.com/source/Che ... andard.txt

pgn enhancements:
https://www.enpassant.dk/chess/palview/enhancedpgn.htm

Chess on xml:
http://www.saremba.de/chessgml/distribution.htm

Chess programming:
https://www.chessprogramming.org/Portable_Game_Notation

Others:
http://www.saremba.de/chessgml/standard ... mplete.htm
https://python-chess.readthedocs.io/en/latest/pgn.html

[sovaz1997]

There was an AMIGA program which had an interesting method to encode PGN games that was extremely compact.

For any given position, a move generator (obviously, all encodings would have to use the same generator) creates a list of legal PGN moves.
Then, the byte written to disk is the 8 bit move number.

Another simple compaction would be to encode EPD positions with a binary form (it takes about 160 bits, IIRC).
And the PGN headers could be integer pairs where the first integer is the id for the tag name in a related database table and the second integer is the id for the value.

It makes sense to have a character "player type" for human/alpha-beta/NN/Hybrid/Centaur/etc.
In the case of chess engines, I think the bare minimum to encode is something like this:

Code: Select all

CREATE TABLE [dbo].[ChessEngine](
	[Engine] [varchar](255) NULL,
	[Elo] [int] NULL,
	[plus] [int] NULL,
	[minus] [int] NULL,
	[score_pct] [float] NULL,
	[draw_pct] [float] NULL,
	[games] [int] NULL
) 

This is a great solution, store the move in 1 byte, I will use it. Also, I have the following idea: do not use one file for many games. Instead, use a single merge file that contains links to files with games. I think it will be more effective.

[Guenther]

I agree. I think we need a binary format, we must get away from the PGN format.

There are already compressed binary schemes for converting from pgn, as scid and cbh.
Please don't forget that PGN originally also had the goal to be easily readable by Humans too!

Thx god I am able to create my own scripts always to convert all kinds of ugly malformed
and misstreated PGNs to something I like - and yes, I often read them like books and
sometimes w/o chess boards too.

[sovaz1997]

I agree. I think we need a binary format, we must get away from the PGN format.

There are already compressed binary schemes for converting from pgn, as scid and cbh.
Please don't forget that PGN originally also had the goal to be easily readable by Humans too!

Thx god I am able to create my own scripts always to convert all kinds of ugly malformed
and misstreated PGNs to something I like - and yes, I often read them like books and
sometimes w/o chess boards too.

Does SCID support the evals of engines as in TCEC (with additional parameters)?
Because of this, I just started the topic. If there is, then the issue is resolved. I wanted to create a program that could show the games of the TCEC (or any other tournament) offline

[Guenther]

I agree. I think we need a binary format, we must get away from the PGN format.

There are already compressed binary schemes for converting from pgn, as scid and cbh.
Please don't forget that PGN originally also had the goal to be easily readable by Humans too!

Thx god I am able to create my own scripts always to convert all kinds of ugly malformed
and misstreated PGNs to something I like - and yes, I often read them like books and
sometimes w/o chess boards too.

Does SCID support the evals of engines as in TCEC (with additional parameters)?
Because of this, I just started the topic. If there is, then the issue is resolved. I wanted to create a program that could show the games of the TCEC (or any other tournament) offline :)

I don't know, never tried it, but there are also several scid flavoured GUIs to try.

[hgm]

PGN is intended as a format to interchange games between humans on text-based media (such as web pages). Binary formats are suitable only for storage in databases. As Guenther points out, several well-developed formats for this purpose already exist, and are in wide-spread use.

If you think you can design better database programs than scid or chessbase, by all means, go ahead. But it has absolutely nothing to do with PGN, and being completely ignorant of what you will be competing against doesn't seem a promising starting point.

[gbtami]

I suggest everyone to read Tord Romstad old suggestion here https://www.reddit.com/r/chess/comments ... pgn_format It was very interesting.

[sovaz1997]

PGN is intended as a format to interchange games between humans on text-based media (such as web pages). Binary formats are suitable only for storage in databases. As Guenther points out, several well-developed formats for this purpose already exist, and are in wide-spread use.

If you think you can design better database programs than scid or chessbase, by all means, go ahead. But it has absolutely nothing to do with PGN, and being completely ignorant of what you will be competing against doesn't seem a promising starting point.

The goal is to add the values of the engines to the game file. Moreover, the engine can be far from one.

Thus, you can get some unification of online tournaments, and then use a separate client to view the tournaments (with eval schedules, etc.). Well, and view games offline. And I want to unify so that it is just more convenient for everyone to use this.

[sovaz1997]

I suggest everyone to read Tord Romstad old suggestion here https://www.reddit.com/r/chess/comments ... pgn_format It was very interesting.

Thank you for the article!

[hgm]

It seems to me that this is mainly driven by programmer interest. "Oh, let's do away with SAN, because it is such a pain to write a parser for that"...

But programmer-friendliness should not have any weight. Approximately 0% of those ending up using the format will program their own parser.