Utility to remove duplicate positions from a PGN?

[sincerelybrent]

Is there a utility to remove duplicate positions from a PGN? Thanks

[sincerelybrent]

Looks like pgnscanner is the answer. Is it common for it to keep finding absolute or positional double positions, even after they've been removed? The first round, it found 9 doubles. I removed them. Then it found 4. I removed them. Then it found 5. I removed them. Now it found 4 more. The total number of positions in the file keeps going down by the number of positions I remove, so I know it's reading current and correct information.

[Dann Corbit]

Duplicate positions, or duplicate games?

pgn-extract removes duplicate games with the -d flag.

I do not understand why you would want to remove duplicate positions unless you are analyzing EPD.
And in that case it is trivial to pipe the sorted EPD to uniq first and get a distinct list

[sincerelybrent]

I was creating my own opening pgn for engine testing (both Silver suites, some Noomen, some Balsa, some Pohl, some Lc0, some TCEC, some of my own...). I found the duplicates.

[Frank Quisinsky]

Hi,

or looking in the FEOBOS documentation for such things (included in download file _basics.zip).
Here I added a file: readme-to-the-important-parameters.txt

http://www.amateurschach.de (menue FEOBOS).

Here you can find the information about tools, parmeters and also the files from FEOBOS without move transpositions and doubles. And of course sorted files with a ranking system for opening books in *.epd, *.pgn and all the other material.

Best
Frank