How to clean multiple pgn files in a folder all at once?

[Jonathan003]

I wonder if it is possible somehow to clean multiple pgn's in a folder all at once?
Now I use Scid on my Windows 11 Pc to clean the pgn's.
I open the pgn's and export the pgn's again, (I overwrite the orgignal pgn's).
I can choose in the export window of SCID If I also want to export variations ore comments.
But that's alot of work when you want to do it with multiple pgn files in a folder.
I wonder if there would be some easyer way?
For example with pgn-extract?

[Ajedrecista]

Hello Jonathan:

If you want to clean all the PGN files and keeping them separated is not important, how about trying to merge all the files into one? Since PGN is a kind of text file, I would suggest give a try with the command prompt, browsing to the folder with all the PGN and then:

Code: Select all

type *.pgn >> merged.pgn

This line should work if merged.pgn name is not already used (chose a different name in other case to avoid overwriting as a backup in case things go wrong). You can apply then your cleaning method to the merged file only.

If keeping separated the original PGN files is compulsory and you want to have X smaller cleaned files instead one bigger cleaned file, merge is not an option anymore and other method must be proposed. Good luck!

Regards from Spain.

Ajedrecista.

[Jonathan003]

Thanks for the sugestion, but I want to keep the pgn's seperated. I only want to clean them, so it dousn't give any problems when I import the pgn's in some opening training apps, ore if I want to make a bin book of one of the pgn's.

[chesskobra]

pgn-extract + a little scripting (here is how I would do using bash, but I am sure there is something like that on windows)

Code: Select all

function makebooks {
	for x in *.pgn 
	do
		pgn-extract -7 $x -o $x-out.pgn
		polyglot makebook -pgn $x-out.pgn -bin $x.bin
	done
}

There could be minor mistakes above, but this is the overall idea. The pgn-extract line is removing all tags except the 7 essential tags, and polyglot command is making a bin book. For book building you can also use a recent tool called jja (the developer is here sometimes, and he answers questions about his tool). pgn-extract has tons of other features for filtering games by different criteria, adding ECO codes, splitting a large pgn file by ECO codes, and so on.

[Jonathan003]

I don't need a scrip for making bin books. I want a script to clean multiple pgn files in a folder, and overwrite the original pgn files with the cleaned ones, in one go. And some option to keep variations or not, to keep comments or not, to keep graphical information like arrows; highlighted squars or not...
I want the pgn's to be clean so they don't give problems when importing them in software to train openings, and don't give problems when making bin books.
And I want the scipt to work on Windows 11.
I'm pretty sure it can be done with pgn-extract, I only have to know how.
I spend an awfully lot of time draging pgn's to SCID and exporting them again, to clean the pgn's.
So such a script would make my work much easyer.

[JoAnnP38]

Here is the .BAT script I used in windows to clean all my PGNs that I am gathering from self-play to use in Pedantic training. Note that the first step cleans and merges the results into merged.pgn. I believe this allows PGN-EXTRACT to better detect duplicate games. It then does a final cleaning that removes duplicates and outputs the files to pedantic-clean.pgn. If you want to overwrite all of your original PGNs you will need to output each file to an intermediate file and then add some post cleaning scripting to copy back over the originals. (I'm not recommending this last step as it may destroy your original files which may or may not be difficult to recreate.) This script will take all PGN files pass on the command line, clean them and then leave the result in Pedantic-clean.pgn. I'm not going to write your script for you, but I'm sure you can figure it out by pouring over the command-line options of PGN-EXTRACT and writing your own .BAT script.

Code: Select all

SET PATH=D:\bin;%PATH%

FOR %%F IN (%*) DO (
	pgn-extract -amerged.pgn -D -s -C -N -V -Z -Tr1-0 -Tr0-1 -Tr1/2-1/2 -lmerge_errors.txt -tstartpos.txt --fixtagstrings -w80 --minmoves 8 --nobadresults "%%F"
)
pgn-extract -Wuci -opedantic-clean.pgn -D -s -C -N -V -Z -lmerge_errors.txt -tstartpos.txt -w128 --minmoves 8 --nobadresults merged.pgn
DEL merged.pgn

[Jonathan003]

I also created a scipt with the help of ChatGPT

Code: Select all

$inputDirectory = "C:\Users\creme\Downloads\Chess\to_clean_pgns\input"
$outputDirectory = "C:\Users\creme\Downloads\Chess\to_clean_pgns\output"
$pgnExtractPath = "C:\Users\creme\Downloads\Chess\to_clean_pgns\pgn-extract.exe"

Get-ChildItem -Path $inputDirectory -Filter *.pgn | ForEach-Object {
    $outputFile = Join-Path $outputDirectory $_.Name
    & $pgnExtractPath -C -N -V -o $outputFile --nocomments $_.FullName
}

I saved the text file to "Clean-PGN.ps1"
And I run the script using PowerShell: .\Clean-PGN.ps1
I have to do some more testing to see how it works.
The pgn files are not overwritten but the cleaned pgn files are saved to another folder. That's also ok for me.

[Jonathan003]

I changed the my approche to make it more user friendly.
I created 4 batch files with the help of ChatGPT 3,5

Put pgn-extract, eco.pgn and the .bat files, in the same folder as the pgn files to clean.
Drag the to clean pgn files to the desired bat file.
The cleaned pgn files are in the output folder.

I'm prety happy with the result, but I have to do more testing.

This is a link to my Google drive for the batch files and a readme:
https://drive.google.com/file/d/1AkZCMn ... ?usp=shari...