Open Chess Game Database Standard

[sarona]

I did use imgBB for the screenshot.

Sorry! I had noticed that issue last week and forgot to use my changed syntax.

Try this. It has worked for me.

ocgdb -db (yourdatabasename).ocgdb.db3 -cpu 4 -dup -plycount 40 -o printall -o remove

Make a copy of the original database. Run the above command. Then browse the game data (I used DB Browser for SQLite) in the original and cleaned databases. Compare the game totals to make sure the doubles have been removed.

[sarona]

. . . And to export the result (without the doubles), back to a pgn database?

I did not read your entire post properly the first time.

To export the cleaned database back to pgn, enter the following command:

ocgdb -pgn (your_pgn_file) -db (your_database_name).ocgdb.db3 -cpu 4 -o export

I read that you were experiencing slow speeds when exporting back to pgn on your hardware. That should not be the case. The speed should be fast.

[Jonathan003]

Dit you download the DB Browser for SQLite from here: https://sqlitebrowser.org/dl/
I installed it to give it a try, but I don't know how to work with it. I think I don't need it to delete duplicates from a pgn?

I haver made a small example pgn databases with 60 games '60_games_with_doubles.pgn'
I know about half of the games are duplicates, because I have searched the database for duplicates with Chessbase.

I don't get it to work to create a new pgn database with the duplicates remove with ocgdb, and I have no idea what I do wrong?

I typed these commands one after the other

Code: Select all

ocgdb -pgn 60_games_with_doubles.pgn -db 60_games_with_doubles.db3 -cpu 4 -o moves
ocgdb -db 60_games_with_doubles.db3 -cpu 4 -dup o printall;remove
ocgdb -pgn 60_games_with_doubles_out.pgn -db 60_games_with_doubles.db3 -cpu 4 -export

ocgdb finds 31 duplicates but it doesn't remove them.

[Jonathan003]

This is the smal example pgn databases with duplicates I used:
https://drive.google.com/file/d/1y56C8E ... sp=sharing

[sarona]

Change line 2 to

ocgdb -db 60_games_with_doubles.db3 -cpu 4 -dup -o printall -o remove

Remember to make a copy of your database before launching the command.

Change line 3 to

ocgdb -pgn 60_games_with_doubles_out.pgn -db 60_games_with_doubles.db3 -cpu 4 -o export

[sarona]

I looked at this more closely once I got home.
ocgdb -db (your_database_name).ocgdb.db3 -cpu 4 -dup --plycount 40 -o printall;remove



Use this to write the database back to pgn (use a different name for the new pgn).
ocgdb -pgn (your_pgn_file) -db (your_database_name).ocgdb.db3 -cpu 4 -export

Sorry for any confusion, Jonathan.

[phhnguyen]

Version Beta 6 released


Changes:
- Fixed bugs
- Store first comments (comments before the first moves) of PGN games
- Search with FENs
- Param -r to write all duplicate games
- Use multi-threads when exporting into a PGN file
- Improve code for getting and displaying games in PGN format
- Improve/clean code

The Position Query Language (PQL) has been improved a bit, thus now users can query with some FEN strings such as:

Code: Select all

ocgdb -db carlsen.ocgdb.db3 -q "fen[rnbqkbnr/pp2pppp/2p5/3pP3/3P4/8/PPP2PPP/RNBQKBNR b KQkq - 0 3, rn1qkbnr/pp2pppp/2p5/3pPb2/3P4/5N2/PPP2PPP/RNBQKB1R b KQkq - 1 4]"

Note that the last two numbers of FENs (halfmove clock and fullmove number) are ignored when comparing. Technically, we convert all FEN strings into hash keys for matching thus all those numbers are discarded automatically.

Look like some people get confused about which duplicates are "true" or "not true" even those games have the same move lists, we add a new parameter -r <file> thus the app can write down all duplicate games into a file and users can verify later.

[Jonathan003]

I looked at this more closely once I got home.
ocgdb -db (your_database_name).ocgdb.db3 -cpu 4 -dup --plycount 40 -o printall;remove



Use this to write the database back to pgn (use a different name for the new pgn).
ocgdb -pgn (your_pgn_file) -db (your_database_name).ocgdb.db3 -cpu 4 -export

Sorry for any confusion, Jonathan.

Thanks I finally got it to work.

I use these 3 lines

Code: Select all

ocgdb -pgn 60_games_with_doubles.pgn -db 60_games_with_doubles.db3 -cpu 4 -o moves
ocgdb -db 60_games_with_doubles.db3 -cpu 4 -dup -o printall;remove
ocgdb -pgn 60_games_with_doubles_out.pgn -db 60_games_with_doubles.db3 -cpu 4 -export

The problem was with line 2 I used the first time.

Code: Select all

ocgdb -db 60_games_with_doubles.db3 -cpu 4 -dup o printall;remove

If I don't type a '-' before the '0' it doesn't work.

I used this as example https://github.com/nguyenpham/ocgdb#:~: ... l%3Bremove

That's why.
These examples also uses the extension .png instead of .pgn for pgn files. I think these are typos because of spell checkers.

[sarona]

Thanks, Pham. I have been using your examples provided and a little bit of trial and error to accomplish things with ocgdb.

I have been attempting to figure out how to save query results to a file with ocgdb. For example: saving all the query results to a seperate pgn file where Q=3, etc.

[Jonathan003]

I'm trying out the new options of ocgdb beta 7.
Was anybody successful detecting duplicates when one game is included in another?
I have tried it with these two games

[Event "Barcelona Ideal Clave op 21st"]
[Site "Barcelona"]
[Date "2017.10.07"]
[Round "2"]
[White "Rojas Nunez, Alberto"]
[Black "Lopez Gomez, Nicolas"]
[Result "0-1"]
[WhiteElo "1276"]
[BlackElo "1526"]
[EventDate "2017.09.30"]
[PlyCount "54"]
[EventType "swiss"]
[EventRounds "8"]
[EventCountry "ESP"]
[SourceTitle "CBM 181 Extra"]
[Source "ChessBase"]
[SourceDate "2017.12.12"]
[SourceVersion "1"]
[SourceVersionDate "2017.12.12"]
[SourceQuality "1"]

1.e4 e5 2.Nf3 Nc6 3.d4 exd4 4.Nxd4 Ne5 5.Nc3 Bb4 6.a3 Bxc3+ 7.bxc3 d6 8.
Bd3 Nf6 9.f4 Nxd3+ 10.cxd3 O-O 11.O-O Re8 12.f5 d5 13.e5 Rxe5 14.Qf3 c5
15.Bf4 Re7 16.Ne2 Bxf5 17.Ng3 Bg4 18.Qf2 b6 19.h3 Bh5 20.Rae1 Bg6 21.Qf3
Qd7 22.c4 Rae8 23.Rd1 d4 24.h4 h6 25.Bxh6 Ng4 26.Bg5 f6 27.Bf4 Ne3 0-1

[Event "Barcelona Ideal Clave op 21st"]
[Site "Barcelona"]
[Date "2017.10.07"]
[Round "2"]
[White "Rojas Nunez, Alberto"]
[Black "Lopez Gomez, Nicolas"]
[Result "0-1"]
[WhiteElo "1276"]
[BlackElo "1526"]
[EventDate "2017.09.30"]
[PlyCount "58"]
[EventType "swiss"]
[EventRounds "8"]
[EventCountry "ESP"]
[SourceTitle "CBM 181 Extra"]
[Source "ChessBase"]
[SourceDate "2017.12.12"]
[SourceVersion "1"]
[SourceVersionDate "2017.12.12"]
[SourceQuality "1"]

1.e4 e5 2.Nf3 Nc6 3.d4 exd4 4.Nxd4 Ne5 5.Nc3 Bb4 6.a3 Bxc3+ 7.bxc3 d6 8.
Bd3 Nf6 9.f4 Nxd3+ 10.cxd3 O-O 11.O-O Re8 12.f5 d5 13.e5 Rxe5 14.Qf3 c5
15.Bf4 Re7 16.Ne2 Bxf5 17.Ng3 Bg4 18.Qf2 b6 19.h3 Bh5 20.Rae1 Bg6 21.Qf3
Qd7 22.c4 Rae8 23.Rd1 d4 24.h4 h6 25.Bxh6 Ng4 26.Bg5 f6 27.Bf4 Ne3 28.h5
Bh7 29.Bxe3 Rxe3 0-1

I used these two commends one after the other

Code: Select all

ocgdb -pgn one_embeded_game.pgn -db one_embeded_game.db3 -cpu 4 -o moves2;discardcomments;discardsites;discardfen;reseteco -plycount 40

Code: Select all

ocgdb -db one_embeded_game.db3 -dup -o embededgames -r report.txt

ocgdb Beta 7 didn't find the embedded double.
Maybe I do something wrong?


Click the image to see it big.
SCID doesn't has problems to detect these count of doubles when the first 4 letters of the player names are the same.


Click the image to see it big