Open Chess Game Database Standard

phhnguyen · Post by **phhnguyen** » Mon Jan 10, 2022 11:00 am

Position searching: 10th attempt

In this attempt, we design and implement a tiny language for position querying. It is tiny since we focus on conditions/expressions only and ignore all other aspects. Users can query other information using SQL queries.

1. The query language
The EBNF (Extended Backus Naur Form) of the language as the below:

Code: Select all

clause = condition { ("and" | "or" | "&&" | "||") condition }
condition = expression { ( "=" | "<" | "<="| " >" | ">=" | "==“ | "!=" | "<>" ) expression }
expression = term  {( "+" | "-" ) term }
term = factor {( "*" | "/" ]) factor} 
factor = number | piece | "(" expression ")"
piece = piecename (<empty> | square | squareset)
piecename = "K" | "Q" | "R" | "B" | "N" | "P" | "k" | "q" | "r" | "b" | "n" | "p" | "white" | "black"
squareset = column | row | "[" (square | squarerange | columnrange | rowrange) { "," (square | squarerange | columnrange | rowrange) } "]"
squarerange = square "-" square
columnrange = column "-" column
rowrange = row "-" row
square = column row 
column = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h"
row = "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8"

Some symbols are mixed between SQL style and C/C++ style (such as users can use both <> and != for not-equal comparison) just for convenience.

A condition/expression may have some chess piece names such as Q (for white Queens), p (for black Pawns). When evaluating, they are counted as cardinalities/total numbers of those chess pieces on the chessboard.

Examples:

Code: Select all

R                the total number of White Rooks
qb3              the total number of Black Queens on square b3
B3               the total number of White Bishops on row 3
bb               the total number of Black Bishops on column b
n[b-e]           the total number of Black Knights from column b to e
P[a4, c5, d5]    the total number of White Pawns on squares a4, c5, and d5

Users can do all calculations between those pieces, numbers and all will turn to conditions (true/false) in the end.

The condition may be implicit or explicit:

Code: Select all

R 		the implicit form of the comparison R != 0
R == 3		the total of White Rooks must be 3
q[5-7] >= 2     the total of Black Queens from row 5 to row 7 must be equal or larger than 2

Some other examples:

Code: Select all

// find all positions having 3 White Queens
Q = 3

// Find all positions having two Black Rooks in the middle squares
r[e4, e5, d4, d5] = 2

// White Pawns in d4, e5, f4, g4, Black King in b7
P[d4, e5, f4, g4] = 4 and kb7

// black Pawns in column b, row 5, from square a2 to d2 must be smaller than 4
p[b, 5, a2 - d2] < 4

// Black Pawns in column c more than 1
pc > 1

// White Pawns in row 3 from 2
P3 >= 2

// Two Bishops in column c, d, e, f
B[c-f] + b[c-f] = 2

// There are 5 white pieces in row 6
white6 = 5

2. Parser

Because the language is very simple and input strings are typically so short, we implement its parser in a simple, straightforward way, using the recursive method. From an input string (query), the parser will create an evaluation tree. That tree will be evaluated at every chess position with the parameters as a set of bitboards of that position. The position and its game will be picked up as the result if the evaluation of the tree is true (not zero).

3. Benchmarks

We query the database which was created from the previous attempt (Attempt 9th). This time we use strings in the above language to query. For example, to query all games with positions having three white Queens, we use the string:

Code: Select all

Q = 3

a. Three White Queens
Query string: “Q = 3”

Code: Select all

Search with query Q = 3...
1. gameId: 13613, fen: r1b3Q1/pp6/1kn5/2bp4/5Q2/4QK2/1pq2P2/5BNR b -  0 23
2. gameId: 185316, fen: r1QQ1r2/p5bk/1p4p1/4p3/4q1p1/6Q1/P6P/R1R3K1 b -  0 35
3. gameId: 203116, fen: 1QQ1rbk1/5ppp/p2q1n2/3p4/4n3/1P1BPr1P/1B3P2/Q1R2RK1 b -  0 34
4. gameId: 286972, fen: 1QQ5/5pk1/5qp1/7p/4B3/1Q3P2/6PP/3R3K b -  0 42
5. gameId: 419965, fen: 1QQ5/4r1pk/Q6p/8/8/7P/6PK/8 b -  0 45
…
50. gameId: 3264750, fen: 2Q3Q1/8/4b3/8/1k6/8/p4K2/Q7 b -  0 65
51. gameId: 3265286, fen: Q2QQR2/6k1/8/8/5KP1/8/8/8 b -  0 71
52. gameId: 3389288, fen: QQ6/8/8/3K2p1/6kp/8/7Q/8 b -  0 63
53. gameId: 3437474, fen: QQ6/8/K7/8/1Q6/8/k7/8 b -  0 94
54. gameId: 3438425, fen: 1QQ2rk1/4q1bp/Q2p1np1/4p3/2P1Pp1P/5P2/5BP1/4KB1R b K  0 28
55. gameId: 3444434, fen: 1Q1QQ3/8/8/5k2/8/8/P7/2K5 b -  0 59
56. gameId: 3445945, fen: 4Q2k/5Q2/p5p1/1p2P3/3P4/P2p2QP/3q4/2r3RK b -  0 48
57. gameId: 3450566, fen: 1Q3Q1Q/6B1/8/1p2p3/4k3/2P1p3/PP2K3/7R b -  0 47

Elapsed: 242118 ms, 04:02, total results: 57, time per results: 4247 ms

b. Two Black Rooks in the middle squares

Query string: “r[e4, e5, d4, d5] = 2”

Code: Select all

1. gameId: 475, fen: 6k1/7p/6p1/2Br1p2/1p2r2P/pP2N1P1/P2K4/8 w -  0 42
2. gameId: 1008, fen: 5bk1/5ppp/p3p3/3rP1P1/Pp1rNP2/1P6/5RKP/4R3 w -  1 31
3. gameId: 1051, fen: 6k1/2q2p1p/6p1/p2r4/1pPr3P/3p4/P2Q1PP1/2R3K1 w -  0 32
4. gameId: 1857, fen: 6k1/2p3p1/p1n2p1p/2nr4/PpN1rP2/1P6/1BP3PP/R3R1K1 w -  0 25
5. gameId: 2927, fen: 5k2/3q1p1Q/3p2p1/p2Pr1P1/2P1r3/2R5/P6P/5R1K w -  0 34
…
8200. gameId: 3452200, fen: 5R2/pp6/2p3k1/4r1pN/3r2P1/1P1P4/P1PK4/8 w -  2 33
8201. gameId: 3452276, fen: 6k1/pp4p1/7p/P2r1P1P/4r1P1/1P6/4K3/R7 w -  0 28
8202. gameId: 3452545, fen: 6k1/6pp/5p1q/3rp3/4r3/1P6/P1P2P2/3KR3 w -  0 34
8203. gameId: 3452636, fen: 2k5/1p6/8/p2r2PR/4r3/1KP1p3/PP6/4R3 w -  0 32
8204. gameId: 3452998, fen: 8/2pb4/3p1k1p/pPpPrp2/P1P1rN2/2R1R1PP/5K2/8 w -  2 52
8205. gameId: 3453043, fen: 5nk1/5p2/4p1p1/3rP2q/pp1r4/4QRPP/P4PK1/1B2R3 w -  1 32
8206. gameId: 3453642, fen: 8/1R3K1k/8/3r4/3r2BP/2n5/8/2B5 w -  10 63
8207. gameId: 3454480, fen: 6k1/1p1n1ppp/2p5/p1PprP2/N3r3/PP2P3/5KPP/R3R3 w -  1 23
8208. gameId: 3455477, fen: 6k1/2p2pp1/p6p/1p1rr3/1Bb5/P5NP/1PBb1PP1/R2R2K1 w -  4 28
8209. gameId: 3455554, fen: 6k1/p4ppp/5n2/1pqrr1N1/2p4P/8/PPQ2PP1/R2R2K1 w -  0 26
8210. gameId: 3456127, fen: 8/5k2/8/3r4/3r1p2/8/2K4Q/8 w -  14 95
8211. gameId: 3456227, fen: 4q3/5pk1/5np1/1p1pr3/p2r2P1/P1N1PP2/1P2R1K1/4Q3 w -  0 50
8212. gameId: 3456229, fen: 8/4qppk/5p1p/3rrP1P/6P1/3R3K/P2Q4/1R6 w -  0 53

Elapsed: 253891 ms, 04:13, total results: 8212, time per results: 30 ms

c. Three White Pawns in d4, e5, f4, g4, Black King in b7

Search with query: “P[d4, e5, f4, g4] = 4 and kb7”

Code: Select all

1. gameId: 841, fen: 8/1k4pp/p1p1pp2/4P3/PK1P1PP1/7P/8/8 w -  1 35
2. gameId: 6425, fen: 3r1n1r/pk1qnpp1/1pp1p2p/3pP2P/NB1P1PP1/P2Q3R/1PP5/2KR4 w -  1 18
3. gameId: 26939, fen: 4r3/pk2npp1/q1p1p2p/2BpP2P/3P1PP1/2P5/P1QK4/R7 w -  0 25
4. gameId: 28376, fen: 8/pk3p2/R1n1p1p1/3pP3/2pP1PPp/2P2B1P/2P5/1rBK1n2 w -  3 52
5. gameId: 33063, fen: 1r5r/1k2np2/1pp1p1p1/p2pP3/P1nP1PPp/2PB3P/2P2B2/RR2K3 w -  1 27
6. gameId: 35027, fen: r6r/pk2nppq/1pn1p2p/3pP2P/b1pP1PPN/P1P1B3/R1PKB3/3Q3R b - f3 0 18
7. gameId: 36328, fen: r1r5/1k1q1p2/1p2p1p1/3pP2p/bn1P1PP1/2p1NB1P/2P2K2/R2Q3R b -  0 42
8. gameId: 39399, fen: 3r1b1r/pkpq2pp/1p2p2n/n2pPp2/3P1PP1/P1NQ3P/1PPBN3/2K3RR b - g3 0 15
9. gameId: 45124, fen: 4r3/1k2nrp1/2b1p1Np/1p1pP2P/p1pP1PP1/P6R/1PB2R2/6K1 b -  0 43
10. gameId: 50501, fen: 3r3r/pkqn1p2/2n1p1pp/1p1pP3/2pP1PP1/2P2N2/PPNQ3P/1K1R3R b - g3 0 20
…
470. gameId: 3316120, fen: 4q1rr/pkn1npbp/1pp1pBp1/3pP3/3P1PP1/1NPB1Q1P/PP6/2KR3R w -  1 19
471. gameId: 3319876, fen: 8/1k2np2/1rp1p1pp/3pP3/P2P1PP1/KN6/2R4P/8 w -  2 36
472. gameId: 3333216, fen: 3r3r/1knnbp2/qp2p1p1/2ppP3/p2P1PP1/2P1BNN1/PP2Q1K1/R4R2 w -  1 19
473. gameId: 3359102, fen: 5r2/1k3p1p/p1p1p1p1/1nP1P3/1NpP1PP1/7P/8/KR6 w -  1 36
474. gameId: 3374676, fen: 8/1k3p2/4p3/p2pP3/p2P1PP1/4K3/Nb6/8 b - g3 0 44
475. gameId: 3381812, fen: 4r1b1/1kp3Kp/2p3p1/B1RpPp2/3P1PP1/P6P/8/8 b -  0 51
476. gameId: 3386119, fen: 7r/pk1b1ppr/1pn1p1p1/3pP3/q1pP1PP1/P1P2R1P/1QPB1R2/5BK1 b - g3 0 23
477. gameId: 3390871, fen: 3r1b1r/pk1qn1p1/1pn1p2p/3pPp1P/3P1PP1/5N1Q/PP1BN1K1/2R4R w -  1 18
478. gameId: 3402434, fen: 3rn2r/pk1q1pbp/1pp1p1p1/3pP3/3P1PP1/3RQNNP/PPP5/2K4R w -  2 20
479. gameId: 3408344, fen: 1rnq3r/pk1b1pp1/1p2p2p/n2pP2P/PBpP1PPN/2P5/2P1B3/R1Q1K2R w KQ  1 17
480. gameId: 3411214, fen: r4q2/bk2nr2/1pp3pp/3pP3/PP1P1PP1/2NQ1RKP/8/2R5 w -  3 29
481. gameId: 3432729, fen: 6n1/1k3pbp/6p1/p1BpP3/P2P1PP1/7P/8/N5K1 w -  1 33
482. gameId: 3444804, fen: 8/pk3ppp/1p2p3/3pP3/P1rP1PPP/R2K4/8/8 w -  0 32
483. gameId: 3452950, fen: 3r1b1r/pkqbnpp1/1p2p2p/n2pP2P/2pP1PP1/P1P1B2B/1P1NN3/R2QK2R w KQ  1 15

Elapsed: 249175 ms, 04:09, total results: 483, time per results: 515 ms

d. There are two Bishops (any side) from column c to f
Search string: “B[c-f] + b[c-f] == 2”

Code: Select all

1. gameId: 3, fen: rn1q1rk1/ppp1ppbp/3p1np1/8/3PPBb1/2N2N2/PPPQ1PPP/R3KB1R w KQ  1 6
2. gameId: 2, fen:  r2qkb1r/pp3ppp/1n1p2b1/1BPPp3/1P1N2P1/5P1P/P7/R1BQK2R b KQkq  0 16
3. gameId: 4, fen: r2q1rk1/1bp1pp1p/p1n2bp1/1p1B4/3P4/2N1P3/PP3PPP/R2QK1NR w KQ  1 11
4. gameId: 1, fen: r4rk1/pb2bppp/8/2ppP3/8/2P1BN2/P4PPP/R4RK1 w -  0 15
5. gameId: 5, fen: rnNq3r/1p1pkppp/p3pn2/8/1bP5/2N5/PP2PPPP/R1BQKB1R b KQ  0 8
7. gameId: 7, fen: 6. gameId: 6, fen: r1b2rk1/pp1nqpbp/2pp1npB/4p3/3PP3/2NB3P/PPPQNPP1/R4RK1 b -  1 10
…
3326756. gameId: 3456396, fen: r2qk2r/bpp2ppp/p2pbn2/8/4P3/P2N1P2/1PPPN1PP/R1BQK2R w KQkq  1 11
3326757. gameId: 3456397, fen: rnb1k2r/pp2npb1/2pp1qpp/4p3/4P3/1BNP1Q1P/PPP2PP1/R1B1K1NR b KQkq  0 9
3326758. gameId: 3456398, fen: r2qkb1r/pbppnpp1/1p5p/3NN3/2B1P3/P2P4/1PP2PPP/R2QK2R w KQkq  0 10
3326759. gameId: 3456393, fen: r2q1rk1/ppp2pp1/2np3p/2b5/4PQ2/2N2NP1/PP3Bb1/R3K3 w Q  0 16
3326760. gameId: 3456399, fen: 1r4r1/2b1k2p/2Pp1pbn/1p1Pp1p1/pP2P1P1/3KBP2/N6P/RN4R1 w -  1 30

Elapsed: 190605 ms, 03:10, total results: 3326760, time per results: 0 ms

This query runs significantly faster than other queries event it has much more results. That is caused by positions typically matched early in the games, leading the move parsers to stop early too (without parsing the rest).

e. Find all games with 5 White pieces on row 6

Search string: “white6 = 5”

Code: Select all

1. gameId: 139197, fen: 8/4q2k/BR1QPbpP/4p3/4P3/4B2r/2KN2n1/8 b -  0 43
2. gameId: 229745, fen: 5b2/p1k2Pp1/P1P1KPPp/3B3P/8/8/8/8 b -  0 46
3. gameId: 410414, fen: 8/8/1PBKPR2/8/3k4/3r4/4r3/8 b -  0 77
4. gameId: 416799, fen: 5q1k/p1r5/P2pBNQP/3P4/5K2/8/8/8 b -  5 56
5. gameId: 448981, fen: 3r3r/4k2p/PPR1B2P/6K1/6P1/8/8/8 b -  0 76
…
50. gameId: 3173451, fen: 2k5/6pp/PPPPK3/8/8/7P/6P1/8 b -  5 67
51. gameId: 3176339, fen: 4b3/2k1P3/R2PBKpP/5p2/5P2/8/7r/8 b -  0 62
52. gameId: 3180607, fen: 1rb2r2/8/ppPPnNQR/5kP1/P3pp2/1Nq5/8/1R4K1 b -  2 43
53. gameId: 3270729, fen: r5k1/8/PK2NRP1/7p/7P/6P1/8/8 b -  0 65
54. gameId: 3276938, fen: n5rk/q4r1p/B1RRQ1pB/p1p1p2n/Pp2P2P/1P4P1/2P2P2/6K1 b -  3 43
55. gameId: 3325056, fen: 3k4/5B1b/1PKPP2P/5p2/5P2/8/8/8 b -  0 67
56. gameId: 3329210, fen: r5k1/8/PR2PPK1/8/8/8/8/8 b -  1 58
57. gameId: 3438015, fen: r3k1r1/1q6/bPpR1PBQ/2Pp4/P2P3P/6P1/8/6K1 b -  2 42

Elapsed: 283500 ms, 04:43, total results: 57, time per results: 4973 ms

4. Quick conclusions for this attempt
- All results are identical to the previous attempt
- As fast as the previous attempt (9th)
- Even though the language is tiny and so simple it can query almost all kinds of position questions
- Due to the simpleness, we believe users won’t have much trouble learning and using this language

The query language and parser work so well!

5. Source code
All code of this Attempt has been pushed already into a new branch “searchwithtextparser”.

Harald · Post by **Harald** » Mon Jan 10, 2022 2:12 pm

Can you include a kind of common table expression like a WITH statement in SQL in your query parser?
Or is that already part of the "and" and "or" handling?
Or anywhere else in the query method?
Or is it allowed to search on the previous result list(s)?
This could improve the speed when an easy expression is evaluated fast and first
that reduces the number of positions for the more complex queries.

phhnguyen · Post by **phhnguyen** » Fri Jan 14, 2022 12:58 pm

Harald wrote: ↑Mon Jan 10, 2022 2:12 pm Can you include a kind of common table expression like a WITH statement in SQL in your query parser?
Or is that already part of the "and" and "or" handling?
Or anywhere else in the query method?
Or is it allowed to search on the previous result list(s)?
This could improve the speed when an easy expression is evaluated fast and first
that reduces the number of positions for the more complex queries.

I am so confused about how that statements could help to speed up. Can you give some examples of how they can help?

If you could, modify the code and test yourself. When searching, I use a very simple statement like the below:

Code: Select all

SELECT * FROM Games

phhnguyen · Post by **phhnguyen** » Sat Jan 15, 2022 12:28 am

Position searching: 11th attempt

In the previous attempt (10th), we insert into the database all moves of a game as a single string and save them in a text field named Moves. It is actually the text moves-part of that game from the PGN file. By doing that way we can convert a PGN file into a SQL database at a very high speed. A database of 3.45 million games could be completed within a minute. We saved time by not converting the move string into a chessboard.

However, when searching, we can’t avoid and have to parse the move string into a chessboard for each game. That processing takes time since it has to work with strings, split them, convert moves from text SAN format into binary one then make those moves.

We may do some exchange: do some more computing when creating databases to save them from searching periods. The converting is done once only but using/searching may be done many times thus that could save a lot of time in general.

In this attempt, when creating the database we parse the text of moves into the chessboard then store them directly in binary forms into the database, and then measure how fast it is when searching.

A. Building the database

A chess move could be encoded into one or two bytes. We try both to see how much data we could save. All move data of a game will be stored in the database as a blob - the only form to store the binary data. From the experience of a previous attempt (when we try to store all hash keys), we now feel a bit “scared” to work with blobs when they are huge in numbers and/or large sizes. In that attempt sometimes the speed was slowed down a lot to crawl and be unacceptable. We will try to see it anyway.

1. Two bytes, add a new field for move blobs
A move could be encoded easily into 2 bytes. We used a simple and straightforward way: 6 bits for each square (from and dest), 3 bits for promotion. For being convenient for the majority of programmers, those squares are auto-converted into Stockfish coordination.

Code: Select all

movedata = from | dest << 6 | promotion << 12;

We add a new field named “Moves2” (2 means 2 bytes) with blob type.

Below are stats for creating:

Code: Select all

#games: 3456399, elapsed: 253447 ms 04:13, speed: 13637 games/s

The good news is that the speed is totally “normal”. It took 4:13 minutes, just a bit longer than Attempt 9 (3:49) - very reasonable since it needs to do some extra work, convert moves and save to blobs. Look like saving to blobs doesn’t cause any serious slowdown.

The database size is 2.65 GB, larger than the one in the previous attempt (vs 2.13 GB, an increase of 0.53 GB or 24%). Not small but not too much either.

However, this time we’re not happy with the creating time. The current code to convert a string into a move is taken from our other project (from Banksia GUI). It checks the move very carefully by generating all moves first and then finding the match. The code also creates the SAN string for that move to use later (the generation of all moves in the previous step actually helps this step to create the SAN string). All works well but not at the highest speed. That was a known problem for a while, but we ignored for focusing on other tasks. Now we improve the function to check without generating moves nor SAN strings.

The new stats as the below:

Code: Select all

#games: 3456399, elapsed: 127287 ms 02:07, speed: 27154 games/s

Good! Almost 2 times faster for the converter (from PGN to SQLite database).

2. Two bytes, remove move strings

If we use move blobs, the move text in the field Moves may become redundant. We remove that field to save space. All move comments are stored on a new table Comments with GameID and PlyAt thus we can find them when needed.

Below are stats:

Code: Select all

#games: 3456399, elapsed: 146434 ms 02:26, speed: 23603 games/s

The database size is now 826 MB only, 2.6 times smaller than the previous attempt (vs 2.13 GB). The time is 2.26 minutes, similar to the above case.

3. One (or 1.5) byte

We got the idea of using 1-byte move encoding from some discussions of SCID teams. Actually, not all moves could be encoded into 1 byte but moves of Queens must use 2 bytes. Thus we may call it 1.5 bytes. We have known this method for a while but did not use it since the algorithm/code to encode and decode is quite complicated, hard to maintain, and hard to use for other chess variants when the benefit maybe not be much.

BTW, this time we have implemented it anyway, to get enough information and as we mentioned, to push all boundaries as far as we could. The move data is stored in the new field named Moves1 (1 means 1 byte). As we have guessed, those functions are terrible buggy when developing. Because of being complicated, those squares are still kept in our coordination, NOT converted into Stockfish coordination for this attempt.

The new stats are as the below:

Code: Select all

#games: 3456399, elapsed: 135459ms 02:15, speed: 25516 games/s

Even the move encoding is more complicated but the building time is the same with the above case (2 bytes). The database size is receded into 553 MB. So far the smallest size, isn’t it?

Compare to the SCID database of the same games, our database is a bit bigger (vs 509 MB of SCID). However, IMHO, now it is clearly comparable and at the same level!!!

B. Searching

We improve the code for searching so now it can auto-detect if there is an encoded move field and how many bytes of that encoding. That is done by checking if the table Games has field Moves, Moves1, Moves2. The search is auto-adjusted by detection of those fields and can work if there are any of them.

1. Move encoded using 2 bytes

Search with query: Q = 3

Code: Select all

1. gameId: 13612, fen: r1b3Q1/pp6/1kn5/2bp4/5Q2/4QK2/1pq2P2/5BNR b -  0 23
2. gameId: 185317, fen: r1QQ1r2/p5bk/1p4p1/4p3/4q1p1/6Q1/P6P/R1R3K1 b -  0 35
3. gameId: 203113, fen: 1QQ1rbk1/5ppp/p2q1n2/3p4/4n3/1P1BPr1P/1B3P2/Q1R2RK1 b -  0 34
4. gameId: 286972, fen: 1QQ5/5pk1/5qp1/7p/4B3/1Q3P2/6PP/3R3K b -  0 42
5. gameId: 419967, fen: 1QQ5/4r1pk/Q6p/8/8/7P/6PK/8 b -  0 45
…
54. gameId: 3438420, fen: 1QQ2rk1/4q1bp/Q2p1np1/4p3/2P1Pp1P/5P2/5BP1/4KB1R b K  0 28
55. gameId: 3444432, fen: 1Q1QQ3/8/8/5k2/8/8/P7/2K5 b -  0 59
56. gameId: 3445945, fen: 4Q2k/5Q2/p5p1/1p2P3/3P4/P2p2QP/3q4/2r3RK b -  0 48
57. gameId: 3450570, fen: 1Q3Q1Q/6B1/8/1p2p3/4k3/2P1p3/PP2K3/7R b -  0 47

Elapsed: 40047 ms, 00:40, total results: 57, time per results: 702 ms

Search with query: r[e4, e5, d4,d5]= 2...

Code: Select all

1. gameId: 476, fen: 6k1/7p/6p1/2Br1p2/1p2r2P/pP2N1P1/P2K4/8 w -  0 42
2. gameId: 1010, fen: 5bk1/5ppp/p3p3/3rP1P1/Pp1rNP2/1P6/5RKP/4R3 w -  1 31
3. gameId: 1054, fen: 6k1/2q2p1p/6p1/p2r4/1pPr3P/3p4/P2Q1PP1/2R3K1 w -  0 32
4. gameId: 1857, fen: 6k1/2p3p1/p1n2p1p/2nr4/PpN1rP2/1P6/1BP3PP/R3R1K1 w -  0 25
5. gameId: 2922, fen: 5k2/3q1p1Q/3p2p1/p2Pr1P1/2P1r3/2R5/P6P/5R1K w -  0 34
…
8208. gameId: 3455479, fen: 6k1/2p2pp1/p6p/1p1rr3/1Bb5/P5NP/1PBb1PP1/R2R2K1 w -  4 28
8209. gameId: 3455555, fen: 6k1/p4ppp/5n2/1pqrr1N1/2p4P/8/PPQ2PP1/R2R2K1 w -  0 26
8210. gameId: 3456126, fen: 8/5k2/8/3r4/3r1p2/8/2K4Q/8 w -  14 95
8211. gameId: 3456227, fen: 8/4qppk/5p1p/3rrP1P/6P1/3R3K/P2Q4/1R6 w -  0 53
8212. gameId: 3456229, fen: 4q3/5pk1/5np1/1p1pr3/p2r2P1/P1N1PP2/1P2R1K1/4Q3 w -  0 50

Elapsed: 40955 ms, 00:40, total results: 8212, time per results: 4 ms

Search with query: P[d4, e5, f4, g4] = 4 and kb7...

Code: Select all

1. gameId: 840, fen: 8/1k4pp/p1p1pp2/4P3/PK1P1PP1/7P/8/8 w -  1 35
2. gameId: 6425, fen: 3r1n1r/pk1qnpp1/1pp1p2p/3pP2P/NB1P1PP1/P2Q3R/1PP5/2KR4 w -  1 18
3. gameId: 26937, fen: 4r3/pk2npp1/q1p1p2p/2BpP2P/3P1PP1/2P5/P1QK4/R7 w -  0 25
4. gameId: 28370, fen: 8/pk3p2/R1n1p1p1/3pP3/2pP1PPp/2P2B1P/2P5/1rBK1n2 w -  3 52
5. gameId: 33062, fen: 1r5r/1k2np2/1pp1p1p1/p2pP3/P1nP1PPp/2PB3P/2P2B2/RR2K3 w -  1 27
…
478. gameId: 3402429, fen: 3rn2r/pk1q1pbp/1pp1p1p1/3pP3/3P1PP1/3RQNNP/PPP5/2K4R w -  2 20
479. gameId: 3408346, fen: 1rnq3r/pk1b1pp1/1p2p2p/n2pP2P/PBpP1PPN/2P5/2P1B3/R1Q1K2R w KQ  1 17
480. gameId: 3411214, fen: r4q2/bk2nr2/1pp3pp/3pP3/PP1P1PP1/2NQ1RKP/8/2R5 w -  3 29
481. gameId: 3432730, fen: 6n1/1k3pbp/6p1/p1BpP3/P2P1PP1/7P/8/N5K1 w -  1 33
482. gameId: 3444805, fen: 8/pk3ppp/1p2p3/3pP3/P1rP1PPP/R2K4/8/8 w -  0 32
483. gameId: 3452951, fen: 3r1b1r/pkqbnpp1/1p2p2p/n2pP2P/2pP1PP1/P1P1B2B/1P1NN3/R2QK2R w KQ  1 15

Elapsed: 43317 ms, 00:43, total results: 483, time per results: 89 ms

Search with query: white6 = 5

Code: Select all

1. gameId: 139193, fen: 8/4q2k/BR1QPbpP/4p3/4P3/4B2r/2KN2n1/8 b -  0 43
2. gameId: 229748, fen: 5b2/p1k2Pp1/P1P1KPPp/3B3P/8/8/8/8 b -  0 46
3. gameId: 410414, fen: 8/8/1PBKPR2/8/3k4/3r4/4r3/8 b -  0 77
4. gameId: 416800, fen: 5q1k/p1r5/P2pBNQP/3P4/5K2/8/8/8 b -  5 56
5. gameId: 448981, fen: 3r3r/4k2p/PPR1B2P/6K1/6P1/8/8/8 b -  0 76
…
54. gameId: 3276934, fen: n5rk/q4r1p/B1RRQ1pB/p1p1p2n/Pp2P2P/1P4P1/2P2P2/6K1 b -  3 43
55. gameId: 3325055, fen: 3k4/5B1b/1PKPP2P/5p2/5P2/8/8/8 b -  0 67
56. gameId: 3329210, fen: r5k1/8/PR2PPK1/8/8/8/8/8 b -  1 58
57. gameId: 3438018, fen: r3k1r1/1q6/bPpR1PBQ/2Pp4/P2P3P/6P1/8/6K1 b -  2 42

Elapsed: 42020 ms, 00:42, total results: 57, time per results: 737 ms

All results are correct and the search time (under 50 seconds) is significantly faster than the previous Attempt (above 4 minutes per query).

2. Move encoded using 1 byte

Search with query: Q = 3

Code: Select all

1. gameId: 13615, fen: r1b3Q1/pp6/1kn5/2bp4/5Q2/4QK2/1pq2P2/5BNR b -  0 23
2. gameId: 185317, fen: r1QQ1r2/p5bk/1p4p1/4p3/4q1p1/6Q1/P6P/R1R3K1 b -  0 35
3. gameId: 203113, fen: 1QQ1rbk1/5ppp/p2q1n2/3p4/4n3/1P1BPr1P/1B3P2/Q1R2RK1 b -  0 34
4. gameId: 286973, fen: 1QQ5/5pk1/5qp1/7p/4B3/1Q3P2/6PP/3R3K b -  0 42
5. gameId: 419965, fen: 1QQ5/4r1pk/Q6p/8/8/7P/6PK/8 b -  0 45
…
54. gameId: 3438421, fen: 1QQ2rk1/4q1bp/Q2p1np1/4p3/2P1Pp1P/5P2/5BP1/4KB1R b K  0 28
55. gameId: 3444435, fen: 1Q1QQ3/8/8/5k2/8/8/P7/2K5 b -  0 59
56. gameId: 3445945, fen: 4Q2k/5Q2/p5p1/1p2P3/3P4/P2p2QP/3q4/2r3RK b -  0 48
57. gameId: 3450570, fen: 1Q3Q1Q/6B1/8/1p2p3/4k3/2P1p3/PP2K3/7R b -  0 47

Elapsed: 45091 ms, 00:45, total results: 57, time per results: 791 ms

Search with query: r[e4, e5, d4,d5]= 2

Code: Select all

1. gameId: 475, fen: 6k1/7p/6p1/2Br1p2/1p2r2P/pP2N1P1/P2K4/8 w -  0 42
2. gameId: 1010, fen: 5bk1/5ppp/p3p3/3rP1P1/Pp1rNP2/1P6/5RKP/4R3 w -  1 31
3. gameId: 1054, fen: 6k1/2q2p1p/6p1/p2r4/1pPr3P/3p4/P2Q1PP1/2R3K1 w -  0 32
4. gameId: 1858, fen: 6k1/2p3p1/p1n2p1p/2nr4/PpN1rP2/1P6/1BP3PP/R3R1K1 w -  0 25
5. gameId: 2922, fen: 5k2/3q1p1Q/3p2p1/p2Pr1P1/2P1r3/2R5/P6P/5R1K w -  0 34
…
8208. gameId: 3455479, fen: 6k1/2p2pp1/p6p/1p1rr3/1Bb5/P5NP/1PBb1PP1/R2R2K1 w -  4 28
8209. gameId: 3455553, fen: 6k1/p4ppp/5n2/1pqrr1N1/2p4P/8/PPQ2PP1/R2R2K1 w -  0 26
8210. gameId: 3456126, fen: 8/5k2/8/3r4/3r1p2/8/2K4Q/8 w -  14 95
8211. gameId: 3456228, fen: 8/4qppk/5p1p/3rrP1P/6P1/3R3K/P2Q4/1R6 w -  0 53
8212. gameId: 3456229, fen: 4q3/5pk1/5np1/1p1pr3/p2r2P1/P1N1PP2/1P2R1K1/4Q3 w -  0 50

Elapsed: 47155 ms, 00:47, total results: 8212, time per results: 5 ms

Search with query: P[d4, e5, f4, g4] = 4 and kb7

Code: Select all

1. gameId: 840, fen: 8/1k4pp/p1p1pp2/4P3/PK1P1PP1/7P/8/8 w -  1 35
2. gameId: 6427, fen: 3r1n1r/pk1qnpp1/1pp1p2p/3pP2P/NB1P1PP1/P2Q3R/1PP5/2KR4 w -  1 18
3. gameId: 26937, fen: 4r3/pk2npp1/q1p1p2p/2BpP2P/3P1PP1/2P5/P1QK4/R7 w -  0 25
4. gameId: 28370, fen: 8/pk3p2/R1n1p1p1/3pP3/2pP1PPp/2P2B1P/2P5/1rBK1n2 w -  3 52
5. gameId: 33062, fen: 1r5r/1k2np2/1pp1p1p1/p2pP3/P1nP1PPp/2PB3P/2P2B2/RR2K3 w -  1 27
…
480. gameId: 3411212, fen: r4q2/bk2nr2/1pp3pp/3pP3/PP1P1PP1/2NQ1RKP/8/2R5 w -  3 29
481. gameId: 3432728, fen: 6n1/1k3pbp/6p1/p1BpP3/P2P1PP1/7P/8/N5K1 w -  1 33
482. gameId: 3444806, fen: 8/pk3ppp/1p2p3/3pP3/P1rP1PPP/R2K4/8/8 w -  0 32
483. gameId: 3452951, fen: 3r1b1r/pkqbnpp1/1p2p2p/n2pP2P/2pP1PP1/P1P1B2B/1P1NN3/R2QK2R w KQ  1 15

Elapsed: 48480 ms, 00:48, total results: 483, time per results: 100 ms

Search with query: white6 = 5

Code: Select all

1. gameId: 139194, fen: 8/4q2k/BR1QPbpP/4p3/4P3/4B2r/2KN2n1/8 b -  0 43
2. gameId: 229747, fen: 5b2/p1k2Pp1/P1P1KPPp/3B3P/8/8/8/8 b -  0 46
3. gameId: 410415, fen: 8/8/1PBKPR2/8/3k4/3r4/4r3/8 b -  0 77
4. gameId: 416800, fen: 5q1k/p1r5/P2pBNQP/3P4/5K2/8/8/8 b -  5 56
5. gameId: 448981, fen: 3r3r/4k2p/PPR1B2P/6K1/6P1/8/8/8 b -  0 76
…
54. gameId: 3276933, fen: n5rk/q4r1p/B1RRQ1pB/p1p1p2n/Pp2P2P/1P4P1/2P2P2/6K1 b -  3 43
55. gameId: 3325055, fen: 3k4/5B1b/1PKPP2P/5p2/5P2/8/8/8 b -  0 67
56. gameId: 3329210, fen: r5k1/8/PR2PPK1/8/8/8/8/8 b -  1 58
57. gameId: 3438017, fen: r3k1r1/1q6/bPpR1PBQ/2Pp4/P2P3P/6P1/8/6K1 b -  2 42

Elapsed: 47736 ms, 00:47, total results: 57, time per results: 837 ms

All results are correct and the search time is under 50 seconds, quite similar to above (2 bytes).

C. Quick conclusions for this attempt
- We can encode chess moves and store them in ready-use binary form. The converting time is very reasonable
- The encoding chess moves could use 2 or 1.5 bytes. All methods have the same speeds for both converting and searching
- Using binary moves could be smaller in size and speed up searching significantly. All queries could be completed within 1 minute, 4 times faster than the previous Attempt

We have got a new achievement: the database size, converting, and searching speeds are totally comparable/very close to SCID - the fastest chess game binary database app!

D. Source code
All code of this Attempt has been pushed already into a new branch “moveblob”.

---
We have almost all the information we need for the decision of the standard. We still want to do a few more attempts but we are quite close to a release. If you want to join, contribute and/or give suggestions, ideas, opinions… before the first official release (version 1) please join/do now.

phhnguyen · Post by **phhnguyen** » Fri Jan 21, 2022 11:00 am

Position searching: 12th attempt

One of the questions that concerned us most is the limit of our databases, which size it could reach, what problems we may have when creating very large databases. So far we have run the program mainly with a PGN file of 3.45 million games (Millibase). That database is good on both quality and quantity: games are mainly from high-level human players, the number of games is quite popular. However, we don’t know what could happen when working with larger numbers of games and other types of players, say chess engines.

On the technical side, there are some things still worrying us and we need answers:
- We don’t hold main data, whenever we get new players, events, games… we write down them all into the database. However, we use only one transaction for the whole database. It started at the beginning and committed at the end when all work was completed. We worry if the transaction keeps all data in the memory, it may eat a lot of memory when creating a huge database
- We keep strings of players, events, sites ourselves in memory with their indexes. That helps a lot for matching to find IDs. However, that may take a lot of memory if their numbers are huge

In this attempt, we work with a very large PGN database downloaded from the Lichess website. The file is lichess_db_standard_rated_2021-08.pgn, 216 GB after uncompressing, containing near 94 million games. That is about 90 times larger in size and 27 times larger on the number of games, compared with our typical test database (2.42 GB, 3.45 million games). In this database, a lot of players are chess engines.

A. Building the database

Test 1
We run the program as it is from the previous attempt (Attempt 11th). The code parses each game, makes all moves then save those moves into the database as a blob.

Below are stats when creating:

Code: Select all

#games: 61502, elapsed: 3817ms 00:03, speed: 16112 games/s
…
#games: 3369247, elapsed: 339003ms 05:39, speed: 9938 games/s
…
#games: 17978488, elapsed: 23329429ms 6:28:49, speed: 770 games/s
…
#games: 35417020, elapsed: 91215314ms 25:20:15, speed: 388 games/s
#games: 35473988, elapsed: 92109326ms 25:35:09, speed: 385 games/s

After more than 25 hours the program converted over 35 million games. However, there was a serious problem: the converting speed slowdown constantly. With that slowdown speed, it may take weeks, months, or even completely stuck forever later. We could not wait for that long anyway and stopped the program.

Status: FAILED

Problems and solutions

We have investigated the input data and realized the main causes:

a) Site tag
Lichess databases used unique URLs for the tag Site in their games. Those URLs are actually URIs (Uniform Resource Identifier). For example, here is one of the URLs:

Code: Select all

https://lichess.org/study/QQWjey3K/HIvmSIiZ

Those URLs equal the total number of games. 94 million games mean 94 million different URLs. Typically we store Site-tag contents in a separate table named Sites and it should have 94 million records accordingly.

b) Comments
Almost all moves of all games in that Lichess database have comments. They are mostly auto-generated by computers as the follows:

Code: Select all

{ [%eval 0.24] [%clk 0:10:00] }

When creating databases with binary-encoded moves using blobs, we take comments out and stored them in the table Comments thus we could generate games with full comments later. In the Lichess database, 94 million games have totally about 6.2 billion moves and a similar number of comments.

Both a) and b) require to write down too many times to different tables. a) requires to write 94 million times and b) requires 6.2 billion times (crazy huge number - we should avoid writing down with that number of times anyway). We must write 94 million times to the table Games too. Writing too many into a few tables causes that slowdown.

There are some solutions:
- For (a) we could change the tag Site into a new one named Source and create a new field with that name in the table Games for that. Now when writing to the table Games we could write into that extra column at the same time. We don’t need to write into the table Site anymore
- For b) we could store moves including their comments in column Moves (text form) thus we could avoid inserting into the table Comments
- Discard Sites and comments. For general purposes, that information of Sites and comments of Lichess are not very useful and we may simply discard them all. By doing that we can save a lot of time to write down as well as space. We believe it is the best solution for general game databases of Lichess

Baseline: measure ready time
We run a very basic step: let the program parse all games without parsing moves nor store anything into the database. Simply it just counts the total of games and measures the time to read and parse all game headers.

Code: Select all

…
#games: 93679549, elapsed: 3552480ms 59:12, speed: 26370 games/s, #blocks: 25777, processed size: 206208 MB

The PGN file has near 94 million games. The converter (now works as a simple reader) spends over 1-hour reading and parsing games (without parsing moves).

Baseline: measure move parsing time
Similar to the above baseline, we parse all moves. The program doesn’t store anything in the database.

Code: Select all

#games: 93679646, elapsed: 3480139ms 58:00, speed: 26918 games/s, #blocks: 25777, processed size: 206208 MB

The program took a similar time as the above measurement. That amazing us since this time we parse all moves which requires much more computing/power. The result could be explained that the reading process took a significantly longer time thus threads could complete their parsing tasks before the next tasks be available.

Test 2
Build database, change Site to Source, store moves in the field Moves (as text) only.

Code: Select all

#games: 93679650, elapsed: 25066985ms 6:57:46, speed: 3737 games/s, #blocks: 25777, processed size: 206208 MB

The program spent near 7 hours converting the whole Lichess database of 94 million games. The SQLite database size is 254 GB, larger than the size of the original PGN file.

The good news is that we converted the database of that size with almost no trouble. The program took not much both computing and memory. Our computer always showed green status all the time converting.

Our worrying about transaction/memory did not become real either (hmm, actually I have big trouble cleaning up my computer to have enough disk space to store some created databases from this attempt

).

The numbers of name strings of players, events, sites are not troubling either. There are 1.6 million players with the total of their string lengths is 17 million. We estimated the program may use about 30 MB to store those strings, indexes, and some extra management information. That is just a small amount of memory in modern computers. Other names such as Events have a much smaller number and take much less memory.

BTW, later we changed the way to call transactions. The program now creates and commit transactions frequently, after a while. That may improve the performance a little bit.

Status: SUCCESSFUL

Test 3
We try the extreme way: convert the database with binary for moves, using 2-byte encoding, discard all information of Site tag and all comments.

Code: Select all

#games: 93679650, elapsed: 5209214ms 1:26:49, speed: 17983 games/s, #blocks: 25777, processed size: 206208 MB

The program runs quite fast now, taking about only 1 hour 26 minutes. The database size is amazingly small at 19.8 GB.

Status: SUCCESSFUL

Test 4
We try the extreme way similar to Test 3 but using 1-byte encoding. All information on the Site tag and all comments are discarded too.

Code: Select all

#games: 93679650, elapsed: 4958538ms 1:22:38, speed: 18892 games/s, #blocks: 25777, processed size: 206208 MB

The program runs quite fast too, taking about only 1 hour 22 minutes. The database size is amazingly small at 13.7 GB.

Look like that size doesn’t make much trouble to store and process as the original PGN file or the one from test 2 (over 200 GB each).

Status: SUCCESSFUL

B. Position searching

Search 1
We test with the database from Test 2 (field Moves, the database size is 254 GB).

Search with query: Q = 3

Code: Select all

1. gameId: 3648, fen: 8/8/2Q5/4K3/3Q3Q/6k1/8/8 b -  5 81
2. gameId: 4017, fen: 6Q1/8/5Q2/1k2Q3/pp6/8/8/7K b -  1 49
3. gameId: 6679, fen: Q4Q2/7k/2Q5/8/6P1/8/7P/6K1 b -  0 50
4. gameId: 8447, fen: 4QQQ1/8/8/8/8/8/BP1K4/5k2 b -  0 51
…
46469. gameId: 93676721, fen: 3KQ3/1Q6/Q7/8/8/8/k7/8 b -  8 72
46470. gameId: 93678068, fen: 8/8/8/8/2Q2Q2/6Q1/4k1N1/5RK1 b -  7 56
46471. gameId: 93678352, fen: 2Q5/8/1R6/3k3p/1Q5N/8/1Q4K1/1NB5 b -  0 43

Elapsed: 6801717 ms, 1:53:21, total results: 60887, time per results: 111 ms

Search with query: P[d4, e5, f4, g4] = 4 and kb7

Code: Select all

Elapsed: 5566237 ms, 1:32:46, total results: 3248, time per results: 1713 ms

Search with query: white6 = 5

Code: Select all

Elapsed: 5637996 ms, 1:33:57, total results: 4245, time per results: 1328 ms

The searching time is about 1 - 2 hours. That is reasonable for scanning the whole of a huge database of over 200 GB.

Search 2
We test with the database from Test 3 (moves in binary of 2 bytes, store in the field Moves2, the database size is 19.8 GB).

Search with query: Q = 3

Code: Select all

Elapsed: 1209782 ms, 20:09, total results: 60887, time per results: 19 ms

Search with query: P[d4, e5, f4, g4] = 4 and kb7

Code: Select all

Elapsed: 1180916 ms, 19:40, total results: 3248, time per results: 363 ms

Search with query: white6 = 5

Code: Select all

Elapsed: 1229161 ms, 20:29, total results: 4245, time per results: 289 ms

Search 3
We test with the database from Test 4 (moves in binary of 1 byte, store in the field Moves1, the database size is 13.7 GB).

Search with query: Q = 3

Code: Select all

Elapsed: 1295627 ms, 21:35, total results: 60887, time per results: 21 ms

Search with query: P[d4, e5, f4, g4] = 4 and kb7

Code: Select all

Elapsed: 1276914 ms, 21:16, total results: 3248, time per results: 393 ms

Search with query: white6 = 5

Code: Select all

Elapsed: 1303730 ms, 21:43, total results: 4245, time per results: 307 ms

Clearly, the searching speeds of Search 2 and 3 are much faster than Search 1 (about 20 times faster). The main reason for that speed is that the database size is much smaller and moves in binary can help to save a lot of computing.

C. Quick conclusions for this attempt

- Using some tricks and new features, our program can convert and work well with huge databases of over 90 million games. Probably ours is only chess game database program that could work with that number of games
- It didn’t have any trouble with computing or memory, computers typically say light or reasonable levels of usage
- Look like the program can work with much larger databases, say, a few times larger than the Lichess database we are using for this attempt
- The database with moves encoded in binary has a small size, very suitable for storing
- The position searching work well, the time is reasonable if moves in text form but much faster if moves in binary ones

D. Source code

We have invested some work to improve the program to be a useful and convenient tool, not only a test base of technologies. It has already some strong features such as auto-create new fields for new PGN tags. The program now can control a lot from the command line, to create databases, benchmarks, or query databases. Some other options have been added but missing controls and we will add them later. It is on the road to becoming an independent and full database tool.

That working branch may merge to the main one soon.

All code of this Attempt has been pushed already into a new branch “largedatabase”.

phhnguyen · Post by **phhnguyen** » Sat Jan 22, 2022 3:15 pm

Version Alpha released

I have just merged the work of the latest attempt (12th) to the branch main and released it as the version Alpha. There are two executable files for Windows and macOS on that release.

Using those executable programs, you may create SQL databases from some PGN files:

Code: Select all

ocgdb -pgn c:\games\big1.png -pgn c:\games\big2.png -db c:\db\big.ocgdb.db3 -cpu 4 -encode 2

You may download some sample databases at https://github.com/nguyenpham/ocgdb-samples

You may query positions from a database:

Code: Select all

ocgdb -db c:\db\big.ocgdb.db3 -cpu 4 -q "Q=3" -q "P[d4, e5, f4, g4] = 4 and kb7"

The app itself is on intensive development to be an independent, useful chess game database tool.

Enjoy

pferd · Post by **pferd** » Sun Jan 23, 2022 9:57 am

I just compiled it on my Linux machine. In order to be successful I had to add -lpthread and -ldl to the linker.

Creating the mb345 database took around 46 seconds with 8cpus. Creating a scid datbase took about 72 seconds.

Searching for 3 White Queens took 11 seconds and 57 results were found. With scid it took less than a second and only 3 games were found. ScidvsPc found 55 games in 1.8 seconds.

phhnguyen · Post by **phhnguyen** » Sun Jan 23, 2022 11:52 am

pferd wrote: ↑Sun Jan 23, 2022 9:57 am I just compiled it on my Linux machine. In order to be successful I had to add -lpthread and -ldl to the linker.

I am not good at creating Makefile. If someone could, please help to create a better one for multi OSs.

pferd wrote: ↑Sun Jan 23, 2022 9:57 am Creating the mb345 database took around 46 seconds with 8cpus. Creating a scid datbase took about 72 seconds.

Searching for 3 White Queens took 11 seconds and 57 results were found. With scid it took less than a second and only 3 games were found. ScidvsPc found 55 games in 1.8 seconds.

Very interesting information!

That surprised me since OBGDB has something significant faster than SCIDs!

I have known now OBGDB could compete fairly with SCIDs for every parameter but it should be... a bit slower when having similar settings/conditions

According to Fulvio on this post, his SCID found 57 results of 3 White Queens too within 3 seconds. Thus I guess you may need to check some settings.

BTW, there are some points I need your help to verify:
1. OCGDB doesn’t work well with virtual cores. If your computer has hyper threads, please set the number of cores to a half, say, -cpu 4, for both creating and searching (or better if you could try both 4 & 8 to see which one is the best)
2. If you create the Games table with moves in text only (Moves column), the program won’t parse games into chessboards when creating databases, thus it may be very fast. However, when searching, it has to parse moves in text into chessboard board before verifying search conditions, thus it is slower. In contrast, SCIDs always parse games when creating databases and get benefits when searching. To get a better comparison, please create databases with moves in binary forms, say, Moves2, by adding a parameter -encode 2

Code: Select all

ocgdb -pgn c:\games\big.png -db c:\db\big.ocgdb.db3 -cpu 4 -encode 2

Thanks

pferd · Post by **pferd** » Sun Jan 23, 2022 2:59 pm

I have a Ryzen 3700x, so 8 physical and 16 virtual cores. I have documented my results with greater detail this time.I do not know whats wrong with my scid version. Maybe I check in the next days.

Comparison of db creation with 8 vs 16 cores with -encode 2:

Code: Select all

/ocgdb -pgn ./mb-3.45.pgn -db ./mb-3.45-cpu8-encode2.ocdbg.db3 -cpu 8 -encode 2                                                                                                                                                                                                  ✔ 
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: create SQL database
        PGN paths:
                ./mb-3.45.pgn
        Database path: ./mb-3.45-cpu8-encode2.ocdbg.db3
        Queries:
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

SQLite database file './mb-3.45-cpu8-encode2.ocdbg.db3' opened successfully
Thread count: 8
Move encoding size: 2 without field Moves
Processing PGN file: './mb-3.45.pgn'
BEGIN TRANSACTION
#games: 228742, elapsed: 3108ms 00:03, speed: 73597 games/s, #blocks: 17, processed size: 136 MB
#games: 444564, elapsed: 5828ms 00:05, speed: 76280 games/s, #blocks: 33, processed size: 264 MB
#games: 660467, elapsed: 8510ms 00:08, speed: 77610 games/s, #blocks: 49, processed size: 392 MB
#games: 876236, elapsed: 11158ms 00:11, speed: 78529 games/s, #blocks: 65, processed size: 520 MB
#games: 1084961, elapsed: 13817ms 00:13, speed: 78523 games/s, #blocks: 81, processed size: 648 MB
#games: 1282580, elapsed: 16451ms 00:16, speed: 77963 games/s, #blocks: 97, processed size: 776 MB
#games: 1481295, elapsed: 18986ms 00:18, speed: 78020 games/s, #blocks: 113, processed size: 904 MB
#games: 1682015, elapsed: 21522ms 00:21, speed: 78153 games/s, #blocks: 129, processed size: 1032 MB
#games: 1878876, elapsed: 24040ms 00:24, speed: 78156 games/s, #blocks: 145, processed size: 1160 MB
#games: 2070568, elapsed: 26567ms 00:26, speed: 77937 games/s, #blocks: 161, processed size: 1288 MB
#games: 2263597, elapsed: 29060ms 00:29, speed: 77893 games/s, #blocks: 177, processed size: 1416 MB
#games: 2454031, elapsed: 31561ms 00:31, speed: 77755 games/s, #blocks: 193, processed size: 1544 MB
#games: 2642990, elapsed: 34080ms 00:34, speed: 77552 games/s, #blocks: 209, processed size: 1672 MB
#games: 2831375, elapsed: 36527ms 00:36, speed: 77514 games/s, #blocks: 225, processed size: 1800 MB
#games: 2994165, elapsed: 38870ms 00:38, speed: 77030 games/s, #blocks: 241, processed size: 1928 MB
#games: 3141649, elapsed: 41146ms 00:41, speed: 76353 games/s, #blocks: 257, processed size: 2056 MB
#games: 3297553, elapsed: 43486ms 00:43, speed: 75830 games/s, #blocks: 273, processed size: 2184 MB
#games: 3456399, elapsed: 46380ms 00:46, speed: 74523 games/s, #blocks: 288, processed size: 2303 MB
Completed! 
    ~/ocgdb/src    main !1 ?3  ./ocgdb -pgn ./mb-3.45.pgn -db ./mb-3.45-cpu16-encode2.ocdbg.db3 -cpu 16 -encode 2                                                                                                                                                                                        ✔  46s  
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: create SQL database
        PGN paths:
                ./mb-3.45.pgn
        Database path: ./mb-3.45-cpu16-encode2.ocdbg.db3
        Queries:
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 16, low Elo: 0

SQLite database file './mb-3.45-cpu16-encode2.ocdbg.db3' opened successfully
Thread count: 16
Move encoding size: 2 without field Moves
Processing PGN file: './mb-3.45.pgn'
BEGIN TRANSACTION
#games: 228742, elapsed: 3228ms 00:03, speed: 70861 games/s, #blocks: 17, processed size: 136 MB
#games: 444564, elapsed: 5954ms 00:05, speed: 74666 games/s, #blocks: 33, processed size: 264 MB
#games: 660467, elapsed: 8613ms 00:08, speed: 76682 games/s, #blocks: 49, processed size: 392 MB
#games: 876236, elapsed: 11208ms 00:11, speed: 78179 games/s, #blocks: 65, processed size: 520 MB
#games: 1084961, elapsed: 13802ms 00:13, speed: 78608 games/s, #blocks: 81, processed size: 648 MB
#games: 1282580, elapsed: 16316ms 00:16, speed: 78608 games/s, #blocks: 97, processed size: 776 MB
#games: 1481295, elapsed: 19104ms 00:19, speed: 77538 games/s, #blocks: 113, processed size: 904 MB
#games: 1682015, elapsed: 21559ms 00:21, speed: 78019 games/s, #blocks: 129, processed size: 1032 MB
#games: 1878876, elapsed: 24002ms 00:24, speed: 78279 games/s, #blocks: 145, processed size: 1160 MB
#games: 2070568, elapsed: 26405ms 00:26, speed: 78415 games/s, #blocks: 161, processed size: 1288 MB
#games: 2263597, elapsed: 28810ms 00:28, speed: 78569 games/s, #blocks: 177, processed size: 1416 MB
#games: 2454031, elapsed: 31137ms 00:31, speed: 78813 games/s, #blocks: 193, processed size: 1544 MB
#games: 2642990, elapsed: 33526ms 00:33, speed: 78834 games/s, #blocks: 209, processed size: 1672 MB
#games: 2831375, elapsed: 35877ms 00:35, speed: 78918 games/s, #blocks: 225, processed size: 1800 MB
#games: 2994165, elapsed: 38083ms 00:38, speed: 78622 games/s, #blocks: 241, processed size: 1928 MB
#games: 3141649, elapsed: 40226ms 00:40, speed: 78099 games/s, #blocks: 257, processed size: 2056 MB
#games: 3297553, elapsed: 42487ms 00:42, speed: 77613 games/s, #blocks: 273, processed size: 2184 MB
#games: 3456399, elapsed: 45369ms 00:45, speed: 76184 games/s, #blocks: 288, processed size: 2303 MB
Completed!

Encode 1 and encode 2 are the same speed when creating the database. Encode 0 only took about 31 seconds.

Size comparison between all databases. It is interesting to me, that using a different number of cores results in a different db size.

Code: Select all

-rw-r--r-- 1 tobias tobias 809336832 23. Jan 15:43 mb-3.45-cpu16-encode2.ocdbg.db3
-rw-r--r-- 1 tobias tobias 196042752 23. Jan 15:46 mb-3.45-cpu8-encode0.ocdbg.db3
-rw-r--r-- 1 tobias tobias 545337344 23. Jan 15:46 mb-3.45-cpu8-encode1.ocdbg.db3
-rw-r--r-- 1 tobias tobias 813903872 23. Jan 15:42 mb-3.45-cpu8-encode2.ocdbg.db3

I could not search the db with encode 0:

Code: Select all

./ocgdb -db ./mb-3.45-cpu8-encode0.ocdbg.db3 -q "Q=3" -cpu 8                                                                                                                                                                                                                      ✔ 
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: query
        PGN paths:
        Database path: ./mb-3.45-cpu8-encode0.ocdbg.db3
        Queries:
                Q=3
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

Thread count: 8
FATAL ERROR: missing move field (Moves or Moves1 or Moves2)
Completed!

The search speed for encode 1 and 2 with 8 threads were comparible. With 4 threads the query took 20 s.

Code: Select all

./ocgdb -db ./mb-3.45-cpu8-encode1.ocdbg.db3 -q "Q=3" -cpu 8                                                                                                                                                                                                                      ✔ 
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: query
        PGN paths:
        Database path: ./mb-3.45-cpu8-encode1.ocdbg.db3
        Queries:
                Q=3
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

Thread count: 8
Search with query Q=3..., searchField: Moves1
1. gameId: 13611, fen: r1b3Q1/pp6/1kn5/2bp4/5Q2/4QK2/1pq2P2/5BNR b -  0 23
2. gameId: 185317, fen: r1QQ1r2/p5bk/1p4p1/4p3/4q1p1/6Q1/P6P/R1R3K1 b -  0 35
3. gameId: 203111, fen: 1QQ1rbk1/5ppp/p2q1n2/3p4/4n3/1P1BPr1P/1B3P2/Q1R2RK1 b -  0 34
4. gameId: 286973, fen: 1QQ5/5pk1/5qp1/7p/4B3/1Q3P2/6PP/3R3K b -  0 42
5. gameId: 419966, fen: 1QQ5/4r1pk/Q6p/8/8/7P/6PK/8 b -  0 45
6. gameId: 793094, fen: Q1Q5/8/1Q6/6k1/3K2p1/1P4P1/8/8 b -  0 60
7. gameId: 824991, fen: BQQ5/5pk1/3Q1b1p/4p1p1/6P1/8/5PP1/4R1K1 b -  0 46
8. gameId: 864359, fen: r1b1kbQQ/2qp1p2/2n1p3/8/3N1P2/8/2P1B1P1/qqBQ1R1K b q  0 20
9. gameId: 904451, fen: 2QQ1rk1/5pp1/8/4QP2/8/q6P/pp4P1/4R2K b -  0 42
10. gameId: 1113376, fen: QQ3rk1/6bp/3p4/3Pnq1r/8/N3p3/P5PP/1R1Q3K b -  0 31
11. gameId: 1160557, fen: 3QQ3/2k5/1p3qp1/5n2/8/4QN2/P4KP1/8 b -  0 51
12. gameId: 1290444, fen: 2QQ1rk1/5p1p/4n1p1/4q3/2R5/r6P/6Q1/3R2NK b -  0 40
13. gameId: 1448906, fen: 1kb3QR/p1p5/1p2q3/8/3Qp3/2P1Q1R1/r4PK1/8 b -  0 40
14. gameId: 1484005, fen: 3Q2k1/4Q3/1q3ppp/8/8/8/6PP/q2r1Q1K b -  0 53
15. gameId: 1486171, fen: 5Q1K/4q1QP/1kpq4/p7/8/5Q2/8/8 b -  0 74
16. gameId: 1512006, fen: 3QQ1nk/6p1/1p3p2/pPp5/q5p1/5P2/5QKP/8 b -  0 44
17. gameId: 1552939, fen: 1QQ2r1k/6p1/7p/8/8/4BqP1/5P1P/5QK1 b -  0 67
18. gameId: 1570381, fen: Q6Q/1kn2Q2/2bq2B1/3p4/8/6N1/6PK/2q5 b -  0 58
19. gameId: 1601475, fen: 2QQ1r1k/6p1/4K2p/3PQ3/1p6/5qP1/8/8 b -  0 55
20. gameId: 1632795, fen: 1QQ2bk1/6r1/3p4/3Pp3/7p/5Pq1/4Q1Np/7K b -  0 39
21. gameId: 1658649, fen: 5Q2/kp6/p3n2q/2p5/8/6Q1/5QK1/8 b -  0 79
22. gameId: 1747920, fen: 4QQ2/8/kn6/p7/1ppP4/PqP5/1P6/1K2Q3 b -  0 50
23. gameId: 1759847, fen: QQ3rk1/7p/4p1p1/3pPr2/3B3q/2P2P2/6Q1/5RK1 b -  0 55
24. gameId: 1804829, fen: Q5n1/1KQ2pqk/2Q5/3N4/4P3/8/5q2/6q1 b -  0 58
25. gameId: 1927150, fen: Q7/6pk/6qp/5r2/5p2/3Q1P2/1R2p3/1K2Q3 b -  0 44
26. gameId: 1978517, fen: 1QQ5/8/8/2RQ4/1k6/6p1/6K1/8 b -  0 78
27. gameId: 2013189, fen: r4QQQ/1b6/pkp3r1/1p1p4/2n2R2/2P5/PP4P1/5RK1 b -  0 33
28. gameId: 2074369, fen: Q1Q5/5pk1/2Q3p1/5q2/7p/6P1/5P1P/2R3K1 b -  0 49
29. gameId: 2078314, fen: QQ2r3/5q1k/8/3p2p1/3B1pQ1/7P/5P1K/4q3 b -  0 54
30. gameId: 2139071, fen: 6Q1/8/2Q5/4p3/3qP3/5Q2/1q3PK1/qk6 b -  0 73
31. gameId: 2150224, fen: QQ1r3k/6pp/2p5/8/1PQ2p1q/4p2P/1R4P1/5KR1 b -  0 40
32. gameId: 2348023, fen: 1Q5Q/K7/6p1/8/6k1/5rq1/7p/3Q4 b -  0 61
33. gameId: 2689356, fen: 2Q1Q2N/R2Q2B1/8/5K2/8/8/P1k2P2/8 b -  0 73
34. gameId: 2951841, fen: 2QQ2R1/1Q6/4K3/4P1P1/8/P7/8/k7 b -  0 84
35. gameId: 2952331, fen: 6Q1/8/1Q6/8/P4N2/1P6/4RP1Q/2k4K b -  0 67
36. gameId: 2968213, fen: 4krQQ/1br5/p2qp3/1p4Q1/3p4/P2B2P1/1PP5/K5n1 b -  0 41
37. gameId: 2982337, fen: 5QQQ/8/8/4B3/5K2/3k4/8/8 b -  0 90
38. gameId: 3011690, fen: 1Q2Q3/P7/8/6K1/8/4Q3/7k/8 b -  0 67
39. gameId: 3016976, fen: 3Q3Q/8/2Q1N3/8/8/3K2k1/8/8 b -  0 75
40. gameId: 3020618, fen: 1QQQ4/2K1P3/8/8/k7/8/8/8 b -  0 93
41. gameId: 3066505, fen: Q7/1Q6/3ppkp1/4p2p/8/4QKPP/2q2P2/6r1 b -  0 46
42. gameId: 3072455, fen: Q2k4/1Q6/1K5Q/8/8/8/8/8 b -  0 70
43. gameId: 3114874, fen: 4QQQ1/8/8/8/8/2K5/2B5/4k3 b -  0 71
44. gameId: 3125226, fen: 1Q2Q3/1Q6/8/6N1/5K2/8/k7/8 b -  0 74
45. gameId: 3168700, fen: 1QQ5/6pk/6nr/8/8/4P1Q1/6PP/5RK1 b -  0 51
46. gameId: 3170156, fen: 2Q5/PP6/8/8/8/8/3Q4/1k2K1Q1 b -  0 95
47. gameId: 3171152, fen: 1QQ5/5ppk/7p/q1pp4/5B2/3b2PP/1Q2r1PK/8 b -  0 41
48. gameId: 3211109, fen: 1QQ1r1k1/5pp1/p3r2q/8/1P2N1Pp/P2Q4/4K2P/R6R b -  0 32
49. gameId: 3220759, fen: 1K4QQ/5Q2/8/8/8/3B4/7k/8 b -  0 77
50. gameId: 3264751, fen: 2Q3Q1/8/4b3/8/1k6/8/p4K2/Q7 b -  0 65
51. gameId: 3265288, fen: Q2QQR2/6k1/8/8/5KP1/8/8/8 b -  0 71
52. gameId: 3389290, fen: QQ6/8/8/3K2p1/6kp/8/7Q/8 b -  0 63
53. gameId: 3437475, fen: QQ6/8/K7/8/1Q6/8/k7/8 b -  0 94
54. gameId: 3438421, fen: 1QQ2rk1/4q1bp/Q2p1np1/4p3/2P1Pp1P/5P2/5BP1/4KB1R b K  0 28
55. gameId: 3444433, fen: 1Q1QQ3/8/8/5k2/8/8/P7/2K5 b -  0 59
56. gameId: 3445945, fen: 4Q2k/5Q2/p5p1/1p2P3/3P4/P2p2QP/3q4/2r3RK b -  0 48
57. gameId: 3450569, fen: 1Q3Q1Q/6B1/8/1p2p3/4k3/2P1p3/PP2K3/7R b -  0 47

Q=3 DONE. Elapsed: 12363 ms, 00:12, total results: 57, time per results: 216 ms


Completed! 
    ~/ocgdb/src    main !1 ?6  ./ocgdb -db ./mb-3.45-cpu8-encode2.ocdbg.db3 -q "Q=3" -cpu 8                                                                                                                                                                                                              ✔  13s  
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: query
        PGN paths:
        Database path: ./mb-3.45-cpu8-encode2.ocdbg.db3
        Queries:
                Q=3
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

Thread count: 8
Search with query Q=3..., searchField: Moves2
1. gameId: 13611, fen: r1b3Q1/pp6/1kn5/2bp4/5Q2/4QK2/1pq2P2/5BNR b -  0 23
2. gameId: 185317, fen: r1QQ1r2/p5bk/1p4p1/4p3/4q1p1/6Q1/P6P/R1R3K1 b -  0 35
3. gameId: 203113, fen: 1QQ1rbk1/5ppp/p2q1n2/3p4/4n3/1P1BPr1P/1B3P2/Q1R2RK1 b -  0 34
4. gameId: 286973, fen: 1QQ5/5pk1/5qp1/7p/4B3/1Q3P2/6PP/3R3K b -  0 42
5. gameId: 419964, fen: 1QQ5/4r1pk/Q6p/8/8/7P/6PK/8 b -  0 45
6. gameId: 793094, fen: Q1Q5/8/1Q6/6k1/3K2p1/1P4P1/8/8 b -  0 60
7. gameId: 824992, fen: BQQ5/5pk1/3Q1b1p/4p1p1/6P1/8/5PP1/4R1K1 b -  0 46
8. gameId: 864360, fen: r1b1kbQQ/2qp1p2/2n1p3/8/3N1P2/8/2P1B1P1/qqBQ1R1K b q  0 20
9. gameId: 904451, fen: 2QQ1rk1/5pp1/8/4QP2/8/q6P/pp4P1/4R2K b -  0 42
10. gameId: 1113376, fen: QQ3rk1/6bp/3p4/3Pnq1r/8/N3p3/P5PP/1R1Q3K b -  0 31
11. gameId: 1160557, fen: 3QQ3/2k5/1p3qp1/5n2/8/4QN2/P4KP1/8 b -  0 51
12. gameId: 1290445, fen: 2QQ1rk1/5p1p/4n1p1/4q3/2R5/r6P/6Q1/3R2NK b -  0 40
13. gameId: 1448906, fen: 1kb3QR/p1p5/1p2q3/8/3Qp3/2P1Q1R1/r4PK1/8 b -  0 40
14. gameId: 1484005, fen: 3Q2k1/4Q3/1q3ppp/8/8/8/6PP/q2r1Q1K b -  0 53
15. gameId: 1486170, fen: 5Q1K/4q1QP/1kpq4/p7/8/5Q2/8/8 b -  0 74
16. gameId: 1512009, fen: 3QQ1nk/6p1/1p3p2/pPp5/q5p1/5P2/5QKP/8 b -  0 44
17. gameId: 1552941, fen: 1QQ2r1k/6p1/7p/8/8/4BqP1/5P1P/5QK1 b -  0 67
18. gameId: 1570381, fen: Q6Q/1kn2Q2/2bq2B1/3p4/8/6N1/6PK/2q5 b -  0 58
19. gameId: 1601475, fen: 2QQ1r1k/6p1/4K2p/3PQ3/1p6/5qP1/8/8 b -  0 55
20. gameId: 1632795, fen: 1QQ2bk1/6r1/3p4/3Pp3/7p/5Pq1/4Q1Np/7K b -  0 39
21. gameId: 1658647, fen: 5Q2/kp6/p3n2q/2p5/8/6Q1/5QK1/8 b -  0 79
22. gameId: 1747922, fen: 4QQ2/8/kn6/p7/1ppP4/PqP5/1P6/1K2Q3 b -  0 50
23. gameId: 1759847, fen: QQ3rk1/7p/4p1p1/3pPr2/3B3q/2P2P2/6Q1/5RK1 b -  0 55
24. gameId: 1804829, fen: Q5n1/1KQ2pqk/2Q5/3N4/4P3/8/5q2/6q1 b -  0 58
25. gameId: 1927150, fen: Q7/6pk/6qp/5r2/5p2/3Q1P2/1R2p3/1K2Q3 b -  0 44
26. gameId: 1978517, fen: 1QQ5/8/8/2RQ4/1k6/6p1/6K1/8 b -  0 78
27. gameId: 2013190, fen: r4QQQ/1b6/pkp3r1/1p1p4/2n2R2/2P5/PP4P1/5RK1 b -  0 33
28. gameId: 2074368, fen: Q1Q5/5pk1/2Q3p1/5q2/7p/6P1/5P1P/2R3K1 b -  0 49
29. gameId: 2078314, fen: QQ2r3/5q1k/8/3p2p1/3B1pQ1/7P/5P1K/4q3 b -  0 54
30. gameId: 2139072, fen: 6Q1/8/2Q5/4p3/3qP3/5Q2/1q3PK1/qk6 b -  0 73
31. gameId: 2150224, fen: QQ1r3k/6pp/2p5/8/1PQ2p1q/4p2P/1R4P1/5KR1 b -  0 40
32. gameId: 2348023, fen: 1Q5Q/K7/6p1/8/6k1/5rq1/7p/3Q4 b -  0 61
33. gameId: 2689355, fen: 2Q1Q2N/R2Q2B1/8/5K2/8/8/P1k2P2/8 b -  0 73
34. gameId: 2951841, fen: 2QQ2R1/1Q6/4K3/4P1P1/8/P7/8/k7 b -  0 84
35. gameId: 2952332, fen: 6Q1/8/1Q6/8/P4N2/1P6/4RP1Q/2k4K b -  0 67
36. gameId: 2968213, fen: 4krQQ/1br5/p2qp3/1p4Q1/3p4/P2B2P1/1PP5/K5n1 b -  0 41
37. gameId: 2982337, fen: 5QQQ/8/8/4B3/5K2/3k4/8/8 b -  0 90
38. gameId: 3011690, fen: 1Q2Q3/P7/8/6K1/8/4Q3/7k/8 b -  0 67
39. gameId: 3016976, fen: 3Q3Q/8/2Q1N3/8/8/3K2k1/8/8 b -  0 75
40. gameId: 3020618, fen: 1QQQ4/2K1P3/8/8/k7/8/8/8 b -  0 93
41. gameId: 3066505, fen: Q7/1Q6/3ppkp1/4p2p/8/4QKPP/2q2P2/6r1 b -  0 46
42. gameId: 3072455, fen: Q2k4/1Q6/1K5Q/8/8/8/8/8 b -  0 70
43. gameId: 3114874, fen: 4QQQ1/8/8/8/8/2K5/2B5/4k3 b -  0 71
44. gameId: 3125226, fen: 1Q2Q3/1Q6/8/6N1/5K2/8/k7/8 b -  0 74
45. gameId: 3168700, fen: 1QQ5/6pk/6nr/8/8/4P1Q1/6PP/5RK1 b -  0 51
46. gameId: 3170155, fen: 2Q5/PP6/8/8/8/8/3Q4/1k2K1Q1 b -  0 95
47. gameId: 3171151, fen: 1QQ5/5ppk/7p/q1pp4/5B2/3b2PP/1Q2r1PK/8 b -  0 41
48. gameId: 3211109, fen: 1QQ1r1k1/5pp1/p3r2q/8/1P2N1Pp/P2Q4/4K2P/R6R b -  0 32
49. gameId: 3220759, fen: 1K4QQ/5Q2/8/8/8/3B4/7k/8 b -  0 77
50. gameId: 3264751, fen: 2Q3Q1/8/4b3/8/1k6/8/p4K2/Q7 b -  0 65
51. gameId: 3265285, fen: Q2QQR2/6k1/8/8/5KP1/8/8/8 b -  0 71
52. gameId: 3389290, fen: QQ6/8/8/3K2p1/6kp/8/7Q/8 b -  0 63
53. gameId: 3437475, fen: QQ6/8/K7/8/1Q6/8/k7/8 b -  0 94
54. gameId: 3438422, fen: 1QQ2rk1/4q1bp/Q2p1np1/4p3/2P1Pp1P/5P2/5BP1/4KB1R b K  0 28
55. gameId: 3444434, fen: 1Q1QQ3/8/8/5k2/8/8/P7/2K5 b -  0 59
56. gameId: 3445945, fen: 4Q2k/5Q2/p5p1/1p2P3/3P4/P2p2QP/3q4/2r3RK b -  0 48
57. gameId: 3450569, fen: 1Q3Q1Q/6B1/8/1p2p3/4k3/2P1p3/PP2K3/7R b -  0 47

Q=3 DONE. Elapsed: 11179 ms, 00:11, total results: 57, time per results: 196 ms

phhnguyen · Post by **phhnguyen** » Sun Jan 23, 2022 10:44 pm

pferd wrote: ↑Sun Jan 23, 2022 2:59 pm I have a Ryzen 3700x, so 8 physical and 16 virtual cores. I have documented my results with greater detail this time.I do not know whats wrong with my scid version. Maybe I check in the next days.

Comparison of db creation with 8 vs 16 cores with -encode 2:

Code: Select all

/ocgdb -pgn ./mb-3.45.pgn -db ./mb-3.45-cpu8-encode2.ocdbg.db3 -cpu 8 -encode 2                                                                                                                                                                                                  ✔ 
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: create SQL database
        PGN paths:
                ./mb-3.45.pgn
        Database path: ./mb-3.45-cpu8-encode2.ocdbg.db3
        Queries:
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

SQLite database file './mb-3.45-cpu8-encode2.ocdbg.db3' opened successfully
Thread count: 8
Move encoding size: 2 without field Moves
Processing PGN file: './mb-3.45.pgn'
BEGIN TRANSACTION
#games: 228742, elapsed: 3108ms 00:03, speed: 73597 games/s, #blocks: 17, processed size: 136 MB
#games: 444564, elapsed: 5828ms 00:05, speed: 76280 games/s, #blocks: 33, processed size: 264 MB
#games: 660467, elapsed: 8510ms 00:08, speed: 77610 games/s, #blocks: 49, processed size: 392 MB
#games: 876236, elapsed: 11158ms 00:11, speed: 78529 games/s, #blocks: 65, processed size: 520 MB
#games: 1084961, elapsed: 13817ms 00:13, speed: 78523 games/s, #blocks: 81, processed size: 648 MB
#games: 1282580, elapsed: 16451ms 00:16, speed: 77963 games/s, #blocks: 97, processed size: 776 MB
#games: 1481295, elapsed: 18986ms 00:18, speed: 78020 games/s, #blocks: 113, processed size: 904 MB
#games: 1682015, elapsed: 21522ms 00:21, speed: 78153 games/s, #blocks: 129, processed size: 1032 MB
#games: 1878876, elapsed: 24040ms 00:24, speed: 78156 games/s, #blocks: 145, processed size: 1160 MB
#games: 2070568, elapsed: 26567ms 00:26, speed: 77937 games/s, #blocks: 161, processed size: 1288 MB
#games: 2263597, elapsed: 29060ms 00:29, speed: 77893 games/s, #blocks: 177, processed size: 1416 MB
#games: 2454031, elapsed: 31561ms 00:31, speed: 77755 games/s, #blocks: 193, processed size: 1544 MB
#games: 2642990, elapsed: 34080ms 00:34, speed: 77552 games/s, #blocks: 209, processed size: 1672 MB
#games: 2831375, elapsed: 36527ms 00:36, speed: 77514 games/s, #blocks: 225, processed size: 1800 MB
#games: 2994165, elapsed: 38870ms 00:38, speed: 77030 games/s, #blocks: 241, processed size: 1928 MB
#games: 3141649, elapsed: 41146ms 00:41, speed: 76353 games/s, #blocks: 257, processed size: 2056 MB
#games: 3297553, elapsed: 43486ms 00:43, speed: 75830 games/s, #blocks: 273, processed size: 2184 MB
#games: 3456399, elapsed: 46380ms 00:46, speed: 74523 games/s, #blocks: 288, processed size: 2303 MB
Completed! 
    ~/ocgdb/src    main !1 ?3  ./ocgdb -pgn ./mb-3.45.pgn -db ./mb-3.45-cpu16-encode2.ocdbg.db3 -cpu 16 -encode 2                                                                                                                                                                                        ✔  46s  
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: create SQL database
        PGN paths:
                ./mb-3.45.pgn
        Database path: ./mb-3.45-cpu16-encode2.ocdbg.db3
        Queries:
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 16, low Elo: 0

SQLite database file './mb-3.45-cpu16-encode2.ocdbg.db3' opened successfully
Thread count: 16
Move encoding size: 2 without field Moves
Processing PGN file: './mb-3.45.pgn'
BEGIN TRANSACTION
#games: 228742, elapsed: 3228ms 00:03, speed: 70861 games/s, #blocks: 17, processed size: 136 MB
#games: 444564, elapsed: 5954ms 00:05, speed: 74666 games/s, #blocks: 33, processed size: 264 MB
#games: 660467, elapsed: 8613ms 00:08, speed: 76682 games/s, #blocks: 49, processed size: 392 MB
#games: 876236, elapsed: 11208ms 00:11, speed: 78179 games/s, #blocks: 65, processed size: 520 MB
#games: 1084961, elapsed: 13802ms 00:13, speed: 78608 games/s, #blocks: 81, processed size: 648 MB
#games: 1282580, elapsed: 16316ms 00:16, speed: 78608 games/s, #blocks: 97, processed size: 776 MB
#games: 1481295, elapsed: 19104ms 00:19, speed: 77538 games/s, #blocks: 113, processed size: 904 MB
#games: 1682015, elapsed: 21559ms 00:21, speed: 78019 games/s, #blocks: 129, processed size: 1032 MB
#games: 1878876, elapsed: 24002ms 00:24, speed: 78279 games/s, #blocks: 145, processed size: 1160 MB
#games: 2070568, elapsed: 26405ms 00:26, speed: 78415 games/s, #blocks: 161, processed size: 1288 MB
#games: 2263597, elapsed: 28810ms 00:28, speed: 78569 games/s, #blocks: 177, processed size: 1416 MB
#games: 2454031, elapsed: 31137ms 00:31, speed: 78813 games/s, #blocks: 193, processed size: 1544 MB
#games: 2642990, elapsed: 33526ms 00:33, speed: 78834 games/s, #blocks: 209, processed size: 1672 MB
#games: 2831375, elapsed: 35877ms 00:35, speed: 78918 games/s, #blocks: 225, processed size: 1800 MB
#games: 2994165, elapsed: 38083ms 00:38, speed: 78622 games/s, #blocks: 241, processed size: 1928 MB
#games: 3141649, elapsed: 40226ms 00:40, speed: 78099 games/s, #blocks: 257, processed size: 2056 MB
#games: 3297553, elapsed: 42487ms 00:42, speed: 77613 games/s, #blocks: 273, processed size: 2184 MB
#games: 3456399, elapsed: 45369ms 00:45, speed: 76184 games/s, #blocks: 288, processed size: 2303 MB
Completed!

Encode 1 and encode 2 are the same speed when creating the database. Encode 0 only took about 31 seconds.

Size comparison between all databases. It is interesting to me, that using a different number of cores results in a different db size.

Code: Select all

-rw-r--r-- 1 tobias tobias 809336832 23. Jan 15:43 mb-3.45-cpu16-encode2.ocdbg.db3
-rw-r--r-- 1 tobias tobias 196042752 23. Jan 15:46 mb-3.45-cpu8-encode0.ocdbg.db3
-rw-r--r-- 1 tobias tobias 545337344 23. Jan 15:46 mb-3.45-cpu8-encode1.ocdbg.db3
-rw-r--r-- 1 tobias tobias 813903872 23. Jan 15:42 mb-3.45-cpu8-encode2.ocdbg.db3

I could not search the db with encode 0:

"Encode 0" means there is not any binary column for moves. In that case, you may need to create a text move column instead (column Moves), using the argument -keepmoves.

pferd wrote: ↑Sun Jan 23, 2022 2:59 pm

Code: Select all

./ocgdb -db ./mb-3.45-cpu8-encode0.ocdbg.db3 -q "Q=3" -cpu 8                                                                                                                                                                                                                      ✔ 
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: query
        PGN paths:
        Database path: ./mb-3.45-cpu8-encode0.ocdbg.db3
        Queries:
                Q=3
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

Thread count: 8
FATAL ERROR: missing move field (Moves or Moves1 or Moves2)
Completed!

The search speed for encode 1 and 2 with 8 threads were comparible. With 4 threads the query took 20 s.

Code: Select all

./ocgdb -db ./mb-3.45-cpu8-encode1.ocdbg.db3 -q "Q=3" -cpu 8                                                                                                                                                                                                                      ✔ 
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: query
        PGN paths:
        Database path: ./mb-3.45-cpu8-encode1.ocdbg.db3
        Queries:
                Q=3
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

Thread count: 8
Search with query Q=3..., searchField: Moves1
1. gameId: 13611, fen: r1b3Q1/pp6/1kn5/2bp4/5Q2/4QK2/1pq2P2/5BNR b -  0 23
2. gameId: 185317, fen: r1QQ1r2/p5bk/1p4p1/4p3/4q1p1/6Q1/P6P/R1R3K1 b -  0 35
3. gameId: 203111, fen: 1QQ1rbk1/5ppp/p2q1n2/3p4/4n3/1P1BPr1P/1B3P2/Q1R2RK1 b -  0 34
4. gameId: 286973, fen: 1QQ5/5pk1/5qp1/7p/4B3/1Q3P2/6PP/3R3K b -  0 42
5. gameId: 419966, fen: 1QQ5/4r1pk/Q6p/8/8/7P/6PK/8 b -  0 45
6. gameId: 793094, fen: Q1Q5/8/1Q6/6k1/3K2p1/1P4P1/8/8 b -  0 60
7. gameId: 824991, fen: BQQ5/5pk1/3Q1b1p/4p1p1/6P1/8/5PP1/4R1K1 b -  0 46
8. gameId: 864359, fen: r1b1kbQQ/2qp1p2/2n1p3/8/3N1P2/8/2P1B1P1/qqBQ1R1K b q  0 20
9. gameId: 904451, fen: 2QQ1rk1/5pp1/8/4QP2/8/q6P/pp4P1/4R2K b -  0 42
10. gameId: 1113376, fen: QQ3rk1/6bp/3p4/3Pnq1r/8/N3p3/P5PP/1R1Q3K b -  0 31
11. gameId: 1160557, fen: 3QQ3/2k5/1p3qp1/5n2/8/4QN2/P4KP1/8 b -  0 51
12. gameId: 1290444, fen: 2QQ1rk1/5p1p/4n1p1/4q3/2R5/r6P/6Q1/3R2NK b -  0 40
13. gameId: 1448906, fen: 1kb3QR/p1p5/1p2q3/8/3Qp3/2P1Q1R1/r4PK1/8 b -  0 40
14. gameId: 1484005, fen: 3Q2k1/4Q3/1q3ppp/8/8/8/6PP/q2r1Q1K b -  0 53
15. gameId: 1486171, fen: 5Q1K/4q1QP/1kpq4/p7/8/5Q2/8/8 b -  0 74
16. gameId: 1512006, fen: 3QQ1nk/6p1/1p3p2/pPp5/q5p1/5P2/5QKP/8 b -  0 44
17. gameId: 1552939, fen: 1QQ2r1k/6p1/7p/8/8/4BqP1/5P1P/5QK1 b -  0 67
18. gameId: 1570381, fen: Q6Q/1kn2Q2/2bq2B1/3p4/8/6N1/6PK/2q5 b -  0 58
19. gameId: 1601475, fen: 2QQ1r1k/6p1/4K2p/3PQ3/1p6/5qP1/8/8 b -  0 55
20. gameId: 1632795, fen: 1QQ2bk1/6r1/3p4/3Pp3/7p/5Pq1/4Q1Np/7K b -  0 39
21. gameId: 1658649, fen: 5Q2/kp6/p3n2q/2p5/8/6Q1/5QK1/8 b -  0 79
22. gameId: 1747920, fen: 4QQ2/8/kn6/p7/1ppP4/PqP5/1P6/1K2Q3 b -  0 50
23. gameId: 1759847, fen: QQ3rk1/7p/4p1p1/3pPr2/3B3q/2P2P2/6Q1/5RK1 b -  0 55
24. gameId: 1804829, fen: Q5n1/1KQ2pqk/2Q5/3N4/4P3/8/5q2/6q1 b -  0 58
25. gameId: 1927150, fen: Q7/6pk/6qp/5r2/5p2/3Q1P2/1R2p3/1K2Q3 b -  0 44
26. gameId: 1978517, fen: 1QQ5/8/8/2RQ4/1k6/6p1/6K1/8 b -  0 78
27. gameId: 2013189, fen: r4QQQ/1b6/pkp3r1/1p1p4/2n2R2/2P5/PP4P1/5RK1 b -  0 33
28. gameId: 2074369, fen: Q1Q5/5pk1/2Q3p1/5q2/7p/6P1/5P1P/2R3K1 b -  0 49
29. gameId: 2078314, fen: QQ2r3/5q1k/8/3p2p1/3B1pQ1/7P/5P1K/4q3 b -  0 54
30. gameId: 2139071, fen: 6Q1/8/2Q5/4p3/3qP3/5Q2/1q3PK1/qk6 b -  0 73
31. gameId: 2150224, fen: QQ1r3k/6pp/2p5/8/1PQ2p1q/4p2P/1R4P1/5KR1 b -  0 40
32. gameId: 2348023, fen: 1Q5Q/K7/6p1/8/6k1/5rq1/7p/3Q4 b -  0 61
33. gameId: 2689356, fen: 2Q1Q2N/R2Q2B1/8/5K2/8/8/P1k2P2/8 b -  0 73
34. gameId: 2951841, fen: 2QQ2R1/1Q6/4K3/4P1P1/8/P7/8/k7 b -  0 84
35. gameId: 2952331, fen: 6Q1/8/1Q6/8/P4N2/1P6/4RP1Q/2k4K b -  0 67
36. gameId: 2968213, fen: 4krQQ/1br5/p2qp3/1p4Q1/3p4/P2B2P1/1PP5/K5n1 b -  0 41
37. gameId: 2982337, fen: 5QQQ/8/8/4B3/5K2/3k4/8/8 b -  0 90
38. gameId: 3011690, fen: 1Q2Q3/P7/8/6K1/8/4Q3/7k/8 b -  0 67
39. gameId: 3016976, fen: 3Q3Q/8/2Q1N3/8/8/3K2k1/8/8 b -  0 75
40. gameId: 3020618, fen: 1QQQ4/2K1P3/8/8/k7/8/8/8 b -  0 93
41. gameId: 3066505, fen: Q7/1Q6/3ppkp1/4p2p/8/4QKPP/2q2P2/6r1 b -  0 46
42. gameId: 3072455, fen: Q2k4/1Q6/1K5Q/8/8/8/8/8 b -  0 70
43. gameId: 3114874, fen: 4QQQ1/8/8/8/8/2K5/2B5/4k3 b -  0 71
44. gameId: 3125226, fen: 1Q2Q3/1Q6/8/6N1/5K2/8/k7/8 b -  0 74
45. gameId: 3168700, fen: 1QQ5/6pk/6nr/8/8/4P1Q1/6PP/5RK1 b -  0 51
46. gameId: 3170156, fen: 2Q5/PP6/8/8/8/8/3Q4/1k2K1Q1 b -  0 95
47. gameId: 3171152, fen: 1QQ5/5ppk/7p/q1pp4/5B2/3b2PP/1Q2r1PK/8 b -  0 41
48. gameId: 3211109, fen: 1QQ1r1k1/5pp1/p3r2q/8/1P2N1Pp/P2Q4/4K2P/R6R b -  0 32
49. gameId: 3220759, fen: 1K4QQ/5Q2/8/8/8/3B4/7k/8 b -  0 77
50. gameId: 3264751, fen: 2Q3Q1/8/4b3/8/1k6/8/p4K2/Q7 b -  0 65
51. gameId: 3265288, fen: Q2QQR2/6k1/8/8/5KP1/8/8/8 b -  0 71
52. gameId: 3389290, fen: QQ6/8/8/3K2p1/6kp/8/7Q/8 b -  0 63
53. gameId: 3437475, fen: QQ6/8/K7/8/1Q6/8/k7/8 b -  0 94
54. gameId: 3438421, fen: 1QQ2rk1/4q1bp/Q2p1np1/4p3/2P1Pp1P/5P2/5BP1/4KB1R b K  0 28
55. gameId: 3444433, fen: 1Q1QQ3/8/8/5k2/8/8/P7/2K5 b -  0 59
56. gameId: 3445945, fen: 4Q2k/5Q2/p5p1/1p2P3/3P4/P2p2QP/3q4/2r3RK b -  0 48
57. gameId: 3450569, fen: 1Q3Q1Q/6B1/8/1p2p3/4k3/2P1p3/PP2K3/7R b -  0 47

Q=3 DONE. Elapsed: 12363 ms, 00:12, total results: 57, time per results: 216 ms


Completed! 
    ~/ocgdb/src    main !1 ?6  ./ocgdb -db ./mb-3.45-cpu8-encode2.ocdbg.db3 -q "Q=3" -cpu 8                                                                                                                                                                                                              ✔  13s  
Open Chess Game Database Standard, Database Builder, (C) 2021
Paras:
        Task: query
        PGN paths:
        Database path: ./mb-3.45-cpu8-encode2.ocdbg.db3
        Queries:
                Q=3
        Move columns (for creating): Moves2
        Options: comment_discard; site_discard; query_print_all; query_print_fen; 
        gameNumberLimit: 281474976710655
        cpu: 8, low Elo: 0

Thread count: 8
Search with query Q=3..., searchField: Moves2
1. gameId: 13611, fen: r1b3Q1/pp6/1kn5/2bp4/5Q2/4QK2/1pq2P2/5BNR b -  0 23
2. gameId: 185317, fen: r1QQ1r2/p5bk/1p4p1/4p3/4q1p1/6Q1/P6P/R1R3K1 b -  0 35
3. gameId: 203113, fen: 1QQ1rbk1/5ppp/p2q1n2/3p4/4n3/1P1BPr1P/1B3P2/Q1R2RK1 b -  0 34
4. gameId: 286973, fen: 1QQ5/5pk1/5qp1/7p/4B3/1Q3P2/6PP/3R3K b -  0 42
5. gameId: 419964, fen: 1QQ5/4r1pk/Q6p/8/8/7P/6PK/8 b -  0 45
6. gameId: 793094, fen: Q1Q5/8/1Q6/6k1/3K2p1/1P4P1/8/8 b -  0 60
7. gameId: 824992, fen: BQQ5/5pk1/3Q1b1p/4p1p1/6P1/8/5PP1/4R1K1 b -  0 46
8. gameId: 864360, fen: r1b1kbQQ/2qp1p2/2n1p3/8/3N1P2/8/2P1B1P1/qqBQ1R1K b q  0 20
9. gameId: 904451, fen: 2QQ1rk1/5pp1/8/4QP2/8/q6P/pp4P1/4R2K b -  0 42
10. gameId: 1113376, fen: QQ3rk1/6bp/3p4/3Pnq1r/8/N3p3/P5PP/1R1Q3K b -  0 31
11. gameId: 1160557, fen: 3QQ3/2k5/1p3qp1/5n2/8/4QN2/P4KP1/8 b -  0 51
12. gameId: 1290445, fen: 2QQ1rk1/5p1p/4n1p1/4q3/2R5/r6P/6Q1/3R2NK b -  0 40
13. gameId: 1448906, fen: 1kb3QR/p1p5/1p2q3/8/3Qp3/2P1Q1R1/r4PK1/8 b -  0 40
14. gameId: 1484005, fen: 3Q2k1/4Q3/1q3ppp/8/8/8/6PP/q2r1Q1K b -  0 53
15. gameId: 1486170, fen: 5Q1K/4q1QP/1kpq4/p7/8/5Q2/8/8 b -  0 74
16. gameId: 1512009, fen: 3QQ1nk/6p1/1p3p2/pPp5/q5p1/5P2/5QKP/8 b -  0 44
17. gameId: 1552941, fen: 1QQ2r1k/6p1/7p/8/8/4BqP1/5P1P/5QK1 b -  0 67
18. gameId: 1570381, fen: Q6Q/1kn2Q2/2bq2B1/3p4/8/6N1/6PK/2q5 b -  0 58
19. gameId: 1601475, fen: 2QQ1r1k/6p1/4K2p/3PQ3/1p6/5qP1/8/8 b -  0 55
20. gameId: 1632795, fen: 1QQ2bk1/6r1/3p4/3Pp3/7p/5Pq1/4Q1Np/7K b -  0 39
21. gameId: 1658647, fen: 5Q2/kp6/p3n2q/2p5/8/6Q1/5QK1/8 b -  0 79
22. gameId: 1747922, fen: 4QQ2/8/kn6/p7/1ppP4/PqP5/1P6/1K2Q3 b -  0 50
23. gameId: 1759847, fen: QQ3rk1/7p/4p1p1/3pPr2/3B3q/2P2P2/6Q1/5RK1 b -  0 55
24. gameId: 1804829, fen: Q5n1/1KQ2pqk/2Q5/3N4/4P3/8/5q2/6q1 b -  0 58
25. gameId: 1927150, fen: Q7/6pk/6qp/5r2/5p2/3Q1P2/1R2p3/1K2Q3 b -  0 44
26. gameId: 1978517, fen: 1QQ5/8/8/2RQ4/1k6/6p1/6K1/8 b -  0 78
27. gameId: 2013190, fen: r4QQQ/1b6/pkp3r1/1p1p4/2n2R2/2P5/PP4P1/5RK1 b -  0 33
28. gameId: 2074368, fen: Q1Q5/5pk1/2Q3p1/5q2/7p/6P1/5P1P/2R3K1 b -  0 49
29. gameId: 2078314, fen: QQ2r3/5q1k/8/3p2p1/3B1pQ1/7P/5P1K/4q3 b -  0 54
30. gameId: 2139072, fen: 6Q1/8/2Q5/4p3/3qP3/5Q2/1q3PK1/qk6 b -  0 73
31. gameId: 2150224, fen: QQ1r3k/6pp/2p5/8/1PQ2p1q/4p2P/1R4P1/5KR1 b -  0 40
32. gameId: 2348023, fen: 1Q5Q/K7/6p1/8/6k1/5rq1/7p/3Q4 b -  0 61
33. gameId: 2689355, fen: 2Q1Q2N/R2Q2B1/8/5K2/8/8/P1k2P2/8 b -  0 73
34. gameId: 2951841, fen: 2QQ2R1/1Q6/4K3/4P1P1/8/P7/8/k7 b -  0 84
35. gameId: 2952332, fen: 6Q1/8/1Q6/8/P4N2/1P6/4RP1Q/2k4K b -  0 67
36. gameId: 2968213, fen: 4krQQ/1br5/p2qp3/1p4Q1/3p4/P2B2P1/1PP5/K5n1 b -  0 41
37. gameId: 2982337, fen: 5QQQ/8/8/4B3/5K2/3k4/8/8 b -  0 90
38. gameId: 3011690, fen: 1Q2Q3/P7/8/6K1/8/4Q3/7k/8 b -  0 67
39. gameId: 3016976, fen: 3Q3Q/8/2Q1N3/8/8/3K2k1/8/8 b -  0 75
40. gameId: 3020618, fen: 1QQQ4/2K1P3/8/8/k7/8/8/8 b -  0 93
41. gameId: 3066505, fen: Q7/1Q6/3ppkp1/4p2p/8/4QKPP/2q2P2/6r1 b -  0 46
42. gameId: 3072455, fen: Q2k4/1Q6/1K5Q/8/8/8/8/8 b -  0 70
43. gameId: 3114874, fen: 4QQQ1/8/8/8/8/2K5/2B5/4k3 b -  0 71
44. gameId: 3125226, fen: 1Q2Q3/1Q6/8/6N1/5K2/8/k7/8 b -  0 74
45. gameId: 3168700, fen: 1QQ5/6pk/6nr/8/8/4P1Q1/6PP/5RK1 b -  0 51
46. gameId: 3170155, fen: 2Q5/PP6/8/8/8/8/3Q4/1k2K1Q1 b -  0 95
47. gameId: 3171151, fen: 1QQ5/5ppk/7p/q1pp4/5B2/3b2PP/1Q2r1PK/8 b -  0 41
48. gameId: 3211109, fen: 1QQ1r1k1/5pp1/p3r2q/8/1P2N1Pp/P2Q4/4K2P/R6R b -  0 32
49. gameId: 3220759, fen: 1K4QQ/5Q2/8/8/8/3B4/7k/8 b -  0 77
50. gameId: 3264751, fen: 2Q3Q1/8/4b3/8/1k6/8/p4K2/Q7 b -  0 65
51. gameId: 3265285, fen: Q2QQR2/6k1/8/8/5KP1/8/8/8 b -  0 71
52. gameId: 3389290, fen: QQ6/8/8/3K2p1/6kp/8/7Q/8 b -  0 63
53. gameId: 3437475, fen: QQ6/8/K7/8/1Q6/8/k7/8 b -  0 94
54. gameId: 3438422, fen: 1QQ2rk1/4q1bp/Q2p1np1/4p3/2P1Pp1P/5P2/5BP1/4KB1R b K  0 28
55. gameId: 3444434, fen: 1Q1QQ3/8/8/5k2/8/8/P7/2K5 b -  0 59
56. gameId: 3445945, fen: 4Q2k/5Q2/p5p1/1p2P3/3P4/P2p2QP/3q4/2r3RK b -  0 48
57. gameId: 3450569, fen: 1Q3Q1Q/6B1/8/1p2p3/4k3/2P1p3/PP2K3/7R b -  0 47

Q=3 DONE. Elapsed: 11179 ms, 00:11, total results: 57, time per results: 196 ms

Thanks for the info!

Clearly, the OCGDB program is faster than SCIDs when creating databases (with the parsing games option). Perhaps, that is caused by OCGDB could use more cores when SCIDs use only 2 cores (one for reading, one for parsing - as an explanation by Fulvio).

On the other hand, SCIDs are faster when searching. Perhaps, that is caused by SCIDs having optimized code and they can access data directly (when OCGDB accesses data via blobs of SQLite). We will look into that issue (added to our to-do list) later to see if we could improve the searching speed for OCGDB.

Open Chess Game Database Standard

Re: Open Chess Game Database Standard

Re: Open Chess Game Database Standard

Re: Open Chess Game Database Standard

Position searching: 11th attempt

Position searching: 12th attempt

OCGDB Version Alpha released

Re: Open Chess Game Database Standard

Re: Open Chess Game Database Standard

Re: Open Chess Game Database Standard

Re: Open Chess Game Database Standard