Open Chess Game Database Standard

Sopel · Post by **Sopel** » Thu Dec 30, 2021 7:09 pm

Fulvio wrote: ↑Thu Dec 30, 2021 6:18 pm
Sopel wrote: ↑Thu Dec 30, 2021 5:34 pm edit. okay ,correction, now I saw there's a "lichess" tab, still there's about 200M games "only"
There is a "gear" icon which let's you select the games by time control, average rating and date.
If you select everything there are 383,251,358 games

I see, thanks.

ndbd · Post by **ndbd** » Thu Dec 30, 2021 10:36 pm

In chess one stores games as sequences of moves. However most of the thread deals with searches for particular positions, for example specific opening positions or partial positions with certain patterns - like a fiancetto bishop, a certain kind of rook ending etc.

There are basially only two ways to deal with that. Either
a) store each position (maybe in a way that you can search for partial positions), or
b) simply go through all stored move sequences, re-create the position and check if it matches your search pattern

The disadvantage of a) is that it takes a lot of space. That's no issue if you run this in a client-server model, where you just equip the server with enough space and memory. For a client-only approach it's not very desireable due to the sheer size.

The disadvantage of b) is that it is potentially very slow. However as ChessBase, SCID and Chess Assistant have shown, it works out pretty well if you store some particular information about each game which can be used to rule out most of the games for typical search queries, and therefore have to inspect only a fraction of all games - which significantly improves speed.

The problem I infer from this thread is that this approach cannot be easily (or can it?) re-constructed using SQL, namely

matching games-list = []
while nextgame:
_if not some condition to rule out the game matches:
___for each position of the game
______if the position matches, add to matching games-list

That's why SQL is _not_ the industry standard for chess databases. I am looking forward to what HCE has done once it is released. Depending on what they offer, they maybe have solved all these issues (I wonder how).

But I would encourage you to take a users perspective.

If I am an amateur I probably don't need a chess database anyway.

If I am a serious tournament player, please convince me why I shouold not shell out the $300 for a commercial database program. In any event I would need a decent database like Big or Mega which is $200. Which I cannot use with a free software program, because none can import the proprietary format.

So next best thing is SCID or one of the derivatives for which one of the free databases is < 1 GB, or this approach which is vastly larger. So instead I should download a 18 GB (maybe zip-compressed) database file with which I can search slower than any of the above.

If it's really about just exchanging games between programs, then there is (zip-compressed) PGN, which I can import into any database program.

phhnguyen · Post by **phhnguyen** » Sun Jan 02, 2022 3:14 pm

Position searching: 8th attempt - all calculated on the fly

This attempt is inspired by the information from discussions in this topic about SCID has calculated almost all things on the fly for position searching, without pre-calculating and storing those data as we did in previous attempts (Attempts 5, 6, 7). This attempt tries to answer consequent/curious questions if we could do the same and how good/bad it is.

In previous works, we measured a lot of parameters, including all kinds of speed we could do. However, some information was still not clear since they weren’t independently measured but with other things (such as creating/storing with SQLite databases). Thus it is better we do and re-measure everything from zero levels. SQL is a new and slow factor in this project. In this attempt, we put it aside to measure some basic features. For all tests, we don’t store any move or game in any form. All information is discarded immediately after passing each game.

1. Tests

Test 1
In this test, we measured how fast to read a large PGN file. We use a PGN file of 2.42 GB (3.45 million games) as the main data to test. We will read that data in multi-blocks without processing anything.

At the first glance, it should be an easy test: just read all data and measure the elapsed periods. However, we quickly realized there’s a “funny” factor involved. The system (mine is a macOS) is quite clever and it may catch some huge parts or even whole data after our first read. That made the elapsed time be quickly reduced to almost zero. When measuring it with some time-consuming tasks, the differences are just small proportions and could be ignored but could be seen clearly in this test. Btw, below are some stats: