Page 5 of 12

Re: Database snapshot

Posted: Wed Jul 31, 2019 9:48 am
by noobpwnftw
Ferdy wrote: Wed Jul 31, 2019 5:55 am
noobpwnftw wrote: Wed Jul 31, 2019 5:10 am finally there exists a "Discover" program people can run, which loads any UCI-compatible engine and play games, contributing moves encountered during the process.
I tried to compile ChessDBDiscover and it seems to work. Created an engine dir and put engine.exe, a renamed stockfish dev engine, then run ChessDBDiscover.exe with default config values.

Image

So what does the ChessDBDiscover.exe is doing? Does it play engine vs engine matches based on StartFEN in config file? If so how many games? Does it send the results to the cloud?
Yes, it keeps playing games and submitting results until you close the program.

Re: Database snapshot

Posted: Wed Jul 31, 2019 10:20 am
by Damir
noobpwnftw wrote: Sat Jul 27, 2019 11:54 pm For those who want to probe my database locally or for other unspecified reasons, here is a full database snapshot of my book project as of today:

ftp://ftp.chessdb.cn/pub/chessdb/data-s ... 190728.tar

The database contains about 3 billion unique chess positions, mostly connected to startpos, analyzed by Stockfish with no less than 22 plies at terminal node and has a very wide multi-pv exploration, the scores been back-propagated using a weighted averaging function, also for most of the positions there is a special field(encoded as 'a0a0') marking known shortest distance of the position from startpos.

Using this database snapshot is as simple as putting the data files under your database folder and launch the server, yet still, I'd recommend you to use the online API and make feature requests if you need any, since it is getting updated constantly and I have no plans to make such kind of snapshots very frequently(while waiting for a contributor to make incremental snapshots possible).

This database snapshot is released into the public domain.
Is it possible to download your database in PGN as PGN format ?

Re: Database snapshot

Posted: Wed Jul 31, 2019 10:24 am
by Damir
or somehow convert it with a tool to PGN?

Re: Database snapshot

Posted: Wed Jul 31, 2019 11:40 am
by Ferdy
Updated the probe_chessdb_book.exe at https://drive.google.com/file/d/1_KM89V ... sp=sharing

Positive mate score with missing winrate is now fixed. winrate is now calculated based on score returned using logistic function. So a mate in 1 has a 100% winrate now.

[d]r1bqkb1r/ppppnppp/5n2/4p1N1/2B1P3/8/PPPP1PPP/RNBQK2R w KQkq - 6 5

Code: Select all

enter fen or epd or q to quit? r1bqkb1r/ppppnppp/5n2/4p1N1/2B1P3/8/PPPP1PPP/RNBQK2R w KQkq - 6 5
(1) Move Stats:
Move    Score   Winrate
Bxf7#   29999   100.00 

(2) PV info of top move:
29999/1 pv Bxf7#

(3) Database stats:
DB Size ( Storage / Queue ) : 64.41 GB / 23.60 GB

Re: Database snapshot

Posted: Wed Jul 31, 2019 2:50 pm
by Ferdy
noobpwnftw wrote: Wed Jul 31, 2019 9:48 am
Ferdy wrote: Wed Jul 31, 2019 5:55 am
noobpwnftw wrote: Wed Jul 31, 2019 5:10 am finally there exists a "Discover" program people can run, which loads any UCI-compatible engine and play games, contributing moves encountered during the process.
I tried to compile ChessDBDiscover and it seems to work. Created an engine dir and put engine.exe, a renamed stockfish dev engine, then run ChessDBDiscover.exe with default config values.

Image

So what does the ChessDBDiscover.exe is doing? Does it play engine vs engine matches based on StartFEN in config file? If so how many games? Does it send the results to the cloud?
Yes, it keeps playing games and submitting results until you close the program.
I tried to run ChessDBDiscover with the position in the image for not less 2 hours. This position is the last without book info.

Image

How do I know if positions from the run are saved in cloud?

Re: Database snapshot

Posted: Thu Aug 01, 2019 7:28 am
by noobpwnftw
Ferdy wrote: Wed Jul 31, 2019 2:50 pm How do I know if positions from the run are saved in cloud?
It will be there, just currently the DB has a very long queue, need a while to process them.

Re: Database snapshot

Posted: Fri Aug 02, 2019 1:26 pm
by Ferdy
Ferdy wrote: Tue Jul 30, 2019 10:40 am
Ovyron wrote: Tue Jul 30, 2019 12:54 am
Ferdy wrote: Tue Jul 30, 2019 12:47 amUse fen and not epd.
If you only have epd, just add "0 1" at the end to make it a fen.
Thanks, now it works.

This is really cool, I kind of feel like I downloaded some huge database that weighs just 9MB :)
Also working on simple html interface, I hope to release it next week.

Image
Html interface is released at https://github.com/fsmosca/ChessDB-Online-Book

Download the whole thing and open index.html.

Or try it at https://fsmosca.github.io/ChessDB-Online-Book/
Make sure to enable javascript.

Re: Database snapshot

Posted: Fri Aug 02, 2019 2:34 pm
by zullil
Ferdy wrote: Fri Aug 02, 2019 1:26 pm
Html interface is released at https://github.com/fsmosca/ChessDB-Online-Book

Download the whole thing and open index.html.

Or try it at https://fsmosca.github.io/ChessDB-Online-Book/
Make sure to enable javascript.
Works fine if I download and open index.html. But the version online you've linked to doesn't work for me.

[EDIT] OK, Chrome is saying "Insecure content blocked. This page is trying to load scripts from unauthenticated sources." If I allow Chrome to "load unsafe scripts", then it works.

Re: Database snapshot

Posted: Fri Aug 02, 2019 3:10 pm
by zullil
How was the "winrate" for each position determined?

Re: Database snapshot

Posted: Fri Aug 02, 2019 3:57 pm
by Ferdy
zullil wrote: Fri Aug 02, 2019 3:10 pm How was the "winrate" for each position determined?
I suspect it is logistic function based on score. The score is from
"the scores been back-propagated using a weighted averaging function" see first post of this thread.

In javascipt when winrate is missing (by design) for mate scores, I use

Code: Select all

// 2019-08-02: ChessDB does not return winrate when score is mate.
// So we will calculate winrate by logistic function
if (typeof winrate === "undefined") {
  const K = 1.2;
  winrate = 100 * 1/(1 + (10**(-K*score/400)));
  winrate = winrate.toFixed(2);
}
Even for non-mate scores, this function is close to the winrate from server.