Knowledge Base 1.0

[Rebel]

Knowledge Base 1.0 is the work of months carefully building a quality Polyglot database with analyzed positions by top engines, mainly by Stockfish 12 and Stockfish 13 covering about 95% of the general opening theory. The estimated total computer time roughly is about 8 years 24/7 single cpu time limited to 5 real months 24/7 due to the use 2 powerful PC's with many cores. The average quality of of the 94 million analyzed positions is around ~3300 elo. The database contains a wealth of information for opening study and for correspondence players. Knowledge Base is made available for SCID for easy browsing through the Polyglot database by clicking on the moves.

....

http://rebel13.nl/home/scid-with-knowledge-base.html

[jryandx]

Wow this is awesome. Is there a way to query this database? I'm not familiar with Polyglot. I'm building a similar database with Stockfish 13 and storing the evaluations in a MongoDB database. This would make a great resource for training neural networks. Maybe consider making it available through Kaggle? https://www.kaggle.com/

[Frank Quisinsky]

Hi Ed,

thank you again for another good work after ...

- Somu 1.7a
- a new forum
- Polyglot 1.7a
- PGN Annotator 1.2

and so many other things you do for us so many years!!

An additional hint for people like SCID!
A good source for a big database for SCID GUI is:
http://caissabase.co.uk/ (4.7 Million games)

Another good source for databases:
http://billwall.phpwebhosting.com/ (Bill Wall).

Both I found last months (new for me).
So I am looking for strong databases for a new FEOBOS project I will start with Klaus Wlotzka soon.
During my search I found these pages.

Back to Ed ...
The PGN Annotator and Polyglot works just fine.
I like it a lot!!

I will looking in the knowledge base 1.0 soon.

Thanks again and Best
Frank

All the links I added in my Link selection!
PS: ECO key from SCID and Chessbase are not really good (what I dislike).
Better key can be found in Chesspad!
And best key's can be found on DVDs from Informator!

[peter]

Knowledge Base 1.0 is the work of months carefully building a quality Polyglot database with analyzed positions by top engines, mainly by Stockfish 12 and Stockfish 13 covering about 95% of the general opening theory. The estimated total computer time roughly is about 8 years 24/7 single cpu time limited to 5 real months 24/7 due to the use 2 powerful PC's with many cores. The average quality of of the 94 million analyzed positions is around ~3300 elo. The database contains a wealth of information for opening study and for correspondence players. Knowledge Base is made available for SCID for easy browsing through the Polyglot database by clicking on the moves.

....

http://rebel13.nl/home/scid-with-knowledge-base.html

Thanks for the great work!
Just to be sure, not to download something again, that I maybe already have, what's the main differenc to Knowledge.bin as a book? Can scores of the stored moves be seen now with Scid too like it worked with BareTail and the Knowledge.bin- book already?

[jefk]

killing two birds with one message;
yep i've combined the knowledge.bin (not bad) and
the cerebellum-merged (with lco analysis) into one book.
(was still possible <2Gb book)

but i'm still puzzled about this polyglot update(s)
and i don't see a need for two options coz sofar both
options still imho look like a little but imperfect
(sure there was a problem (e4 first book etc) but for
my earlier work it was not such a problem in practice).

Yet now upon inspection (mergebooks by Ed works quite well
just like earlier versions) i noticed that it's not mergin
book moves from book1 100 pct and 2 100 pct into both
50 pct (or 70/30 or so' its now 100/0 thats all). yes you
need to be a nerd to notice that but i like gambits etc
and maybe Ed Rebel can also release his polygot (*) on
github or otherwise sourceforge like i do sometimes
https://sourceforge.net/projects/chess-gambiteer/

default engine set to 1430
Whiz kids readers here can try to beat the program, set the
gambit engine to eg 2000 (or higher) and try to beat it.

(*) fork from HGM polyglot but then x64 and compiled release

PS chess is solved (draw) but imho (repeating myself) we need
an extra parameter, sharpness (stronger engines then can crush
weaker engines (or human) when using a 'sharper' book),
for humans even GM there's still stuff to learn (even when
you cannot beat the top engine)

[jryandx]

"PS chess is solved (draw) but imho (repeating myself) we need
an extra parameter, sharpness (stronger engines then can crush
weaker engines (or human) when using a 'sharper' book),
for humans even GM there's still stuff to learn (even when
you cannot beat the top engine)"

No way. Chess is solved when we know, for each position, the guaranteed outcome if the best moves are played from that point forward.

[Rebel]

Wow this is awesome. Is there a way to query this database? I'm not familiar with Polyglot. I'm building a similar database with Stockfish 13 and storing the evaluations in a MongoDB database. This would make a great resource for training neural networks. Maybe consider making it available through Kaggle? https://www.kaggle.com/

I think I have about 25 million analyzed Stockfish 13 (opening) positions in EPD format. It looks like this:

Code: Select all

r2r2k1/ppp1nppp/3p1q2/8/2N1PP2/2P2Q2/PP4PP/R4RK1 b - - bm d5; ce -3; acd 20;
1r2k2r/p1pqbppp/5n2/3p4/8/8/PPP1QPPP/RNB1R1K1 b k - bm O-O; ce 85; acd 20;
2krr3/p1pp2pp/b1pb1n2/R7/2P4P/PqN1B3/1P2QPP1/R3KB2 b Q - bm Rxe3; ce 564; acd 20;
2kr3r/ppp2pp1/2pq4/2b1n2p/4P1b1/1BN1B1Q1/PPP2PPP/R4RK1 w - - bm f4; ce 174; acd 20;
Q1bk1b1r/p1p1qppp/3p1n2/4P3/2P5/1P6/PB3PPP/RN2KB1R b KQ - bm Nd7; ce -917; acd 20;
r3r1k1/2p2ppp/1p1q1n2/3p4/6b1/2P3P1/PPQN1PBP/R3R1K1 w - - bm a4; ce 0; acd 20;
r1bq1rk1/bpp2ppp/p2p4/P2Pn3/5R2/1N6/1PP1BPPP/2BQ1RK1 w - - bm Be3; ce -4; acd 20;

I can make it available for download if you think it is usable.

[Rebel]

Knowledge Base 1.0 is the work of months carefully building a quality Polyglot database with analyzed positions by top engines, mainly by Stockfish 12 and Stockfish 13 covering about 95% of the general opening theory. The estimated total computer time roughly is about 8 years 24/7 single cpu time limited to 5 real months 24/7 due to the use 2 powerful PC's with many cores. The average quality of of the 94 million analyzed positions is around ~3300 elo. The database contains a wealth of information for opening study and for correspondence players. Knowledge Base is made available for SCID for easy browsing through the Polyglot database by clicking on the moves.

....

http://rebel13.nl/home/scid-with-knowledge-base.html

Thanks for the great work!
Just to be sure, not to download something again, that I maybe already have, what's the main differenc to Knowledge.bin as a book? Can scores of the stored moves be seen now with Scid too like it worked with BareTail and the Knowledge.bin- book already?

That's the whole point of moving to SCID, instead of the annoying Baretail moves, scores and depth are now visible in SCID and also one now can easily browse through openings by clicking on book moves.

[jryandx]

I think I have about 25 million analyzed Stockfish 13 (opening) positions in EPD format. It looks like this:

Code: Select all

r2r2k1/ppp1nppp/3p1q2/8/2N1PP2/2P2Q2/PP4PP/R4RK1 b - - bm d5; ce -3; acd 20;
1r2k2r/p1pqbppp/5n2/3p4/8/8/PPP1QPPP/RNB1R1K1 b k - bm O-O; ce 85; acd 20;
2krr3/p1pp2pp/b1pb1n2/R7/2P4P/PqN1B3/1P2QPP1/R3KB2 b Q - bm Rxe3; ce 564; acd 20;
2kr3r/ppp2pp1/2pq4/2b1n2p/4P1b1/1BN1B1Q1/PPP2PPP/R4RK1 w - - bm f4; ce 174; acd 20;
Q1bk1b1r/p1p1qppp/3p1n2/4P3/2P5/1P6/PB3PPP/RN2KB1R b KQ - bm Nd7; ce -917; acd 20;
r3r1k1/2p2ppp/1p1q1n2/3p4/6b1/2P3P1/PPQN1PBP/R3R1K1 w - - bm a4; ce 0; acd 20;
r1bq1rk1/bpp2ppp/p2p4/P2Pn3/5R2/1N6/1PP1BPPP/2BQ1RK1 w - - bm Be3; ce -4; acd 20;

I can make it available for download if you think it is usable.

Yeah, it's definitely usable for many purposes imo. I mean, anyone can generate it, but it takes a lot of time like you say. I'm recording games of Stockfish 13 with search depth set to 20, play against search depths 1,2,3....20 and I've built a GUI to play though these games as a way to try and learn chess (www.thechesstrainer.com) But I also want to train transformer neural networks to predict these evals, so the more data that's already available for training the better.

[Rebel]

I think I have about 25 million analyzed Stockfish 13 (opening) positions in EPD format. It looks like this:

Code: Select all

r2r2k1/ppp1nppp/3p1q2/8/2N1PP2/2P2Q2/PP4PP/R4RK1 b - - bm d5; ce -3; acd 20;
1r2k2r/p1pqbppp/5n2/3p4/8/8/PPP1QPPP/RNB1R1K1 b k - bm O-O; ce 85; acd 20;
2krr3/p1pp2pp/b1pb1n2/R7/2P4P/PqN1B3/1P2QPP1/R3KB2 b Q - bm Rxe3; ce 564; acd 20;
2kr3r/ppp2pp1/2pq4/2b1n2p/4P1b1/1BN1B1Q1/PPP2PPP/R4RK1 w - - bm f4; ce 174; acd 20;
Q1bk1b1r/p1p1qppp/3p1n2/4P3/2P5/1P6/PB3PPP/RN2KB1R b KQ - bm Nd7; ce -917; acd 20;
r3r1k1/2p2ppp/1p1q1n2/3p4/6b1/2P3P1/PPQN1PBP/R3R1K1 w - - bm a4; ce 0; acd 20;
r1bq1rk1/bpp2ppp/p2p4/P2Pn3/5R2/1N6/1PP1BPPP/2BQ1RK1 w - - bm Be3; ce -4; acd 20;

I can make it available for download if you think it is usable.

Yeah, it's definitely usable for many purposes imo. I mean, anyone can generate it, but it takes a lot of time like you say. I'm recording games of Stockfish 13 with search depth set to 20, play against search depths 1,2,3....20 and I've built a GUI to play though these games as a way to try and learn chess (www.thechesstrainer.com) But I also want to train transformer neural networks to predict these evals, so the more data that's already available for training the better.

http://rebel13.nl/dl/sf13-epd.7z

40 million.