Create/discover chess puzzles

[natopigro]

Hi all,
I have in idea about how to define programmatically "what is a chess puzzle/composition", or, more precisely, what is not a puzzle.
I would like to train and test my idea but, as usually in data science, I miss data.

I am looking for:
1) a database puzzles, possibly in fen notation, for the train and validation phase;
2) a 5-6-7-8-man starting position (not the related solution tree) where I can discover new puzzles
3) and/or a chess position random generator having the mumber of pieces as parameter

Here is what I've been able to found up to now:
a) the syzyg database syzygy-tables.info, I can download the files but they are in .rtbz format. If I could read them probably I would be albe to extract the starting positions, but I can't. Any advice? (I can basically only work with python)
b) a random chess position generator rosettacode.org/wiki/Generate_random_chess_position, it could be useful to generate the starting position but there's no way to limit the number of pieces and the outcome positions are "always" unnatural
c) here https://chess.stackexchange.com/questio ... pgn-or-fen there are usefull links to Mate in 2-3-4 puzzles but I am not only looking for mate puzzles. Look for instance this problem
[fen]3R4/8/8/8/1r4kB/1B2K3/8/8 w - - 0 1[/fen]

My most pressing need are the point 2) and 3).

My goal is to find new puzzles, my dream is to find some new beatiful puzzles like these
https://en.chessbase.com/post/wonderful ... mpositions#
or this
[fen]5rk1/pp4pp/4p3/2R3Q1/3n4/2q4r/P1P2PPP/5RK1 b - - 0 1[/fen]

[phhnguyen]

Just a few thoughts about making puzzles

IMO, there are two main types of puzzles:

1) Mate puzzles

I don’t think machine learning can help much here. You need an engine to verify the mating line for a given position. Mate puzzles are typically picked up from game databases (based on their results) or EGTBs.

To generate mate puzzles from an EGTB, you need to understand its data and the belonging library code deeply. Basically, all EGTB files don’t store chess positions but values of distance-to-convert/mate (DTC/DTM). The locations/index of each value actually is mapped to a unique chess position. You need to understand how to calculate the index from a given position. To generate, firstly you need to scan all data to find which indexes have the right DTC/DTM, say DTM = 16 for mate-in-8, then convert those indexes back to chess positions. All those tasks are doable but may be so headaches and buggy for beginners.

2) Blunder puzzles: find a good/right solution for a given position

You still need an engine for verifying, but before that, you need to find interesting positions. Typically people select them (puzzle positions) from already-analyzed games thus they can find out which moves are blunders. Lichess uses that method to create a huge puzzle database of over 2 million positions since it has a larger number of analyzed games. You may use that database to train your program. If you could find interesting/puzzle-potential positions without analyzing them first, you will make a big success.

Good luck and have fun with your adventure

[natopigro]

Ok, I didn't noticed that an EGTB has its own format.
So, my question is: where can I find, let say, the 5-man starting position database in fen format?

I will to use for sure an engine to evaluate the position. Starting from multiple evaluations I'll train a classifier.

The lichess database could spare me a lot of time, is it avaiable or querable somewhere?

Just an edit for my first post, point b): I've found how to modify the rosettacode to generate more interensting random position.

[phhnguyen]

Ok, I didn't noticed that an EGTB has its own format.
So, my question is: where can I find, let say, the 5-man starting position database in fen format?

No, you can't. A single endgame file may have too many positions to store as FEN strings. For example, KBPvKQ has 23 million positions. All 5-men are over 100 files.

As I have mentioned. You need to convert from indexes into chess positions, then FEN strings if you still need them.

Frankly speaking, if you can't find the right code somewhere, you may need at least a few months to develop that code because EGTB itself is a hard topic.

The result of that kind of puzzle generator is a bit boring IMHO: it may produce a huge number of quite-similar positions, say, a few hundred thousand mate-in-8 positions for a given material.

I will to use for sure an engine to evaluate the position. Starting from multiple evaluations I'll train a classifier.

The lichess database could spare me a lot of time, is it avaiable or querable somewhere?

https://database.lichess.org/#puzzles

[Ferdy]

Hi all,
I have in idea about how to define programmatically "what is a chess puzzle/composition", or, more precisely, what is not a puzzle.
I would like to train and test my idea but, as usually in data science, I miss data.

I am looking for:
1) a database puzzles, possibly in fen notation, for the train and validation phase;
2) a 5-6-7-8-man starting position (not the related solution tree) where I can discover new puzzles
3) and/or a chess position random generator having the mumber of pieces as parameter

Here is what I've been able to found up to now:
a) the syzyg database syzygy-tables.info, I can download the files but they are in .rtbz format. If I could read them probably I would be albe to extract the starting positions, but I can't. Any advice? (I can basically only work with python)
b) a random chess position generator rosettacode.org/wiki/Generate_random_chess_position, it could be useful to generate the starting position but there's no way to limit the number of pieces and the outcome positions are "always" unnatural
c) here https://chess.stackexchange.com/questio ... pgn-or-fen there are usefull links to Mate in 2-3-4 puzzles but I am not only looking for mate puzzles. Look for instance this problem
[fen]3R4/8/8/8/1r4kB/1B2K3/8/8 w - - 0 1[/fen]

My most pressing need are the point 2) and 3).

My goal is to find new puzzles, my dream is to find some new beatiful puzzles like these
https://en.chessbase.com/post/wonderful ... mpositions#
or this
[fen]5rk1/pp4pp/4p3/2R3Q1/3n4/2q4r/P1P2PPP/5RK1 b - - 0 1[/fen]

I have a github repo called chess-chiller that can be used to generate puzzles, it has some parameters to experiment. Basically it reads positions from games in the pgn file and if condition is meet it will save the position. This can be improved further by counting the attack and defense, mobility and others to capture some tactical motifs. If you plan on building a model to classify puzzles you can use it but it needs a label perhaps [1 to 10] where 10 is the best puzzle. It should be fun.

Also Tord has a blog on puzzle creation using chess.jl