Texel tuning Zurichess quiet like generator

[xr_a_y]

I wonder if we have a tool to extract quiet positions from a lot of pgn and label them like it is done in Zurichess Texel tuning data. Yeah another lazy man question ...

[jdart]

I wonder if we have a tool to extract quiet positions from a lot of pgn and label them like it is done in Zurichess Texel tuning data. Yeah another lazy man question ...

I have a program (pgnselect) based on Arasan source to extract positions from PGN files, but it does not only select quiescent ones.

I also have a python script to label positions by running games using cutechess-cli.

Sources:
https://github.com/jdart1/arasan-chess/ ... select.cpp
https://github.com/jdart1/arasan-chess/ ... sitions.py

some description:
https://arasanchess.org/programr.shtml

[xr_a_y]

Good source.

Here's my plan, I'm going to use all Minic tourney pgn, use your extractor, then extract quiet position using Minic -qsearch option and Minic -eval option, if both score are the same, I'll call the position quiet. After that i'll run your python script and modify it to run 3 matches from the quiet position at very fast TC like 40/1sec with stockfish and finally only affect a score if the 3 games have the same result. This is gonna tale some weeks ...

[xr_a_y]

So I changed my mind and implemented my own stuff for parsing pgn.
For each CCRL 40/40 game, I'm extracting all position between move 2 and end-6.
For each position I compare "material score", if the game is materially "even" ( < 80cp), I compare static score with qsearch score. If there are the same (diff < 30cp), I am keeping the position as quiet. It's going to take some hours ...

[AndrewGrant]

I have the following : https://gist.github.com/AndyGrant/03ffb ... 5c742da6bb

Few options at the top of the file. Does however pull both quiet and noisy positions.

[Ferdy]

I wonder if we have a tool to extract quiet positions from a lot of pgn and label them like it is done in Zurichess Texel tuning data. Yeah another lazy man question ...

Just a suggestion, new pgn-extract has a nice feature with option

Code: Select all

--quiescent N

https://www.cs.kent.ac.uk/people/staff/ ... /help.html

Code: Select all

pgn-extract --plylimit 20 --quiescent 4 --output mygame.pgn big.pgn

The last 4 plies has no checks, captures and promotions.
Example this output.
[pgn][Event "LAT-ch"] [Site "LAT"] [Date "1954.??.??"] [Round "?"] [White "Tal, Mihail"] [Black "Paradniek"] [Result "*"] [ECO "D68"] 1.d4 d5 2.c4 e6 3.Nc3 Nf6 4.Nf3 Nbd7 5.Bg5 Be7 6.e3 c6 7.Bd3 O-O 8.O-O dxc4 9.Bxc4 Nd5 10.Bxe7 Qxe7 11.Rc1 Nxc3 12.Rxc3 e5 13.Qc2 exd4 14.exd4 Qf6 15.Qe4 Nb6 16.Bb3 *[/pgn]

But what you need is not the end position but 4 plies before (takeback moves: Bb3, Nb6, Qe4, Qf6) like this.

[d]r1b2rk1/pp1nqppp/2p5/8/2BP4/2R2N2/PPQ2PPP/5RK1 b - - 0 14
There is a high probability that that position is quiet indeed because the next 4 plies in the game has no captures, checks and promotions. I remember Gaviota is doing something like this.

So how do you get the fen 4 plies before? There is also an option in pgn-extract where you can cut the pgn at specific plies.

Code: Select all

pgn-extract --dropply -4 --output final.pgn mygame.pgn

And typical pgn output is

Code: Select all

[Event "LAT-ch"]
[Site "LAT"]
[Date "1954.??.??"]
[Round "?"]
[White "Tal, Mihail"]
[Black "Paradniek"]
[Result "*"]
[FEN "r1b2rk1/pp1nqppp/2p5/8/2BP4/2R2N2/PPQ2PPP/5RK1 b - - 0 14"]
[SetUp "1"]

14...Qf6 15.Qe4 Nb6 16.Bb3 *

That FEN tag contains a value r1b2rk1/pp1nqppp/2p5/8/2BP4/2R2N2/PPQ2PPP/5RK1 b - - 0 14, which will be your target.

[d]r1b2rk1/pp1nqppp/2p5/8/2BP4/2R2N2/PPQ2PPP/5RK1 b - - 0 14
With couple of manipulations that would not be too difficult to extract.

Be warned however that by taking back N plies it is possible that the position is not quiet at all. Example after taking back N plies you may end up with

[d]r1bq1rk1/pp3ppp/2p1pn2/b3P3/2B5/P1N1P3/1P3PPP/R1BQ1RK1 b - - 0 11
Black's knight is threatened by a pawn. One way to solve such position is to change the side to move (black doing the nullmove) and run an engine or Sf at small time to see its score and bestmove, if eval is high and bestmove is a capture that means there is a direct threat in 1 ply so don't save such position for training. This can be a bottleneck in the generation but the advantage of finding positions with successive quiet moves is also fast.

[xr_a_y]

Thanks for the last inputs.

In the mean time, I went on with my method.
So I made a program to parse 1.069.000 CCRL 40/40 games.
Keeping only the ones with two players > 2600 elo, that is 475.200 games.
I extracted all position from half move 16 to endgame-6 in all those games.
28.861.583 positions are "even in material" (50cp margin).
And only 1.178.625 are quiet in the sense I described before (margin of 30cp between static and qsearch score).

I can post the generated edp if someone is interested.

[xr_a_y]

Oops, as always a bug was present the real numbers are

Keeping only the ones with two players > 2600 elo, that is 475.200 games.
I extracted all position from half move 16 to endgame-6 in all those games.
4.131.929 positions are "even in material" (50cp margin).
And only 761.612 are quiet in the sense I described before (margin of 30cp between static and qsearch score).

[fabianVDW]

Hi, I would definitly be willing to throw these positions into the tuner of FabChess, but only if they are labeled?

Greetings,
Fabi

[jdart]

Keeping only the ones with two players > 2600 elo

I don't think there is advantage in excluding games from weaker players. You want a variety of positions. The objective is to have a decent sample of possible positions that an actual search may evaluate and these will be quite varied and many of them uneven (for example, positions the q-search will cut off due to high eval).

--Jon