Page 1 of 4

Texel tuning Zurichess quiet like generator

Posted: Tue Aug 06, 2019 5:48 pm
by xr_a_y
I wonder if we have a tool to extract quiet positions from a lot of pgn and label them like it is done in Zurichess Texel tuning data. Yeah another lazy man question ...

Re: Texel tuning Zurichess quiet like generator

Posted: Tue Aug 06, 2019 6:48 pm
by jdart
xr_a_y wrote:
Tue Aug 06, 2019 5:48 pm
I wonder if we have a tool to extract quiet positions from a lot of pgn and label them like it is done in Zurichess Texel tuning data. Yeah another lazy man question ...
I have a program (pgnselect) based on Arasan source to extract positions from PGN files, but it does not only select quiescent ones.

I also have a python script to label positions by running games using cutechess-cli.

Sources:
https://github.com/jdart1/arasan-chess/ ... select.cpp
https://github.com/jdart1/arasan-chess/ ... sitions.py

some description:
https://arasanchess.org/programr.shtml

Re: Texel tuning Zurichess quiet like generator

Posted: Tue Aug 06, 2019 8:27 pm
by xr_a_y
Good source.

Here's my plan, I'm going to use all Minic tourney pgn, use your extractor, then extract quiet position using Minic -qsearch option and Minic -eval option, if both score are the same, I'll call the position quiet. After that i'll run your python script and modify it to run 3 matches from the quiet position at very fast TC like 40/1sec with stockfish and finally only affect a score if the 3 games have the same result. This is gonna tale some weeks ...

Re: Texel tuning Zurichess quiet like generator

Posted: Wed Aug 07, 2019 8:10 pm
by xr_a_y
So I changed my mind and implemented my own stuff for parsing pgn.
For each CCRL 40/40 game, I'm extracting all position between move 2 and end-6.
For each position I compare "material score", if the game is materially "even" ( < 80cp), I compare static score with qsearch score. If there are the same (diff < 30cp), I am keeping the position as quiet. It's going to take some hours ...

Re: Texel tuning Zurichess quiet like generator

Posted: Wed Aug 07, 2019 10:34 pm
by AndrewGrant
I have the following : https://gist.github.com/AndyGrant/03ffb ... 5c742da6bb

Few options at the top of the file. Does however pull both quiet and noisy positions.

Re: Texel tuning Zurichess quiet like generator

Posted: Thu Aug 08, 2019 5:51 am
by Ferdy
xr_a_y wrote:
Tue Aug 06, 2019 5:48 pm
I wonder if we have a tool to extract quiet positions from a lot of pgn and label them like it is done in Zurichess Texel tuning data. Yeah another lazy man question ...
Just a suggestion, new pgn-extract has a nice feature with option

Code: Select all

--quiescent N
https://www.cs.kent.ac.uk/people/staff/ ... /help.html

Code: Select all

pgn-extract --plylimit 20 --quiescent 4 --output mygame.pgn big.pgn
The last 4 plies has no checks, captures and promotions.
Example this output.


But what you need is not the end position but 4 plies before (takeback moves: Bb3, Nb6, Qe4, Qf6) like this.


There is a high probability that that position is quiet indeed because the next 4 plies in the game has no captures, checks and promotions. I remember Gaviota is doing something like this.

So how do you get the fen 4 plies before? There is also an option in pgn-extract where you can cut the pgn at specific plies.

Code: Select all

pgn-extract --dropply -4 --output final.pgn mygame.pgn
And typical pgn output is

Code: Select all

[Event "LAT-ch"]
[Site "LAT"]
[Date "1954.??.??"]
[Round "?"]
[White "Tal, Mihail"]
[Black "Paradniek"]
[Result "*"]
[FEN "r1b2rk1/pp1nqppp/2p5/8/2BP4/2R2N2/PPQ2PPP/5RK1 b - - 0 14"]
[SetUp "1"]

14...Qf6 15.Qe4 Nb6 16.Bb3 *

That FEN tag contains a value r1b2rk1/pp1nqppp/2p5/8/2BP4/2R2N2/PPQ2PPP/5RK1 b - - 0 14, which will be your target.


With couple of manipulations that would not be too difficult to extract.

Be warned however that by taking back N plies it is possible that the position is not quiet at all. Example after taking back N plies you may end up with


Black's knight is threatened by a pawn. One way to solve such position is to change the side to move (black doing the nullmove) and run an engine or Sf at small time to see its score and bestmove, if eval is high and bestmove is a capture that means there is a direct threat in 1 ply so don't save such position for training. This can be a bottleneck in the generation but the advantage of finding positions with successive quiet moves is also fast.

Re: Texel tuning Zurichess quiet like generator

Posted: Thu Aug 08, 2019 3:48 pm
by xr_a_y
Thanks for the last inputs.

In the mean time, I went on with my method.
So I made a program to parse 1.069.000 CCRL 40/40 games.
Keeping only the ones with two players > 2600 elo, that is 475.200 games.
I extracted all position from half move 16 to endgame-6 in all those games.
28.861.583 positions are "even in material" (50cp margin).
And only 1.178.625 are quiet in the sense I described before (margin of 30cp between static and qsearch score).

I can post the generated edp if someone is interested.

Re: Texel tuning Zurichess quiet like generator

Posted: Thu Aug 08, 2019 5:37 pm
by xr_a_y
Oops, as always a bug was present the real numbers are

Keeping only the ones with two players > 2600 elo, that is 475.200 games.
I extracted all position from half move 16 to endgame-6 in all those games.
4.131.929 positions are "even in material" (50cp margin).
And only 761.612 are quiet in the sense I described before (margin of 30cp between static and qsearch score).

Re: Texel tuning Zurichess quiet like generator

Posted: Thu Aug 08, 2019 6:43 pm
by fabianVDW
Hi, I would definitly be willing to throw these positions into the tuner of FabChess, but only if they are labeled?

Greetings,
Fabi

Re: Texel tuning Zurichess quiet like generator

Posted: Fri Aug 09, 2019 1:58 am
by jdart
Keeping only the ones with two players > 2600 elo
I don't think there is advantage in excluding games from weaker players. You want a variety of positions. The objective is to have a decent sample of possible positions that an actual search may evaluate and these will be quite varied and many of them uneven (for example, positions the q-search will cut off due to high eval).

--Jon