txt: automated chess engine tuning

[Sergei S. Markoff]

One more difference is that I don't use QS search when tuning, only static eval. That's why I need quiet positions.
I have a lot of params to tune including PSQ, king shelter patterns and so on that's why I need a bigger positions set.

[jdart]

I use a shallow search (2 plies by default) so the positions do not have to be quiet.

Results are sensitive to the training set that is used. I used the output of my standard fast-time control rating gauntlet (36,000 games against 6 different opponents). That is over 1 million positions.

--Jon

[Sergei S. Markoff]

I use 20-core Xeon in parallel mode to calculate deviation. In needs about 9 sec. to calculate deviation for about 22 mln positions in training set.
I think when tuning some param you should be sure that it affects some representative subset of the whole position set and probably should be some diminishing return for the quality of each param when increasing this subset absolute size. Anyway I think the the most important enemies of this approach are: 1) overclassification when your param affects only a small subset of positions. I think we need to test this hypothesis using
a. Cross-validation (tune param on 75% of your set and that check if it improves fitting for the remaining 25%).
b. Produce new positions set from the tuned engine games and validate param value against it, because param change will obviously alter distribution of positions affected by param in the engine game positions set.
2. Nonlinear effects caused by search/eval interaction. Probably method b should help here.
I think it's a good idea to create learning "conveyor" with such a steps:

1. Play some thousands of games of the engine vs test engines set.
2. Extract positions.
3. Update learning positions set: extract positions from games, delete older positions in learning set replacing them for the new ones.
4. Tune eval weights vs new positions set.
5. Goto 1

[jdart]

Yes, I am doing that and so are most people I think.

--Jon