txt: automated chess engine tuning

[Sergei S. Markoff]

It's funny that in SmarThink I'm using nearly the same tuning method.
I have about 30 mln relatively "quiet" positions extracted from SmarThink games. I used a very simple criteria: position is "quiet" if the move for this position was not a capture or promotion.

Then I use 1.0 + pow(10.0, -((double)eval) / 400.0) sigmoid, it's funny that thare are no K coeff in my sigmoid whan Texel has and that's only difference.

But what I have is error weight that depends on number of plies until game end. The main idea is that we're trying to obtain some approximation of "true" position eval from game result, but game result can be caused by some error made *after* a position occurence in the game.

remaining plies 1.0 - mean absolute error
0 0,91502994
1 0,91340348
2 0,90854397
3 0,90694811
4 0,90077156
5 0,89893914
6 0,89295661
7 0,89087872
8 0,88610333
9 0,88570349
10 0,87991417
11 0,87757882
12 0,87145933
13 0,86850624
14 0,86266026
15 0,85930579
16 0,85357369
...

So I just used this value to weight errors.

[brtzsnr]

I haven't seen any improvement tuning with more than 500k positions. I normally tune with 1mln and just before releasing I retune with 2mln positions.

I also use captures (but not checks). Looking at quiet moves only is a significant regression for zurichess (20-30 ELO).

K was discussed in another thread. Is basically a scaling coefficient if you only want to add new feautures without retuning the old parameters. I usually tune everything so I don't care.

[Sergei S. Markoff]

One more difference is that I don't use QS search when tuning, only static eval. That's why I need quiet positions.
I have a lot of params to tune including PSQ, king shelter patterns and so on that's why I need a bigger positions set.

[jdart]

I use a shallow search (2 plies by default) so the positions do not have to be quiet.

Results are sensitive to the training set that is used. I used the output of my standard fast-time control rating gauntlet (36,000 games against 6 different opponents). That is over 1 million positions.

--Jon

[Sergei S. Markoff]

I use 20-core Xeon in parallel mode to calculate deviation. In needs about 9 sec. to calculate deviation for about 22 mln positions in training set.
I think when tuning some param you should be sure that it affects some representative subset of the whole position set and probably should be some diminishing return for the quality of each param when increasing this subset absolute size. Anyway I think the the most important enemies of this approach are: 1) overclassification when your param affects only a small subset of positions. I think we need to test this hypothesis using
a. Cross-validation (tune param on 75% of your set and that check if it improves fitting for the remaining 25%).
b. Produce new positions set from the tuned engine games and validate param value against it, because param change will obviously alter distribution of positions affected by param in the engine game positions set.
2. Nonlinear effects caused by search/eval interaction. Probably method b should help here.
I think it's a good idea to create learning "conveyor" with such a steps:

1. Play some thousands of games of the engine vs test engines set.
2. Extract positions.
3. Update learning positions set: extract positions from games, delete older positions in learning set replacing them for the new ones.
4. Tune eval weights vs new positions set.
5. Goto 1

[jdart]

Yes, I am doing that and so are most people I think.

--Jon