One more difference is that I don't use QS search when tuning, only static eval. That's why I need quiet positions.
I have a lot of params to tune including PSQ, king shelter patterns and so on that's why I need a bigger positions set.
txt: automated chess engine tuning
Moderators: hgm, Rebel, chrisw
-
- Posts: 227
- Joined: Mon Sep 12, 2011 11:27 pm
- Location: Moscow, Russia
Re: txt: automated chess engine tuning
The Force Be With You!
-
- Posts: 4367
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: txt: automated chess engine tuning
I use a shallow search (2 plies by default) so the positions do not have to be quiet.
Results are sensitive to the training set that is used. I used the output of my standard fast-time control rating gauntlet (36,000 games against 6 different opponents). That is over 1 million positions.
--Jon
Results are sensitive to the training set that is used. I used the output of my standard fast-time control rating gauntlet (36,000 games against 6 different opponents). That is over 1 million positions.
--Jon
-
- Posts: 227
- Joined: Mon Sep 12, 2011 11:27 pm
- Location: Moscow, Russia
Re: txt: automated chess engine tuning
I use 20-core Xeon in parallel mode to calculate deviation. In needs about 9 sec. to calculate deviation for about 22 mln positions in training set.
I think when tuning some param you should be sure that it affects some representative subset of the whole position set and probably should be some diminishing return for the quality of each param when increasing this subset absolute size. Anyway I think the the most important enemies of this approach are: 1) overclassification when your param affects only a small subset of positions. I think we need to test this hypothesis using
a. Cross-validation (tune param on 75% of your set and that check if it improves fitting for the remaining 25%).
b. Produce new positions set from the tuned engine games and validate param value against it, because param change will obviously alter distribution of positions affected by param in the engine game positions set.
2. Nonlinear effects caused by search/eval interaction. Probably method b should help here.
I think it's a good idea to create learning "conveyor" with such a steps:
1. Play some thousands of games of the engine vs test engines set.
2. Extract positions.
3. Update learning positions set: extract positions from games, delete older positions in learning set replacing them for the new ones.
4. Tune eval weights vs new positions set.
5. Goto 1
I think when tuning some param you should be sure that it affects some representative subset of the whole position set and probably should be some diminishing return for the quality of each param when increasing this subset absolute size. Anyway I think the the most important enemies of this approach are: 1) overclassification when your param affects only a small subset of positions. I think we need to test this hypothesis using
a. Cross-validation (tune param on 75% of your set and that check if it improves fitting for the remaining 25%).
b. Produce new positions set from the tuned engine games and validate param value against it, because param change will obviously alter distribution of positions affected by param in the engine game positions set.
2. Nonlinear effects caused by search/eval interaction. Probably method b should help here.
I think it's a good idea to create learning "conveyor" with such a steps:
1. Play some thousands of games of the engine vs test engines set.
2. Extract positions.
3. Update learning positions set: extract positions from games, delete older positions in learning set replacing them for the new ones.
4. Tune eval weights vs new positions set.
5. Goto 1
The Force Be With You!
-
- Posts: 4367
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: txt: automated chess engine tuning
Yes, I am doing that and so are most people I think.
--Jon
--Jon