Sampling is ok for tuning two parameters or so. Otherwise it is terribly slow.
O wait if you tune it badly it generalizes better. So it will do better on evaluating unseen positions. Somewhere there is an optimum between bad tuning and 'overtuning'.
Other constraint is that tuning should not cost too much time so better use hill climbing with restart. Or genetic algorithm (with restart?)
tuning for the uninformed
Moderators: hgm, Rebel, chrisw
-
- Posts: 433
- Joined: Fri Jan 16, 2015 4:02 pm
Re: tuning for the uninformed
I tried pearson correlation in the past, but it's not a good measure. Right now my experiments with evolving the eval parameters use texel tuning on a set of 100K positions (so I can train ~6000 models per day).
In terms of the error rate, I got very close to the hand tuned model (stable version). If you want to see the fully automatic trained eval with almost 0 human intervention check [1] or [2]. The evolved version is 100 Elo weaker than the stable version of Zurichess. I need to add back the pawns cache, though.
[1] https://bitbucket.org/brtzsnr/zurichess ... ew-default
[2] https://bitbucket.org/brtzsnr/zurichess ... ew-default
In terms of the error rate, I got very close to the hand tuned model (stable version). If you want to see the fully automatic trained eval with almost 0 human intervention check [1] or [2]. The evolved version is 100 Elo weaker than the stable version of Zurichess. I need to add back the pawns cache, though.
[1] https://bitbucket.org/brtzsnr/zurichess ... ew-default
[2] https://bitbucket.org/brtzsnr/zurichess ... ew-default
zurichess - http://www.zurichess.xyz
Re: tuning for the uninformed
Yeah I looked at texel tuning but it looked rather complicated. This pearson was implemented in an hour during the morning commute
Re: tuning for the uninformed
If anyone is willing to explain the Texel tuning method tht would be great!
Sofar I understand I have to let it play (well, run QS + eval on FENs) millions of games and then do something with the evaluation-value. But what? I don't understand the wiki explanation.
Sofar I understand I have to let it play (well, run QS + eval on FENs) millions of games and then do something with the evaluation-value. But what? I don't understand the wiki explanation.
-
- Posts: 160
- Joined: Sat Jan 28, 2017 1:29 pm
- Location: The Netherlands
Re: tuning for the uninformed
The basic idea is pretty simple: calculate the error of the evaluation when it is compared to the actual outcome of the positions. Lower a particular evaluation parameter and check if the error has improved, if not, higher the parameter, if again not improved, keep the original value. Do this for all parameters until you have reached the lowest error.flok wrote:If anyone is willing to explain the Texel tuning method tht would be great!
Sofar I understand I have to let it play (well, run QS + eval on FENs) millions of games and then do something with the evaluation-value. But what? I don't understand the wiki explanation.
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: tuning for the uninformed
I don't know how to get an estimate of the value of a position without using other engines.
Maybe use your own engine and re-search it on shallow depth. But who will say that that gives a right estimate if your current evaluation is bad.
You can not evaluate every test position manually for that is too much work.
Maybe use your own engine and re-search it on shallow depth. But who will say that that gives a right estimate if your current evaluation is bad.
You can not evaluate every test position manually for that is too much work.
-
- Posts: 160
- Joined: Sat Jan 28, 2017 1:29 pm
- Location: The Netherlands
Re: tuning for the uninformed
quiet-labeled.epd contains the outcome of every positionHenk wrote:I don't know how to get an estimate of the value of a position without using other engines.
Maybe use your own engine and re-search it on shallow depth. But who will say that that gives a right estimate if your current evaluation is bad.
You can not evaluate every test position manually for that is too much work.
-
- Posts: 7220
- Joined: Mon May 27, 2013 10:31 am
Re: tuning for the uninformed
Don't understand. What is quiet-labeled.epd?sandermvdb wrote:quiet-labeled.epd contains the outcome of every positionHenk wrote:I don't know how to get an estimate of the value of a position without using other engines.
Maybe use your own engine and re-search it on shallow depth. But who will say that that gives a right estimate if your current evaluation is bad.
You can not evaluate every test position manually for that is too much work.
-
- Posts: 160
- Joined: Sat Jan 28, 2017 1:29 pm
- Location: The Netherlands
Re: tuning for the uninformed
Sorry, that is one of the testsets by Alexandru Mosoi, the author of Zurichess. It conains quiet positions including the outcome of the game.Henk wrote:Don't understand. What is quiet-labeled.epd?sandermvdb wrote:quiet-labeled.epd contains the outcome of every positionHenk wrote: Maybe use your own engine and re-search it on shallow depth. But who will say that that gives a right estimate if your current evaluation is bad.
You can not evaluate every test position manually for that is too much work.
Re: tuning for the uninformed
What do you mean by that?sandermvdb wrote:The basic idea is pretty simple: calculate the error of the evaluation when it is compared to the actual outcome of the positions.
Do you mean the following:
- fen as input
- calc move with an eval val
- calc eval of the move that should've been moved
- compare these two (how? percentual difference? or what?)