Tune cut margins with Texel/gaviota tuning method
Moderators: bob, hgm, Harvey Williamson
Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
 Fabio Gobbato
 Posts: 125
 Joined: Fri Apr 11, 2014 8:45 am
 Contact:
Tune cut margins with Texel/gaviota tuning method
In my engine I used a similar approach to Texel/Gaviota tuning method with good results.
I was wondering about tune with this method futilitydelta pruning and razoring margin.
The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.
Does anyone already try this approach and with what results?
I was wondering about tune with this method futilitydelta pruning and razoring margin.
The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.
Does anyone already try this approach and with what results?
Re: Tune cut margins with Texel/gaviota tuning method
Last time I did has scaling problem. Giving 200 ms per position will work for real games at tc 40/30sec, but at longer tc no more. Later I found some bugs in search and eval. Probably if I will try again it could be a gain in longer tc. I suggest to go ahead with your plan.
The other tuning variation I am toying around is using the move. Given some positions, let strong engines evaluate it, then tune parameters eval or search such that your engine will choose a move that is suggested by that strong engine.
Future plan will be a combination. Tune the parameters according to the best move and score.
The other tuning variation I am toying around is using the move. Given some positions, let strong engines evaluate it, then tune parameters eval or search such that your engine will choose a move that is suggested by that strong engine.
Future plan will be a combination. Tune the parameters according to the best move and score.
Re: Tune cut margins with Texel/gaviota tuning method
I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.
I was wondering about tune with this method futilitydelta pruning and razoring margin.
The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.
Does anyone already try this approach and with what results?
If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.
You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.
Maybe: positions _searched *(1 + avg_error )^2 ?
Just my 2 cents.
PS: If you have success with this formula, let me know!!
Re: Tune cut margins with Texel/gaviota tuning method
I realized that the formula deserves a try. I'm going to test it this evening!!asanjuan wrote: Maybe: positions _searched *(1 + avg_error )^2 ?

 Posts: 3825
 Joined: Fri Mar 10, 2006 4:23 am
 Location: http://www.arasanchess.org
Re: Tune cut margins with Texel/gaviota tuning method
I have had relatively poor results with this tuning method, even though it seems to work for others.
Many changes produce only very small to the mean error. So basically you are trying to find the lowest point on an almost flat multidimensional surface. When you do find that, IMO you can't have a lot of confidence that this is the global minimum. And I have also found that changes suggested in this way may fail to produce better results when actual game matches are run.
It would be nice if using a surrogate measure that is faster than running games matches did work, but so far I am not convinced that it does, in general.
Jon
Many changes produce only very small to the mean error. So basically you are trying to find the lowest point on an almost flat multidimensional surface. When you do find that, IMO you can't have a lot of confidence that this is the global minimum. And I have also found that changes suggested in this way may fail to produce better results when actual game matches are run.
It would be nice if using a surrogate measure that is faster than running games matches did work, but so far I am not convinced that it does, in general.
Jon

 Posts: 1327
 Joined: Sun Jul 17, 2011 9:14 am
Re: Tune cut margins with Texel/gaviota tuning method
I think avg_error needs to be defined here. Average error of the score compared to alphabeta? Compared to the endresult (a la eval tuning)? Average error compared to the search without the pruning method?asanjuan wrote:I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.
I was wondering about tune with this method futilitydelta pruning and razoring margin.
The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.
Does anyone already try this approach and with what results?
If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.
You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.
Maybe: positions _searched *(1 + avg_error )^2 ?
Just my 2 cents.
PS: If you have success with this formula, let me know!!
Matthew:out
Some believe in the almighty dollar.
I believe in the almighty printf statement.
I believe in the almighty printf statement.
Re: Tune cut margins with Texel/gaviota tuning method
I used an optimization method that was able to come up with weights that produced a lower evaluation error than my original engine produced.I have had relatively poor results with this tuning method, even though it seems to work for others.
However the error optimized weights played worse than the original ones (tuned by game play). Not by a huge margin but beyond statistical doubt.
And one advice, don't feed in to many positions where the game outcome is a draw. The tuner might find out that the best eval function that produces the lowest error is "return 0".
Thomas...
Re: Tune cut margins with Texel/gaviota tuning method
I mean: avg_error is the average error returned by the search , or taking Peter's formula:ZirconiumX wrote:I think avg_error needs to be defined here. Average error of the score compared to alphabeta? Compared to the endresult (a la eval tuning)? Average error compared to the search without the pruning method?asanjuan wrote:I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.
I was wondering about tune with this method futilitydelta pruning and razoring margin.
The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.
Does anyone already try this approach and with what results?
If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.
You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.
Maybe: positions _searched *(1 + avg_error )^2 ?
Just my 2 cents.
PS: If you have success with this formula, let me know!!
Matthew:out
E = 1/N * sum(i=1,N, (result  Sigmoid(search(pos, depth, alpha, beta)))^2)
depth must be fixed and deep enough to use the search parameters that we are trying to tune.
Don't you think it makes sense?
Re: Tune cut margins with Texel/gaviota tuning method
asanjuan wrote: I mean: avg_error is the average error returned by the search , or taking Peter's formula:
E = 1/N * sum(i=1,N, (result  Sigmoid(search(pos, depth, alpha, beta)))^2)
depth must be fixed and deep enough to use the search parameters that we are trying to tune.
Don't you think it makes sense?
Yesterday, after implementing the average error for the search at fixed depth, I took 20.000 arbitrary positions and I was playing a little bit with some search margins. I could measure the average error without using the pruning, and after that, measuring the error while changing the margins.
Then I could DETECT that at depth = 3 the razor margin was a blunder, and so were the margins applied at depth 1 and 2 for the static fail high pruning (edit: I don't know if there is a consensus about the name for this tecnique: if depth ==1 and eval >beta + margin plus some conditions more, then return beta)
After changing manually the margins, the node count dropped while keeping a good average error, and it seems to play better. The match is running. Too early to tell, still within the error bars, but looks promising...
Maybe it is not a good tunning method, but a good detector for not making mistakes.
Re: Tune cut margins with Texel/gaviota tuning method
I use it for search parameters, using the results of long (=300s) searches from randomized positions (~100k) as oracle. Then I gauge the fitness of 5s searches on these positions with varying search parameters. I have poor results trying to minimize the evaluation error. Instead I just count the number of matching moves.Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.
I was wondering about tune with this method futilitydelta pruning and razoring margin.
The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.
Does anyone already try this approach and with what results?
[Account deleted]