Tune cut margins with Texel/gaviota tuning method

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
Fabio Gobbato
Posts: 123
Joined: Fri Apr 11, 2014 8:45 am
Contact:

Tune cut margins with Texel/gaviota tuning method

Post by Fabio Gobbato » Thu Sep 11, 2014 11:02 am

In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?

Ferdy
Posts: 3816
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Tune cut margins with Texel/gaviota tuning method

Post by Ferdy » Thu Sep 11, 2014 11:46 am

Last time I did has scaling problem. Giving 200 ms per position will work for real games at tc 40/30sec, but at longer tc no more. Later I found some bugs in search and eval. Probably if I will try again it could be a gain in longer tc. I suggest to go ahead with your plan.

The other tuning variation I am toying around is using the move. Given some positions, let strong engines evaluate it, then tune parameters eval or search such that your engine will choose a move that is suggested by that strong engine.

Future plan will be a combination. Tune the parameters according to the best move and score.

User avatar
asanjuan
Posts: 210
Joined: Thu Sep 01, 2011 3:38 pm
Location: Seville, Spain

Re: Tune cut margins with Texel/gaviota tuning method

Post by asanjuan » Thu Sep 11, 2014 12:17 pm

Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?
I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.

If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.

You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.

Maybe: positions _searched *(1 + avg_error )^2 ?

Just my 2 cents.

PS: If you have success with this formula, let me know!!

User avatar
asanjuan
Posts: 210
Joined: Thu Sep 01, 2011 3:38 pm
Location: Seville, Spain

Re: Tune cut margins with Texel/gaviota tuning method

Post by asanjuan » Thu Sep 11, 2014 12:35 pm

asanjuan wrote: Maybe: positions _searched *(1 + avg_error )^2 ?
I realized that the formula deserves a try. I'm going to test it this evening!!

jdart
Posts: 3720
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Tune cut margins with Texel/gaviota tuning method

Post by jdart » Thu Sep 11, 2014 1:50 pm

I have had relatively poor results with this tuning method, even though it seems to work for others.

Many changes produce only very small to the mean error. So basically you are trying to find the lowest point on an almost flat multi-dimensional surface. When you do find that, IMO you can't have a lot of confidence that this is the global minimum. And I have also found that changes suggested in this way may fail to produce better results when actual game matches are run.

It would be nice if using a surrogate measure that is faster than running games matches did work, but so far I am not convinced that it does, in general.

--Jon

ZirconiumX
Posts: 1327
Joined: Sun Jul 17, 2011 9:14 am

Re: Tune cut margins with Texel/gaviota tuning method

Post by ZirconiumX » Thu Sep 11, 2014 2:20 pm

asanjuan wrote:
Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?
I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.

If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.

You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.

Maybe: positions _searched *(1 + avg_error )^2 ?

Just my 2 cents.

PS: If you have success with this formula, let me know!!
I think avg_error needs to be defined here. Average error of the score compared to alpha-beta? Compared to the end-result (a la eval tuning)? Average error compared to the search without the pruning method?

Matthew:out
Some believe in the almighty dollar.

I believe in the almighty printf statement.

tpetzke
Posts: 686
Joined: Thu Mar 03, 2011 3:57 pm
Location: Germany
Contact:

Re: Tune cut margins with Texel/gaviota tuning method

Post by tpetzke » Thu Sep 11, 2014 2:36 pm

I have had relatively poor results with this tuning method, even though it seems to work for others.
I used an optimization method that was able to come up with weights that produced a lower evaluation error than my original engine produced.

However the error optimized weights played worse than the original ones (tuned by game play). Not by a huge margin but beyond statistical doubt.

And one advice, don't feed in to many positions where the game outcome is a draw. The tuner might find out that the best eval function that produces the lowest error is "return 0".

Thomas...
Thomas...

=======
http://macechess.blogspot.com - iCE Chess Engine

User avatar
asanjuan
Posts: 210
Joined: Thu Sep 01, 2011 3:38 pm
Location: Seville, Spain

Re: Tune cut margins with Texel/gaviota tuning method

Post by asanjuan » Thu Sep 11, 2014 2:43 pm

ZirconiumX wrote:
asanjuan wrote:
Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?
I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.

If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.

You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.

Maybe: positions _searched *(1 + avg_error )^2 ?

Just my 2 cents.

PS: If you have success with this formula, let me know!!
I think avg_error needs to be defined here. Average error of the score compared to alpha-beta? Compared to the end-result (a la eval tuning)? Average error compared to the search without the pruning method?

Matthew:out
I mean: avg_error is the average error returned by the search , or taking Peter's formula:

E = 1/N * sum(i=1,N, (result - Sigmoid(search(pos, depth, alpha, beta)))^2)

depth must be fixed and deep enough to use the search parameters that we are trying to tune.

Don't you think it makes sense?

User avatar
asanjuan
Posts: 210
Joined: Thu Sep 01, 2011 3:38 pm
Location: Seville, Spain

Re: Tune cut margins with Texel/gaviota tuning method

Post by asanjuan » Fri Sep 12, 2014 11:44 am

asanjuan wrote: I mean: avg_error is the average error returned by the search , or taking Peter's formula:

E = 1/N * sum(i=1,N, (result - Sigmoid(search(pos, depth, alpha, beta)))^2)

depth must be fixed and deep enough to use the search parameters that we are trying to tune.

Don't you think it makes sense?


Yesterday, after implementing the average error for the search at fixed depth, I took 20.000 arbitrary positions and I was playing a little bit with some search margins. I could measure the average error without using the pruning, and after that, measuring the error while changing the margins.

Then I could DETECT that at depth = 3 the razor margin was a blunder, and so were the margins applied at depth 1 and 2 for the static fail high pruning (edit: I don't know if there is a consensus about the name for this tecnique: if depth ==1 and eval >beta + margin plus some conditions more, then return beta)

After changing manually the margins, the node count dropped while keeping a good average error, and it seems to play better. The match is running. Too early to tell, still within the error bars, but looks promising...

Maybe it is not a good tunning method, but a good detector for not making mistakes.

mvk
Posts: 589
Joined: Tue Jun 04, 2013 8:15 pm

Re: Tune cut margins with Texel/gaviota tuning method

Post by mvk » Fri Sep 12, 2014 6:29 pm

Fabio Gobbato wrote:In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?
I use it for search parameters, using the results of long (=300s) searches from randomized positions (~100k) as oracle. Then I gauge the fitness of 5s searches on these positions with varying search parameters. I have poor results trying to minimize the evaluation error. Instead I just count the number of matching moves.
[Account deleted]

Post Reply