Tune cut margins with Texel/gaviota tuning method

[Fabio Gobbato]

In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?

[Ferdy]

Last time I did has scaling problem. Giving 200 ms per position will work for real games at tc 40/30sec, but at longer tc no more. Later I found some bugs in search and eval. Probably if I will try again it could be a gain in longer tc. I suggest to go ahead with your plan.

The other tuning variation I am toying around is using the move. Given some positions, let strong engines evaluate it, then tune parameters eval or search such that your engine will choose a move that is suggested by that strong engine.

Future plan will be a combination. Tune the parameters according to the best move and score.

[asanjuan]

In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?

I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.

If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.

You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.

Maybe: positions _searched *(1 + avg_error )^2 ?

Just my 2 cents.

PS: If you have success with this formula, let me know!!

[asanjuan]

Maybe: positions _searched *(1 + avg_error )^2 ?

I realized that the formula deserves a try. I'm going to test it this evening!!

[jdart]

I have had relatively poor results with this tuning method, even though it seems to work for others.

Many changes produce only very small to the mean error. So basically you are trying to find the lowest point on an almost flat multi-dimensional surface. When you do find that, IMO you can't have a lot of confidence that this is the global minimum. And I have also found that changes suggested in this way may fail to produce better results when actual game matches are run.

It would be nice if using a surrogate measure that is faster than running games matches did work, but so far I am not convinced that it does, in general.

--Jon

[ZirconiumX]

In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?

I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.

If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.

You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.

Maybe: positions _searched *(1 + avg_error )^2 ?

Just my 2 cents.

PS: If you have success with this formula, let me know!!

I think avg_error needs to be defined here. Average error of the score compared to alpha-beta? Compared to the end-result (a la eval tuning)? Average error compared to the search without the pruning method?

Matthew:out

[tpetzke]

I have had relatively poor results with this tuning method, even though it seems to work for others.

I used an optimization method that was able to come up with weights that produced a lower evaluation error than my original engine produced.

However the error optimized weights played worse than the original ones (tuned by game play). Not by a huge margin but beyond statistical doubt.

And one advice, don't feed in to many positions where the game outcome is a draw. The tuner might find out that the best eval function that produces the lowest error is "return 0".

Thomas...

[asanjuan]

In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?

I am using the same method to tune evaluation, and I have been thinking in doing this kind of tunning too for the search, but I realized that it can't work with the usual error calculation.

If you prune, you introduce a new error, then, the minimization of the error function will produce margins big enough to not to introduce the new error, this is, big margins that will disable the pruning in practice.

You need a new error calculation, or a new "fitness function" to try to measure if the new margin will be ok for game play, or a function that takes into consideration the average error and the number of positions searched.
Whatever the function is, i don't know.

Maybe: positions _searched *(1 + avg_error )^2 ?

Just my 2 cents.

PS: If you have success with this formula, let me know!!

I think avg_error needs to be defined here. Average error of the score compared to alpha-beta? Compared to the end-result (a la eval tuning)? Average error compared to the search without the pruning method?

Matthew:out

I mean: avg_error is the average error returned by the search , or taking Peter's formula:

E = 1/N * sum(i=1,N, (result - Sigmoid(search(pos, depth, alpha, beta)))^2)

depth must be fixed and deep enough to use the search parameters that we are trying to tune.

Don't you think it makes sense?

[asanjuan]

I mean: avg_error is the average error returned by the search , or taking Peter's formula:

E = 1/N * sum(i=1,N, (result - Sigmoid(search(pos, depth, alpha, beta)))^2)

depth must be fixed and deep enough to use the search parameters that we are trying to tune.

Don't you think it makes sense?

Yesterday, after implementing the average error for the search at fixed depth, I took 20.000 arbitrary positions and I was playing a little bit with some search margins. I could measure the average error without using the pruning, and after that, measuring the error while changing the margins.

Then I could DETECT that at depth = 3 the razor margin was a blunder, and so were the margins applied at depth 1 and 2 for the static fail high pruning (edit: I don't know if there is a consensus about the name for this tecnique: if depth ==1 and eval >beta + margin plus some conditions more, then return beta)

After changing manually the margins, the node count dropped while keeping a good average error, and it seems to play better. The match is running. Too early to tell, still within the error bars, but looks promising...

Maybe it is not a good tunning method, but a good detector for not making mistakes.

[mvk]

In my engine I used a similar approach to Texel/Gaviota tuning method with good results.

I was wondering about tune with this method futility-delta pruning and razoring margin.

The advantage is that you can tune this values taking account the evaluation error of the search function of millions positions.
It should be quicker than play thousands and thousands of games.
In the next days I would try this thing.

Does anyone already try this approach and with what results?

I use it for search parameters, using the results of long (=300s) searches from randomized positions (~100k) as oracle. Then I gauge the fitness of 5s searches on these positions with varying search parameters. I have poor results trying to minimize the evaluation error. Instead I just count the number of matching moves.