New Idea For Automated Tuning

[jordanbray]

I'm experimenting with a new idea for tuning a chess engine, I was wondering how you would predict this to run. Program is running in the background now, so I should have an answer (for my engine) in a few days.

The idea is to extract quiet positions from GM games. This is done by:
1) Making sure (after a proper search) that material is equal.
2) Making sure there is no checkmate.
3) After a large number of games at fast time control (using 100 games now), the expected value for white is between .35 and .65.

After all those games are finished, extract the expected win chance for white. From there, use Texcels tuning method only on the starting position.

After that has completed, ensure the elo of the engine has increased, and start all over.

What caveats would you expect me to run into using this method?

[Bloodbane]

If you do that you tune the evaluation function to handle reasonable, equal positions. Almost all positions encountered during the search are not so. So it might or might not work. I'd have more confidence if the tuning set contained unbalanced positions, possibly even dead won or lost ones. Also, your proposed method seems to be pretty slow to run, so I'm not that confident it's going to give good results fast enough.

[jordanbray]

If you do that you tune the evaluation function to handle reasonable, equal positions. Almost all positions encountered during the search are not so. So it might or might not work. I'd have more confidence if the tuning set contained unbalanced positions, possibly even dead won or lost ones. Also, your proposed method seems to be pretty slow to run, so I'm not that confident it's going to give good results fast enough.

Yea, that is likely to happen. IMHO, though, most easily winning positions are going to be easier to compute as such for the engine, so I'm not as worried about it. I've had a lot of failures with automated tuning, so this is the next iteration of that idea.

I may remove the condition on the nature of the position. We'll see how this set of tuning goes...

[Ferdy]

I'm experimenting with a new idea for tuning a chess engine, I was wondering how you would predict this to run. Program is running in the background now, so I should have an answer (for my engine) in a few days.

The idea is to extract quiet positions from GM games. This is done by:
1) Making sure (after a proper search) that material is equal.
2) Making sure there is no checkmate.
3) After a large number of games at fast time control (using 100 games now), the expected value for white is between .35 and .65.

After all those games are finished, extract the expected win chance for white. From there, use Texcels tuning method only on the starting position.

After that has completed, ensure the elo of the engine has increased, and start all over.

What caveats would you expect me to run into using this method?

Since you are running full games, did you try CLOP?

[Ferdy]

I've had a lot of failures with automated tuning, so this is the next iteration of that idea.

One quick way to test the tuning system is by assigning crazy values to your parameters, say starting values for pawnValue = 250cp, knightValue = 150 and so on. See if the tuner could come up with a good combination of values.
There are situations that the tuner could no longer improve as your parameters were already good. Or perhaps the parameters exposed for tuning are not enough to improve the engine performance.

[jordanbray]

In case anyone else has a similar idea, this didn't work for me. I was trying to avoid all the "traps" of automated tuning, but still seemed to fall into them.

In this case, I found that tuning only 2000 positions was not enough to properly compute many of the values. The scores for each position was more accurate (than if there was only one game), but the smaller number of positions lead to some values not getting modified, and others having too much weight associated with them.

I do, overall, like the idea of using positions that have no tactics for automated tuning, but I can't seem to prove thats a good idea.

Its a shame I can never seem to get automated tuning to work. It seems like it should in principle, but I always seem to be overfitting the problem. I am now back to hand-tuning, because no automated tuning testing has proved an elo increase.

[Ferdy]

In case anyone else has a similar idea, this didn't work for me. I was trying to avoid all the "traps" of automated tuning, but still seemed to fall into them.

In this case, I found that tuning only 2000 positions was not enough to properly compute many of the values. The scores for each position was more accurate (than if there was only one game), but the smaller number of positions lead to some values not getting modified, and others having too much weight associated with them.

I do, overall, like the idea of using positions that have no tactics for automated tuning, but I can't seem to prove thats a good idea.

Its a shame I can never seem to get automated tuning to work. It seems like it should in principle, but I always seem to be overfitting the problem. I am now back to hand-tuning, because no automated tuning testing has proved an elo increase.

One thing I observed is that it is difficult to improve by auto tuning if you have a bug somewhere. Of course you will not know this today, later it is good to come back auto tuning again.

[Engin]

Nor with CLOP, Texel or SPSA tuning method is really working !

some values seems to working, some are never working or believable that they can be work in games.

Better using you own mind and tuning with hand coded values and testing games with them to see how they performs.

[Engin]

i lost to much time with tuning methods in the last month ago !

if somebody is said he had success with tuning then its a lie or he made something other changes that made a luck progress.