New Idea For Automated Tuning

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
jordanbray
Posts: 51
Joined: Mon Aug 11, 2014 1:01 am

New Idea For Automated Tuning

Post by jordanbray » Sat May 16, 2015 6:04 pm

I'm experimenting with a new idea for tuning a chess engine, I was wondering how you would predict this to run. Program is running in the background now, so I should have an answer (for my engine) in a few days.

The idea is to extract quiet positions from GM games. This is done by:
1) Making sure (after a proper search) that material is equal.
2) Making sure there is no checkmate.
3) After a large number of games at fast time control (using 100 games now), the expected value for white is between .35 and .65.

After all those games are finished, extract the expected win chance for white. From there, use Texcels tuning method only on the starting position.

After that has completed, ensure the elo of the engine has increased, and start all over.

What caveats would you expect me to run into using this method?

User avatar
Bloodbane
Posts: 154
Joined: Thu Oct 03, 2013 2:17 pm

Re: New Idea For Automated Tuning

Post by Bloodbane » Sat May 16, 2015 6:13 pm

If you do that you tune the evaluation function to handle reasonable, equal positions. Almost all positions encountered during the search are not so. So it might or might not work. I'd have more confidence if the tuning set contained unbalanced positions, possibly even dead won or lost ones. Also, your proposed method seems to be pretty slow to run, so I'm not that confident it's going to give good results fast enough.
Functional programming combines the flexibility and power of abstract mathematics with the intuitive clarity of abstract mathematics.
https://github.com/mAarnos

jordanbray
Posts: 51
Joined: Mon Aug 11, 2014 1:01 am

Re: New Idea For Automated Tuning

Post by jordanbray » Sat May 16, 2015 7:54 pm

Bloodbane wrote:If you do that you tune the evaluation function to handle reasonable, equal positions. Almost all positions encountered during the search are not so. So it might or might not work. I'd have more confidence if the tuning set contained unbalanced positions, possibly even dead won or lost ones. Also, your proposed method seems to be pretty slow to run, so I'm not that confident it's going to give good results fast enough.
Yea, that is likely to happen. IMHO, though, most easily winning positions are going to be easier to compute as such for the engine, so I'm not as worried about it. I've had a lot of failures with automated tuning, so this is the next iteration of that idea.

I may remove the condition on the nature of the position. We'll see how this set of tuning goes...

Ferdy
Posts: 3645
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: New Idea For Automated Tuning

Post by Ferdy » Sun May 17, 2015 1:10 am

jordanbray wrote:I'm experimenting with a new idea for tuning a chess engine, I was wondering how you would predict this to run. Program is running in the background now, so I should have an answer (for my engine) in a few days.

The idea is to extract quiet positions from GM games. This is done by:
1) Making sure (after a proper search) that material is equal.
2) Making sure there is no checkmate.
3) After a large number of games at fast time control (using 100 games now), the expected value for white is between .35 and .65.

After all those games are finished, extract the expected win chance for white. From there, use Texcels tuning method only on the starting position.

After that has completed, ensure the elo of the engine has increased, and start all over.

What caveats would you expect me to run into using this method?
Since you are running full games, did you try CLOP?

Ferdy
Posts: 3645
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: New Idea For Automated Tuning

Post by Ferdy » Sun May 17, 2015 2:15 am

jordanbray wrote:I've had a lot of failures with automated tuning, so this is the next iteration of that idea.
One quick way to test the tuning system is by assigning crazy values to your parameters, say starting values for pawnValue = 250cp, knightValue = 150 and so on. See if the tuner could come up with a good combination of values.
There are situations that the tuner could no longer improve as your parameters were already good. Or perhaps the parameters exposed for tuning are not enough to improve the engine performance.

jordanbray
Posts: 51
Joined: Mon Aug 11, 2014 1:01 am

Re: New Idea For Automated Tuning

Post by jordanbray » Sat May 23, 2015 4:34 pm

In case anyone else has a similar idea, this didn't work for me. I was trying to avoid all the "traps" of automated tuning, but still seemed to fall into them.

In this case, I found that tuning only 2000 positions was not enough to properly compute many of the values. The scores for each position was more accurate (than if there was only one game), but the smaller number of positions lead to some values not getting modified, and others having too much weight associated with them.

I do, overall, like the idea of using positions that have no tactics for automated tuning, but I can't seem to prove thats a good idea.

Its a shame I can never seem to get automated tuning to work. It seems like it should in principle, but I always seem to be overfitting the problem. I am now back to hand-tuning, because no automated tuning testing has proved an elo increase.

Ferdy
Posts: 3645
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: New Idea For Automated Tuning

Post by Ferdy » Sat May 23, 2015 4:47 pm

jordanbray wrote:In case anyone else has a similar idea, this didn't work for me. I was trying to avoid all the "traps" of automated tuning, but still seemed to fall into them.

In this case, I found that tuning only 2000 positions was not enough to properly compute many of the values. The scores for each position was more accurate (than if there was only one game), but the smaller number of positions lead to some values not getting modified, and others having too much weight associated with them.

I do, overall, like the idea of using positions that have no tactics for automated tuning, but I can't seem to prove thats a good idea.

Its a shame I can never seem to get automated tuning to work. It seems like it should in principle, but I always seem to be overfitting the problem. I am now back to hand-tuning, because no automated tuning testing has proved an elo increase.
One thing I observed is that it is difficult to improve by auto tuning if you have a bug somewhere. Of course you will not know this today, later it is good to come back auto tuning again.

Engin
Posts: 831
Joined: Mon Jan 05, 2009 6:40 pm
Location: Germany
Full name: Engin Üstün
Contact:

Re: New Idea For Automated Tuning

Post by Engin » Sat May 30, 2015 5:30 pm

Nor with CLOP, Texel or SPSA tuning method is really working !

some values seems to working, some are never working or believable that they can be work in games.

Better using you own mind and tuning with hand coded values and testing games with them to see how they performs.

Engin
Posts: 831
Joined: Mon Jan 05, 2009 6:40 pm
Location: Germany
Full name: Engin Üstün
Contact:

Re: New Idea For Automated Tuning

Post by Engin » Sat May 30, 2015 5:52 pm

i lost to much time with tuning methods in the last month ago !

if somebody is said he had success with tuning then its a lie or he made something other changes that made a luck progress.

Post Reply