Wanted: evolving Auto Tester for UCI

[mjlef]

OK, here is what I would like (and I am even willing to pay a little for it):

Autotester that runs like Arena, loading UCI chess engines.
Lets me specify what UCI variable to randomize, and the range (say "Pawn" from "70" to "120" and "Knight Mobility" from "0 to 10").
Program generated multiple "program" which vary only in the setting of those variables via the UCI options commands.
Program tests the various programs, and saves the ones that win the most, to then evolve new versions.
Repeat for X generations

Basically, genetics...only I am too lazy to write this myself.

Any takers? If something like this already exists, let me know.

Mark

[Steve Maughan]

Mark,

Great idea! I'd love to write it but unfortunately don't have the time due to work.

Steve

[Jaap Weidemann]

Hi Mark,

PolyGlot would be a good starting point for this. It would not take much effort to change it to suite your needs.

Jaap

[Odd Gunnar Malin]

Maybe I missunderstand, but this seems really simple.
I don't know Arena but with Shredder (Classic) or ChessPartner this is done in a few minutes.

Just create a program that write .eng file for Shredder or .cpevar files for ChesPartner.

If this is too much trouble, just create the files manualy, shouldn't take too much time either.

If Arena don't have ini files for engines, you can always use InBetween for this.

Odd Gunnar

[Dann Corbit]

OK, here is what I would like (and I am even willing to pay a little for it):

Autotester that runs like Arena, loading UCI chess engines.
Lets me specify what UCI variable to randomize, and the range (say "Pawn" from "70" to "120" and "Knight Mobility" from "0 to 10").
Program generated multiple "program" which vary only in the setting of those variables via the UCI options commands.
Program tests the various programs, and saves the ones that win the most, to then evolve new versions.
Repeat for X generations

Basically, genetics...only I am too lazy to write this myself.

Any takers? If something like this already exists, let me know.

Mark

It sounds a lot like this:
http://people.ict.usc.edu/~gomboc/publi ... rdance.pdf

Maybe you can ask Dave Gomboc for his research harness.

[Steve Maughan]

Dann,

I think Mark is referring to a genetic algorithm. This defines an engine as a set of genes (gene = UCI parameter). There would be a population of engines that would each play x number of games. This would be a generation. The engines would then "breed" to create the next generation where the engines with the engines that won the most games were more likely to pass their genes on. In this way you'd optimize the set of parameters for any given engine. Take a look at this article which explain genetic algorithms much better.

http://en.wikipedia.org/wiki/Genetic_algorithm

I think it's a great idea and I'm sure it would work with enough games.

Best regards,

Steve

[mjlef]

OK, here is what I would like (and I am even willing to pay a little for it):

Autotester that runs like Arena, loading UCI chess engines.
Lets me specify what UCI variable to randomize, and the range (say "Pawn" from "70" to "120" and "Knight Mobility" from "0 to 10").
Program generated multiple "program" which vary only in the setting of those variables via the UCI options commands.
Program tests the various programs, and saves the ones that win the most, to then evolve new versions.
Repeat for X generations

Basically, genetics...only I am too lazy to write this myself.

Any takers? If something like this already exists, let me know.

Mark

It sounds a lot like this:
http://people.ict.usc.edu/~gomboc/publi ... rdance.pdf

Maybe you can ask Dave Gomboc for his research harness.

The automated playing software they used was by Yngvi Björnsson, so I sent him an email. thanks!

[CRoberson]

Sort of did that.

Except:
1) Wrote a Simulated Annealer instead of a Genetic Algorithm (SA more appropriate).
2) Used it on test positions instead of games.

The end result was that it worked rather well but the learning curve flattened too quikly.
Efforts to fix it, made it perform better but the resulting engine played
worse in matches. Improved it a bit by reducing the number of terms
that were adjusted during training instead of all at once.

Been planning to do this with games - been doing it manually for
the past few months, but intelligent ? tuning instead of random.
I get 7 hours of matches out between adjustments. Even with
intelligent adjustments, convergance takes time. However, I've
gained 100 ELO in the past 2 weeks.

There are several neat side effects. One of which is the zeroing of
a well known good term. When that happens, you found a bug!

[Pradu]

There are several neat side effects. One of which is the zeroing of a well known good term. When that happens, you found a bug!

Did this by chance happen (without bugs) for backward pawn penalties in your program? I did some experiments for eval tuning in Buzz and the backward pawn penaltt averages were actually negative and the standard deviation in the population (I was using evolutionary algorithms) was rather high for this term.

BTW, congrats on your +100 ELO.

[CRoberson]

Backwards pawns are in a special case class.

A negative value means you have insufficient conditions for giving
the penalty.

There are lot of times in chess when it is good to have a backwards
pawn - various stonewall attacks and stonewall defences as
seen in several variations of the Dutch defence.