CLOP for Noisy Black-Box Parameter Optimization

[Zlaire]

I am curious to take a look if you can send your .dat and .clop file to me.

Rémi

Not sure where to send them so uploaded to my site:

http://apptinus.se/KingEval.dat
http://apptinus.se/KingEval.clop

p3 is ampli and p4 bias for my king ending PST that looks like this:

Code: Select all

-20 -15 -10 -10 -10 -10 -15 -20
-15  -5   0   0   0   0  -5 -15
-10   0   5   5   5   5   0 -10
-10   0   5  10  10   5   0 -10
-10   0   5  10  10   5   0 -10
-10   0   5   5   5   5   0 -10
-15  -5   0   0   0   0  -5 -15
-20 -15 -10 -10 -10 -10 -15 -20

Due to some silly coding the ampli and bias is carried out in sequence, so

Code: Select all

newValue = oldValue + 0 + oldValue * amp / 100
newValue = newValue + bias + newValue * amp /100

Shouldn't matter, but when writing this I realized something.. can I be sure that p1-p11 are sent to the engine in order? If p4 would come before p3 it would mess everything up... with some weird results I'm sure.

[Rémi Coulom]

It seems your psq has only 7 different values. I don't understand how you can make it 11 parameters. What are the meanings of P1-P11?

Rémi

[Zlaire]

It seems your psq has only 7 different values. I don't understand how you can make it 11 parameters. What are the meanings of P1-P11?

It's not only psq in those 11 parameters. This is what they represent:

p1 - ampli for king psq, non ending
p2 - bias for king psg, non ending
p3 - ampli for king psq, ending
p4 - bias for king psq, ending
p5 - ampli for king attack table (king attack evaluation)
p6 - bias for king attack table
p7 - Missing pawn cover constant
p8 - Missing bishop finachetto constant
p9 - Pawn storm multiplier
p10 - Moved f-pawn before castle penalty
p11 - Moved pawn in front of castled king penalty multiplier

So p1+p2, p3+p4 and p5+p6 adjust three different piece square tables. The others are more normal values (and are looking fairly reasonable).

[Rémi Coulom]

It seems your psq has only 7 different values. I don't understand how you can make it 11 parameters. What are the meanings of P1-P11?

It's not only psq in those 11 parameters. This is what they represent:

p1 - ampli for king psq, non ending
p2 - bias for king psg, non ending
p3 - ampli for king psq, ending
p4 - bias for king psq, ending
p5 - ampli for king attack table (king attack evaluation)
p6 - bias for king attack table
p7 - Missing pawn cover constant
p8 - Missing bishop finachetto constant
p9 - Pawn storm multiplier
p10 - Moved f-pawn before castle penalty
p11 - Moved pawn in front of castled king penalty multiplier

So p1+p2, p3+p4 and p5+p6 adjust three different piece square tables. The others are more normal values (and are looking fairly reasonable).

I see. CLOP is clearly over-fitting the data: it has only 48 central samples, with 100% winning rate. This may happen when tuning many parameters. I will fix it in the next version. That should not be very difficult to do, but I have little time now. Probably in January.

Rémi

[Zlaire]

I see. CLOP is clearly over-fitting the data: it has only 48 central samples, with 100% winning rate. This may happen when tuning many parameters. I will fix it in the next version. That should not be very difficult to do, but I have little time now. Probably in January.

How would you suggest I run the tuning instead for now?

Simply reduce the number of parameters?

(thank you for a great tool by the way, this is the first problem I've run into with it)

[Rémi Coulom]

I see. CLOP is clearly over-fitting the data: it has only 48 central samples, with 100% winning rate. This may happen when tuning many parameters. I will fix it in the next version. That should not be very difficult to do, but I have little time now. Probably in January.

How would you suggest I run the tuning instead for now?

Simply reduce the number of parameters?

(thank you for a great tool by the way, this is the first problem I've run into with it)

Yes. Just tune p3 and p4 alone.

Since so many people are reporting similar problems, I may run a few experiments this week-end to improve CLOP in high dimensions. I have some very simple ideas, like limiting the number of iterations, that may work well and would require no programming efforts. I'll report later about that.

Rémi

[Sergei S. Markoff]

Some my ideas on tuning: http://www.talkchess.com/forum/viewtopi ... 83&t=40964

[Zlaire]

1500 games in with only p3 and p4 and it seems to be settling in at pretty much the same values.

Could this have to do with two parameters adjusting a full psq? Anyone else getting similar results?

Uploaded the .dat and .clop files again.

http://apptinus.se/KingEval2.clop
http://apptinus.se/KingEval2.dat

[Zlaire]

Oh, never mind I think. Did some changes to the parameter input handling I didn't account for. Looks much more sensible now.

[Rémi Coulom]

I uploaded a new version. I included a Windows version this time.

Change log:

2011-11-05: 0.0.9
- Stronger regularization (avoid overfitting in high dimensions)
- "Merge Replications" option in gui -> faster, better display
- Performance optimization of display and loading of large data files
- Removed "-ansi" option for Windows compilation
- Shrinking parameter ranges does not lose integer data any more
- Removed confusing columns: max-1, max-2, ...
- More explanations in the doc: biased win rate + GammaParameter

http://remi.coulom.free.fr/CLOP/

Rémi