Texel tuning method with a small number of positions

[PK]

I managed to put together a quick and dirty implementation of Texel tuning method. In fact, right now it is nothing more than a tool to facilitate manual tuning (albeit guided by decreasing difference between evaluation and score, calculated with the sigmoid function from Peter's original post). So far, nothing new - except for the fact that I was too eager to try the algorithm and initially used something like 70.000 positions (sampled from the epd file by Alexandro Mosoi) instead of 7 millions. Surprisingly, even that was enough to show some gain - provided that I tuned only major evaluation terms. I mean, tuning piece/square tables like that would be out of the question (tried and failed) but for piece values it was already OK. About 120.000 positions allowed me to extract some smallish gain from passed pawn scores.

It probably means that if this tuning method is applied in descending order (from big to small terms) then one can shave quite a number of quiescence search calls by only gradually increasing the number of positions used.

[jdart]

The risk is that you have a good fit to those 70,000 positions but a poor fit to the larger universe of possible positions.

--Jon

[PK]

The risk is that you have a good fit to those 70,000 positions but a poor fit to the larger universe of possible positions.

Yes, I see that problem, that's why I run the short gauntlet tournaments of 1000 games afterwards, which sort of negates the time gain. However, tweaking 10 parameters I'd normally run 10 such gauntlets (probably longer ones, too), now I run one.

Another filtering idea would be to use a few batches of games, tuning for several iterations on one (picked randomly) and verifying the gain on, say, 5 others. Change would be accepted if all 5 control batches show the gain.
[/quote]

[Ferdy]

I managed to put together a quick and dirty implementation of Texel tuning method. In fact, right now it is nothing more than a tool to facilitate manual tuning (albeit guided by decreasing difference between evaluation and score, calculated with the sigmoid function from Peter's original post). So far, nothing new - except for the fact that I was too eager to try the algorithm and initially used something like 70.000 positions (sampled from the epd file by Alexandro Mosoi) instead of 7 millions. Surprisingly, even that was enough to show some gain - provided that I tuned only major evaluation terms. I mean, tuning piece/square tables like that would be out of the question (tried and failed) but for piece values it was already OK. About 120.000 positions allowed me to extract some smallish gain from passed pawn scores.

I don't understand that last statement, would you explain a little bit more on the following.

About 120.000 positions allowed me to extract some smallish gain from passed pawn scores.

[Ferdy]

The risk is that you have a good fit to those 70,000 positions but a poor fit to the larger universe of possible positions.

--Jon

I believe there is nothing wrong with small number of training positions as long as the engine when tested on the actual game tests involving positions that were not in training sets, would perform better.

[PK]

@Ferdinand, here are the details:

Using a set of 70.000 positions I succeeded in tuning:

- piece values (split into midgame and endgame)
- bishop, knight and rook pair values
- a couple of imbalance values (exchange, rook for 2 minors, pawn+knight bonus)

Using a set of 120.000 positions I successfully tuned two values:

- a percentage by which our passed pawn score is decreased if a passer is blocked
- a percentage by which our passed pawn score is increased if we control a square in front of it

On both sets tuned values failed in the test tournament on:

- pawn piece/square table
- percentage adjustement of material value and mobility values
- fianchetto bonus

[Ferdy]

@Ferdinand, here are the details:

Using a set of 70.000 positions I succeeded in tuning:

- piece values (split into midgame and endgame)
- bishop, knight and rook pair values
- a couple of imbalance values (exchange, rook for 2 minors, pawn+knight bonus)

Using a set of 120.000 positions I successfully tuned two values:

- a percentage by which our passed pawn score is decreased if a passer is blocked
- a percentage by which our passed pawn score is increased if we control a square in front of it

On both sets tuned values failed in the test tournament on:

- pawn piece/square table
- percentage adjustement of material value and mobility values
- fianchetto bonus

That is better.

So on,

Code: Select all

- pawn piece/square table
- percentage adjustement of material value and mobility values
- fianchetto bonus

We may say that its values are already optimal.