Question on recommended tuning techniques

[spinosarus123]

Hello,
New chess programmer here. I successfully tuned my engines evaluation tables (psqt etc) using texel tuning. I did not measure the difference exactly but it gained around 300 elo over several tuning sessions. For a couple of variables in my search function like razoring and time management I have to resort to another tuning technique.

I have tried to read about which tuning systems are recommended. It seems that CLOP is the standard, is this still true? Is there newer source code than Rémi's CLOP-0.0.9? I just managed to get it to compile under qt5 with some minor tweaks, but I have not tried it yet.

It seems that the stockfish teams switched to SPSA but not necessarily because it was better, but because CLOP was hard to integrate with their distributed computing system.

/Isak

[jdart]

I used CLOP a long time ago, but it can be very slow to converge. The basic problem is, you are trying to optimize an objective--game results--that is very noisy, i.e. there is a lot of random variation in outcomes, on top of the real differences you are trying to measure. Then on top some parameters have quite small effects when varied. There are other methods besides CLOP that are likely better, but they will still require long tuning runs.

[gaard]

I had a lot of success with https://github.com/fsmosca/Lakas I remember it being a PITA to setup, but the results were positive. If I had to do it again, I would probably write a wrapper around cutechess using NOMAD, which is super robust up to 20 dimensions, and scales well up to 50. NOMAD also lets you use surrogates, so while the optimizer is running, you could parallelize getting quicker approximation of your parameters with a higher granularity, at a much faster time control, which can speed things up immensely.

[Chessnut1071]

Hello,
New chess programmer here. I successfully tuned my engines evaluation tables (psqt etc) using texel tuning. I did not measure the difference exactly but it gained around 300 elo over several tuning sessions. For a couple of variables in my search function like razoring and time management I have to resort to another tuning technique.

I have tried to read about which tuning systems are recommended. It seems that CLOP is the standard, is this still true? Is there newer source code than Rémi's CLOP-0.0.9? I just managed to get it to compile under qt5 with some minor tweaks, but I have not tried it yet.

It seems that the stockfish teams switched to SPSA but not necessarily because it was better, but because CLOP was hard to integrate with their distributed computing system.

/Isak

The best method I found for optimizing engine parameters is a modified Hooke & Jeeves nonlinear optimization. This method is the best I've seen for chess, although I modified sections of the pattern and direction phase to better reflect chess games with reduced deltas. I use the MHJ method to optimize checkmate solutions currently up to 24-ply. Here are the issues.

First, the optimization works great on the games included in the optimization; however, when you run games that weren't included, results can be very disappointing. You must use thousands of games to build a generic checkmate solver. Results degrade significantly as more games are included.

The key here is the function and most important what's in it. I currently get 90 - 99% reduction in engine calls-and subsequently time-with the following:

1) history; what move worked best for this ply
2) history: last successful move
3) check, double check & discovered check
4) Piece type P - K
5) Material effect (points gained or lost)
6) attack on King's diagonal, rank, file and king squares.
7) open files for rooks and queens
8) pawn promotion
9) ent passant

My interest is discovering the generic checkmate function found by a chess programmer who came up with a system which is 100s of times faster than mine on a very old-22 years old-computer back in 2006. Unfortunately, he died and did not disclose how he did it. If I find it, I will publish it.

If you are using C# you are welcome to the C# code for the optimization algorithm. The input is simply the parameter list and the weight on the evaluation function.

Also, if anybody knows of addition parameters to use in the checkmate function that have a significant impact, list away. Problem is, they have to work across 1,000s of games and the best I found are history and check. Also, three phases: white 1st move, white all other moves, and black all moves.