TalkChess.com

[Gary]

I've been working on the ThreatAttacks evaluation in Stockfish, trying to come up with an improvement. So, I decided to try CLOP out.

I CLOPtomized the 5 parameters (code diff: https://github.com/glinscott/Stockfish/compare/master...improve_evaluate_threats). After 7500 games, CLOP was reporting excellent numbers in the weighted ELO of 28, with 95% LCB of 19.

The Max tab was showing a Mean Elo of 21 though, with 95% LCB of -3. There was no Max found.

Running 1500 games at 40/2sec+0.01 against base Stockfish, ended up at -3 ELO. So, it's close to being good, but not an improvement yet.

Any thoughts on where I could be going wrong with my CLOP usage? Just not enough games in CLOP land? I tested CLOP at 40/1sec initially, and the values I got tested at about -30ELO. So, it's definitely doing better at 40/2sec, but, still - not beating base Stockfish .

[Marco Costalba]

I am very interested in your tests with CLOP, I have looked at your patch and here are some quick comments:

1) 7500 games is defenitely to low number for 5 parameters tuning

2) You not only have added tuning code but have also changed the semantic unifying pawn threats with others pieces threats and implementing MultipleThreatBonus idea. I'd suggest to proceed in a more step by step approach. First rewrite the code in the way that you like but preserving the same functionality (verify with ./stockfish bench) this is very important to don't become crazy following multiple patterns at the same time, remember that you are chasing for 5 ELO maximum increase so it is mandatory to keep external noise level at the minimum. Then, once you have added your tuning code and be sure that functionality is the same, then start tuning but I'd suggest to start with 2-3 parameters maximum or, OTH, run much longer, say 30K-40K games.

4) If instead you want to test new ideas, I'd suggest to test one by one.

3) CLOP makes no miracles and tuning is hard and requires a lot of self discipline, given that SF is a mature engine it is even harder. But don't give up: efforts will be rewarded in the long term, you just need a lot of "patience and perseverance", this can well be the chess engine developer motto

[Gary]

Thanks Marco! I agree that the patch is attempting to take on too much in one go. It was my first attempt with CLOP, so more of an experiment.

Verifying with bench was indeed very important. When all parameters were set at 100 initially, it was same as default. And CLOP did cluster the values around 100, so it seems to be working well.

I'll try breaking it up into smaller chunks and seeing how it goes.

[Rémi Coulom]

The win rates CLOP reports are biased towards optimistic values. So you should expect that the strength of the program you get at the end is inferior to what CLOP reports.

If you wish to optimize for small improvements (a few Elo points), then you need to play many more games.

Rémi

[Gary]

[Gary]

[Marco Costalba]

[Rémi Coulom]

[Gary]

[Gary]