Working with CLOP

[Robert Pope]

I finally got CLOP up and running on my computer. That was a real headache, since I didn't realize for quite a while that I needed to install python to run the python cutechess-cli script, and then I couldn't get the PATH variable to stick.

Now that it is running and I am doing my first optimization test (just of standard piece values), I have a few questions:

1. How do you know if a term has been optimized "enough"? If it takes 500,000 games, so be it. But if the terms are already pretty much as good as they can get after 50,000 games, how can I recognize that?

2. My CLOP file has me running 3 processors on my Quad. But invariably, after a few hundred games, I start to get processors that drop out. A soft pause won't close out the threads, and I manually have to do a hard close and then go into Task Manager and kill an instance of cutechess-cli and the engine I was playing.

I'm guessing it must be my engine that is terminating prematurely, since it is never the one left idling, but how can I troubleshoot this? I have no idea which game it stuck on to find a corresponding log file.

[nionita]

1. How do you know if a term has been optimized "enough"? If it takes 500,000 games, so be it. But if the terms are already pretty much as good as they can get after 50,000 games, how can I recognize that?

This is one of the problems I found when optimizing with Clop, that you don't have at any time at least the best local optimum you got so far. This because there is none, the "measurements" are (very) noisy.

But after "enough" games (what is enough?) the parameters in the tab "Max", on the "Mean" side (left) vary not so much anymore. Some of them even less. In this case it happened to me that the parameters were pretty good. (But, also, sometimes they were not so good...)

So my method is: let Clop running. From time to time take the current parameters (if existing, Max parameters, otherwise Mean parameters from the Max tab) and make a run of a few tousands games with older versions of the engine (or, even better, with a few different engines). When the improvements are small you will need even something like 20 tousand games. Then you can see if that version is better then your current version or not. So you can improve the engine, but actually you never get the optimum.

I never reached a moment when Clop said "the parameters are optimal", even for 2 parameters. Probably to reach this point you have to play millions of games.

Another point: if you want to optimise more then 4-5 parameters, this will take forever. And Clop gets slower and slower (but then you can uncheck the automatic update of the GUI and this gets better).

[jdart]

I have never had much luck with CLOP. It can be slow to converge. And it is a local optimizer so even if it converges it may not be finding the globally best set of values.

But I have run it with thousands of games distributed over multiple cores/machines w/o problems.

--Jon

[Bloodbane]

Disclaimer: I haven't used used CLOP for computer chess yet though I plan to do so very soon.

I've had positive experience with it in general and almost always the parameters given by CLOP have been better than the originals. I've also ran into some weird edge cases where CLOP first converges for some time and then for no reason diverges completely, and then starts to converge again very very slowly, so slowly in fact that even with around 100000 samples (around 4 days worth of computer time on an i7-4790K with 7 threads) it hasn't converged again properly. Also, CLOP has been close to useless with >8 dimensions but that is not anything surprising.

[mar]

My suggestion is: don't waste time on CLOP. If you want to tune eval there are vastly superior and much faster methods that actually converge.

[Robert Pope]

My suggestion is: don't waste time on CLOP. If you want to tune eval there are vastly superior and much faster methods that actually converge.

What do you recommend, then?

[mar]

This: http://www.talkchess.com/forum/viewtopi ... 22&t=50823
From what I understood Miguel did something very similar (as can be seen in the thread).

[nionita]

This: http://www.talkchess.com/forum/viewtopi ... 22&t=50823
From what I understood Miguel did something very similar (as can be seen in the thread).

But this is something completely different. While for Clop there is a theoretical proof that, if it converges, you got a local optimum, for that method there is nothing proved. It may work, it may not.

Also, as I said, I got improvements with Clop. But it takes time, and it's not suited for many parameters at once.

[mar]

Nothing proved? It has been used sucessfully in several engines giving significant gains in gameplay
(unless your eval is already well tuned which seems very unlikely if you just guesstimate the values).

I have never seen CLOP converging. It always seems to choose some random not "outright wrong" values
(read it rejects wrong values quickly).
I never played more than 100k games as it would take forever.
Perhaps it needs millions of games for a single parameter - I say no, thanks - didn't work for me.

It didn't converge for a single parameter in a day on a quad iirc (at least not for me).

However I was able to tune my _whole_ eval using Peter's method in one day or so (60 eval terms + 736 psq values square by square).
How long would it take with CLOP?
So good luck with that. Your experience is obviously different from mine.

[Robert Pope]

Nothing proved? It has been used sucessfully in several engines giving significant gains in gameplay
(unless your eval is already well tuned which seems very unlikely if you just guesstimate the values).

I have never seen CLOP converging. It always seems to choose some random not "outright wrong" values
(read it rejects wrong values quickly).
I never played more than 100k games as it would take forever.
Perhaps it needs millions of games for a single parameter - I say no, thanks - didn't work for me.

It didn't converge for a single parameter in a day on a quad iirc (at least not for me).

However I was able to tune my _whole_ eval using Peter's method in one day or so (60 eval terms + 736 psq values square by square).
How long would it take with CLOP?
So good luck with that. Your experience is obviously different from mine.

Thanks for the pointer to Peter's method. I think I will see if I can get any immediate gains from CLOP first, since it requires very little front-end effort now that I have it running, and I have more CPU cycles than free time available right now.