CLOP: when to stop?

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
zenpawn
Posts: 279
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

CLOP: when to stop?

Post by zenpawn » Mon Nov 07, 2016 11:11 pm

Does CLOP automatically stop once it has settled on some values or does it require somehow interpreting the colors of the Hessian or Eigenvectors tabs of the GUI?

brianr
Posts: 336
Joined: Thu Mar 09, 2006 2:01 pm

Re: CLOP: when to stop?

Post by brianr » Mon Nov 07, 2016 11:32 pm

You have to decide. I used to turn off the plotting updates after a while as the overhead was significant. You can start and stop the run and then update the graph and decide when you think you are near a good result.

Then, you should test in regular games with the optimized values. I never found any CLOP results to be an actual improvement, despite many seemingly promising runs. Texel-type tuning can be more fruitful, but after a large initial jump it gets harder to wring more improvements (at least without the massive resources of something like Fishtest).

Good luck.

PS IIRC need many 10s of thousands of runs...

zenpawn
Posts: 279
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: CLOP: when to stop?

Post by zenpawn » Mon Nov 07, 2016 11:38 pm

brianr wrote:You have to decide. I used to turn off the plotting updates after a while as the overhead was significant. You can start and stop the run and then update the graph and decide when you think you are near a good result.
Hmmm... Any tips on what to look for as a clue? There are lots of numbers on the individual tabs, and even color coding, but I'm not sure what to make of it all.
brianr wrote:Then, you should test in regular games with the optimized values. I never found any CLOP results to be an actual improvement, despite many seemingly promising runs. Texel-type tuning can be more fruitful, but after a large initial jump it gets harder to wring more improvements (at least without the massive resources of something like Fishtest).
Yeah, Texel tuning definitely intrigues me as well, but there doesn't seem to be a ready-made program to execute the algorithm on any 'ole engine like CLOP.
brianr wrote: Good luck.

PS IIRC need many 10s of thousands of runs...
Thanks, and ouch on the 10xk runs, lol.

ZirconiumX
Posts: 1327
Joined: Sun Jul 17, 2011 9:14 am

Re: CLOP: when to stop?

Post by ZirconiumX » Tue Nov 08, 2016 2:21 am

zenpawn wrote:
brianr wrote:Then, you should test in regular games with the optimized values. I never found any CLOP results to be an actual improvement, despite many seemingly promising runs. Texel-type tuning can be more fruitful, but after a large initial jump it gets harder to wring more improvements (at least without the massive resources of something like Fishtest).
Yeah, Texel tuning definitely intrigues me as well, but there doesn't seem to be a ready-made program to execute the algorithm on any 'ole engine like CLOP.
It requires more effort, yes, but the 150 elo leap my engine made with it shows it can produce big rewards. The problem with such a universal tuner is the communication latencies are massive.

Taking the zurichess epd file as an example, we have a 765k position file. Let's say it takes N time to parse a FEN. You have to go through R rounds of tuning, where I've found the 10k rounds figure fairly accurate. So that's 765k positions x 10k rounds x N seconds. Assuming you can parse a FEN in a millisecond (possibly far less, I have not checked), that's 765k x 10k x 0.001 = 7.65 million seconds spent parsing FEN positions. That's 88.5 days.

And the obvious answer of "why don't we just parse the FENs once and store them in the engine?" ruins the universality of the universal tuner.
Some believe in the almighty dollar.

I believe in the almighty printf statement.

jdart
Posts: 3720
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: CLOP: when to stop?

Post by jdart » Tue Nov 08, 2016 2:40 am

You have to go through R rounds of tuning, where I've found the 10k rounds figure fairly accurate.
This is CLOP you are talking about, or Texel? For the latter my tuning program typically converges in 50-60 iterations. It uses either Adagrad or ADAM (https://arxiv.org/pdf/1412.6980.pdf) to find the optimum.

--Jon

ZirconiumX
Posts: 1327
Joined: Sun Jul 17, 2011 9:14 am

Re: CLOP: when to stop?

Post by ZirconiumX » Tue Nov 08, 2016 2:28 pm

jdart wrote:
You have to go through R rounds of tuning, where I've found the 10k rounds figure fairly accurate.
This is CLOP you are talking about, or Texel? For the latter my tuning program typically converges in 50-60 iterations. It uses either Adagrad or ADAM (https://arxiv.org/pdf/1412.6980.pdf) to find the optimum.

--Jon
Texel. My tuning method is by no means fancy - pick a random variable, add a small random number to it, run the error calculation, if it's lower, it becomes the new best.

Also, my tuning uses random restarts, so it's probably far from optimal. The 150 elo gain speaks for itself, however.
Some believe in the almighty dollar.

I believe in the almighty printf statement.

AlvaroBegue
Posts: 916
Joined: Tue Mar 09, 2010 2:46 pm
Location: New York
Full name: Álvaro Begué (RuyDos)

Re: CLOP: when to stop?

Post by AlvaroBegue » Tue Nov 08, 2016 2:53 pm

jdart wrote:
You have to go through R rounds of tuning, where I've found the 10k rounds figure fairly accurate.
This is CLOP you are talking about, or Texel? For the latter my tuning program typically converges in 50-60 iterations. It uses either Adagrad or ADAM (https://arxiv.org/pdf/1412.6980.pdf) to find the optimum.

--Jon
I tried some first-order gradient-descent algorithms (momentum, Adam) and the convergence was too slow. Perhaps I could play around with larger learning rates...

Anyway, yesterday I managed to get L-BFGS working and my first tests seem very promising.

zenpawn
Posts: 279
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: CLOP: when to stop?

Post by zenpawn » Tue Nov 08, 2016 5:30 pm

ZirconiumX wrote:
zenpawn wrote: Yeah, Texel tuning definitely intrigues me as well, but there doesn't seem to be a ready-made program to execute the algorithm on any 'ole engine like CLOP.
It requires more effort, yes, but the 150 elo leap my engine made with it shows it can produce big rewards. The problem with such a universal tuner is the communication latencies are massive.
I'd be happy to implement it myself, but the math is apparently over my head. :(

User avatar
cdani
Posts: 2103
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: CLOP: when to stop?

Post by cdani » Tue Nov 08, 2016 6:42 pm

zenpawn wrote:
ZirconiumX wrote:
zenpawn wrote: Yeah, Texel tuning definitely intrigues me as well, but there doesn't seem to be a ready-made program to execute the algorithm on any 'ole engine like CLOP.
It requires more effort, yes, but the 150 elo leap my engine made with it shows it can produce big rewards. The problem with such a universal tuner is the communication latencies are massive.
I'd be happy to implement it myself, but the math is apparently over my head. :(
Some code that maybe can help about Texel tuning:

http://www.talkchess.com/forum/viewtopi ... =&start=20

zenpawn
Posts: 279
Joined: Sat Aug 06, 2016 6:31 pm
Location: United States

Re: CLOP: when to stop?

Post by zenpawn » Wed Nov 09, 2016 12:26 am

cdani wrote: Some code that maybe can help about Texel tuning:

http://www.talkchess.com/forum/viewtopi ... =&start=20
Hi Daniel,

Is this some sort of pre-step to the Texel tuning itself? :?

Post Reply