ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

CLOP for Noisy Black-Box Parameter Optimization
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Flat
View previous topic :: View next topic  
Author Message
Rémi Coulom



Joined: 24 Apr 2006
Posts: 350
PostPost subject: Re: CLOP for Noisy Black-Box Parameter Optimization    Posted: Sun Oct 23, 2011 8:55 pm Reply to topic Reply with quote
Ferdy wrote:

1. Is there a way for this gui to plot the number of games along x-axis and win rate along y-axis? I just want to know what happened to the win rate as we increased number of games.

You can not do it with the gui. But you can open the data file with the command-line version, and redirect output to a file, then plot it with a tool such as gnuplot, for instance.

Quote:

2. I am doing some interaction like for example if I give queen_value 960 1040 (default 1000, delta -40/+40), and then after say around 5000 games not happy with this range and I change to 95 1005 (delta -5/+5) would this be just fine?


No, there is a problem with shrinking range. I will fix it. But you should not want to do that. CLOP is good at figuring out the right range. For tuning a parameter such as Queen value, just use a wide range (100-2000, for instance). It is very likely CLOP will do better by itself than if you try to tweak the range manually.

The only good reason to manually change the range during optimization is to start with a narrow range in order to indicate a good point in parameter space, then widen the range and let CLOP do its magic.

Quote:
3. If I will use for example only 1 parameter to tune say queen_value 95 1005 (-5/+5 around default) how many games would you like to run so that you can say that the result is reliable?


You should not use such a narrow range. The more games, the more accurate the result. You can get an idea of how CLOP is uncertain about the value by looking at the interval over which it takes samples. But use a wide default interval.

Quote:

4. If I run and test only for 1 starting position for example sicilian najdorf -
e2e4 c7c5 g1f3 d7d6 d2d4 c5d4 f3d4 g8f6 b1c3 a7a6. Then test it against 20 different engines. The engine to be tuned should play both colors. Then add some search and eval parameters like fulity_margin and piece mobilitites for the eval, is this also that clop can handle optimizing parameters on a certain opening?


If your connection script plays games from that position only, then yes, CLOP will optimize for that position. But you should make sure there is some randomness in the experiment. If the program and opening are deterministic, it is likely to cause problems because the same game will be played many times. The usual approach is to randomize openings. You can also use opponents with a random evaluation, or randomize your own.

Rémi
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Subject Author Date/Time
CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Thu Sep 01, 2011 10:32 am
      Re: CLOP for Noisy Black-Box Parameter Optimization Joona Kiiski Fri Sep 02, 2011 8:52 am
            Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Sep 02, 2011 9:45 am
                  Re: CLOP for Noisy Black-Box Parameter Optimization Marco Costalba Fri Sep 02, 2011 5:34 pm
            Re: CLOP for Noisy Black-Box Parameter Optimization Zach Wegner Fri Sep 02, 2011 4:44 pm
                  Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Sep 02, 2011 7:58 pm
      Re: CLOP for Noisy Black-Box Parameter Optimization Rein Halbersma Fri Sep 02, 2011 10:07 pm
            Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sat Sep 03, 2011 9:24 am
                  Re: CLOP for Noisy Black-Box Parameter Optimization Rein Halbersma Sat Sep 03, 2011 5:01 pm
                        Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sat Sep 03, 2011 5:28 pm
      Re: CLOP for Noisy Black-Box Parameter Optimization Andres Valverde Sat Sep 03, 2011 10:28 am
            Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sat Sep 03, 2011 10:40 am
      Re: CLOP for Noisy Black-Box Parameter Optimization Peter Österlund Sat Sep 03, 2011 6:54 pm
            Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sat Sep 03, 2011 7:34 pm
                  Re: CLOP for Noisy Black-Box Parameter Optimization Ernest Bonnem Sun Sep 04, 2011 4:49 pm
      Re: CLOP for Noisy Black-Box Parameter Optimization Don Dailey Wed Sep 07, 2011 4:57 pm
            Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Wed Sep 07, 2011 5:47 pm
                  Re: CLOP for Noisy Black-Box Parameter Optimization Don Dailey Wed Sep 07, 2011 6:09 pm
                        Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Wed Sep 07, 2011 6:26 pm
                        Re: CLOP for Noisy Black-Box Parameter Optimization F. Bluemers Wed Sep 07, 2011 6:39 pm
      Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Wed Sep 07, 2011 8:19 pm
            Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Thu Sep 08, 2011 4:44 pm
                  Re: CLOP for Noisy Black-Box Parameter Optimization Larry Kaufman Fri Sep 09, 2011 3:36 am
                        Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Sep 09, 2011 6:11 am
                              Re: CLOP for Noisy Black-Box Parameter Optimization Larry Kaufman Fri Sep 09, 2011 2:43 pm
                                    Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Sep 09, 2011 7:47 pm
                                          Re: CLOP for Noisy Black-Box Parameter Optimization Marco Costalba Sun Nov 06, 2011 8:24 am
                                                Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sun Nov 06, 2011 8:30 am
                                          Re: CLOP for Noisy Black-Box Parameter Optimization Marco Costalba Sun Nov 06, 2011 8:50 am
                                                Re: CLOP for Noisy Black-Box Parameter Optimization Peter Österlund Sun Nov 06, 2011 1:11 pm
                                                      Re: CLOP for Noisy Black-Box Parameter Optimization Marco Costalba Sun Nov 06, 2011 4:53 pm
                                                      Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sun Nov 06, 2011 5:14 pm
                                                            Re: CLOP for Noisy Black-Box Parameter Optimization Peter Österlund Sun Nov 06, 2011 11:11 pm
                                                                  Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Mon Nov 07, 2011 11:18 am
                                                Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sun Nov 06, 2011 5:05 pm
                                    Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Sep 09, 2011 7:52 pm
                                          Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sat Sep 10, 2011 9:49 am
                                                Re: CLOP for Noisy Black-Box Parameter Optimization Michel Van den Bergh Sun Sep 25, 2011 8:18 am
                                                      Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sun Sep 25, 2011 8:47 am
                                                            Re: CLOP for Noisy Black-Box Parameter Optimization Michel Van den Bergh Mon Sep 26, 2011 10:55 am
                                                                  Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Mon Sep 26, 2011 1:40 pm
                                                                        Re: CLOP for Noisy Black-Box Parameter Optimization Michel Van den Bergh Tue Sep 27, 2011 7:54 am
                                                                              Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Tue Sep 27, 2011 8:39 am
                                                                                    Re: CLOP for Noisy Black-Box Parameter Optimization Michel Van den Bergh Thu Sep 29, 2011 2:04 pm
                                                                                          Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Thu Sep 29, 2011 3:38 pm
                                                                        Re: CLOP for Noisy Black-Box Parameter Optimization Daniel Shawul Mon Oct 17, 2011 2:03 am
                                                                              Re: CLOP for Noisy Black-Box Parameter Optimization Michel Van den Bergh Mon Oct 17, 2011 7:51 am
                                                                              Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Mon Oct 17, 2011 7:56 am
                                                                                    Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Mon Oct 17, 2011 8:02 am
                                                                                          Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Mon Oct 17, 2011 8:11 am
                                                                                    Re: CLOP for Noisy Black-Box Parameter Optimization Daniel Shawul Mon Oct 17, 2011 9:54 am
                                                                                          Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Mon Oct 17, 2011 11:10 am
      Re: CLOP for Noisy Black-Box Parameter Optimization david nash Wed Oct 05, 2011 12:19 am
      Re: CLOP - a trial for xiangqi engine Edward Yu Fri Oct 07, 2011 11:58 am
            Re: CLOP - a trial for xiangqi engine Rémi Coulom Fri Oct 07, 2011 1:17 pm
                  Re: CLOP - a trial for xiangqi engine Edward Yu Fri Oct 07, 2011 4:01 pm
                        Re: CLOP - a trial for xiangqi engine Rémi Coulom Fri Oct 07, 2011 4:30 pm
                              Re: CLOP - a trial for xiangqi engine Edward Yu Sat Oct 08, 2011 3:59 am
                                    Re: CLOP - a trial for xiangqi engine Rémi Coulom Sat Oct 08, 2011 7:49 am
                                          Re: CLOP - a trial for xiangqi engine Edward Yu Tue Oct 11, 2011 4:04 pm
                                                Re: CLOP - a trial for xiangqi engine Rémi Coulom Tue Oct 11, 2011 4:22 pm
                                                      Re: CLOP - a trial for xiangqi engine Daniel Shawul Tue Oct 11, 2011 4:49 pm
                                                            Re: CLOP - a trial for xiangqi engine Rémi Coulom Tue Oct 11, 2011 5:27 pm
                                                                  Issue.... Michel Van den Bergh Sat Oct 15, 2011 2:51 pm
                                                                        Re: Issue.... Rémi Coulom Sun Oct 16, 2011 6:30 am
                                                                              Re: Issue.... Michel Van den Bergh Sun Oct 16, 2011 8:10 am
                                                                                    Re: Issue.... Rémi Coulom Sun Oct 16, 2011 9:02 am
                                                                                          Re: Issue.... Michel Van den Bergh Sun Oct 16, 2011 3:10 pm
                                                                                          Re: Issue.... Rémi Coulom Sun Oct 16, 2011 7:25 pm
                              Re: CLOP Michel Van den Bergh Sat Oct 08, 2011 6:00 am
                                    Re: CLOP Rémi Coulom Sat Oct 08, 2011 7:53 am
                                          Re: CLOP Rémi Coulom Sat Oct 08, 2011 8:27 am
      Re: CLOP for Noisy Black-Box Parameter Optimization Ferdinand Mosca Tue Oct 11, 2011 8:22 am
            Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Tue Oct 11, 2011 1:38 pm
                  Re: CLOP for Noisy Black-Box Parameter Optimization Ferdinand Mosca Sun Oct 23, 2011 7:39 pm
                        Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sun Oct 23, 2011 8:55 pm
                              Re: CLOP for Noisy Black-Box Parameter Optimization Ferdinand Mosca Sun Oct 23, 2011 10:47 pm
                                    Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Fri Nov 04, 2011 9:42 am
                                          Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Nov 04, 2011 9:58 am
                                                Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Fri Nov 04, 2011 10:17 am
                                                      Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Nov 04, 2011 10:30 am
                                                            Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Fri Nov 04, 2011 10:44 am
                                                                  Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Nov 04, 2011 10:56 am
                                                                        Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Fri Nov 04, 2011 11:04 am
                                                                              Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Fri Nov 04, 2011 11:56 am
                                                                                    Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Fri Nov 04, 2011 4:16 pm
                                                                                          Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Fri Nov 04, 2011 5:48 pm
      Re: CLOP for Noisy Black-Box Parameter Optimization Sergei S. Markoff Fri Nov 04, 2011 3:02 pm
      Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sat Nov 05, 2011 7:45 pm
            Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Sat Nov 05, 2011 10:49 pm
            Re: CLOP for Noisy Black-Box Parameter Optimization Ilari Pihlajisto Sat Nov 05, 2011 11:17 pm
                  Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Sun Nov 06, 2011 12:06 am
                        Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sun Nov 06, 2011 8:00 am
                              Re: CLOP for Noisy Black-Box Parameter Optimization Jonatan Pettersson Sun Nov 06, 2011 10:20 am
                                    Re: CLOP for Noisy Black-Box Parameter Optimization Rémi Coulom Sun Nov 06, 2011 5:10 pm
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads