Tuning again

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4271
Joined: Thu Aug 18, 2011 10:04 am

Tuning again

Post by Rebel » Tue Nov 01, 2011 10:20 am

Joona post [ http://74.220.23.57/forum/viewtopic.php?t=40662 ] brings back sweet memories.

For the use of self-play I at the time wrote a small util (see below) that emulates a match between 2 equal engines in order to find out how many games it would take before every try (round) would give a reliable result. I consider a reliable result in the range of 49.9 - 50.1%

After all 1% is 6-7 elo points.

Running the utility shows that 10,000 games so now and then still may produce a 49-51% result so one is still left with an 6-7 elo error margin.

Only after 100,000 games things become stable.

Since I don't have the hardware to play 100,000 games I limit myself to 4000. When it shows an improvement I run it again with a different database. Kind of verification process. Then I make a decision.

Thoughts ?

The C-code then with apologies for the "goto" use, I am raised with that.

Ed

------------------------------------------------------------------

Code: Select all

#include <stdio.h>
#include <stdlib.h>

void main&#40;)            // emulate matches

&#123;       int r,x,max,c; float win,loss,draw,f1,f2,f3,f4; char w&#91;200&#93;; int rnd,d,e;

        srand&#40;rnd&#41;;

again&#58;  printf&#40;"Number of Games "); gets&#40;w&#41;; max=atoi&#40;w&#41;;

loop&#58;   x=0; win=0; loss=0; draw=0; printf&#40;"\n");

next&#58;   if &#40;x==max&#41; goto einde;

        r=rand&#40;); r=r&3; if &#40;r==0&#41; goto next;
        if &#40;r==1&#41; win++;
        if &#40;r==2&#41; loss++;
        if &#40;r==3&#41; draw++;
        x++; if &#40;x==&#40;max/4&#41;) goto disp;
             if &#40;x==&#40;max/2&#41;) goto disp;
             if &#40;x==&#40;max/4&#41;+&#40;max/2&#41;) goto disp;
             if &#40;x==max&#41; goto disp;
        goto next;


disp&#58;   f1=win+&#40;draw/2&#41;; f2=loss+&#40;draw/2&#41;; f4=x; f3=&#40;f1*100&#41;/f4; d=f1; e=f2;
        printf&#40;"%d-%d (%.1f%%)  ",d,e,f3&#41;;
        goto next;

einde&#58;  c=getch&#40;); if &#40;c=='q') return;
        if &#40;c=='a') &#123; printf&#40;"\n\n"); goto again; &#125;
        goto loop;

&#125;


Edmund
Posts: 668
Joined: Mon Dec 03, 2007 2:01 pm
Location: Barcelona, Spain
Contact:

Re: Tuning again

Post by Edmund » Tue Nov 01, 2011 11:42 am

Going for a predefined LOS margin is much more accurate. Why are you tackling the problem from the other side?

User avatar
Rebel
Posts: 4271
Joined: Thu Aug 18, 2011 10:04 am

Re: Tuning again

Post by Rebel » Tue Nov 01, 2011 4:13 pm

What's LOS ?

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: Tuning again

Post by zamar » Tue Nov 01, 2011 4:20 pm

Rebel wrote:Since I don't have the hardware to play 100,000 games I limit myself to 4000.
Maybe time to buy a new machine? Let's say we use 5 seconds for one match.

100000 * 5s / (24 * 60 * 60) = 5.78 days.

This is approximately the time, it took to optimize one parameter set :-)
Joona Kiiski

User avatar
Rebel
Posts: 4271
Joined: Thu Aug 18, 2011 10:04 am

Re: Tuning again

Post by Rebel » Tue Nov 01, 2011 6:11 pm

5 secs for a whole game? Never tried that :shock: This generation surely has a whole new elo explanation.

With Arena running on a quad I can do do 4 matches simultaneously at 8 ply producing 6000 games a day.

But I will try your 5 secs idea.

zamar
Posts: 613
Joined: Sun Jan 18, 2009 6:03 am

Re: Tuning again

Post by zamar » Tue Nov 01, 2011 7:25 pm

Rebel wrote:5 secs for a whole game? Never tried that :shock:
Well, to be more exact we used 5s+0.1s/move time controls but ran 4 matches in parallel, so it approximately resulted in 4games/20s.
Joona Kiiski

bob
Posts: 20358
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Tuning again

Post by bob » Tue Nov 01, 2011 8:04 pm

Rebel wrote:What's LOS ?
Likelihood Of Superiority

BayesElo will provide this.

mcostalba
Posts: 2681
Joined: Sat Jun 14, 2008 7:17 pm

Re: Tuning again

Post by mcostalba » Tue Nov 01, 2011 9:30 pm

Rebel wrote:5 secs for a whole game? Never tried that :shock: This generation surely has a whole new elo explanation.

With Arena running on a quad I can do do 4 matches simultaneously at 8 ply producing 6000 games a day.

But I will try your 5 secs idea.
Running games at fixed depth (especially so low like 8 plies) has some drawback, running in a GUI like Arena has even more drawbacks, I'd suggest a command line tournament manager like cutechess-cli and run on time.

BTW your C is very assemblish, lovely stuff, really, no joking: it has a kind of vintage fashion.

User avatar
Rebel
Posts: 4271
Joined: Thu Aug 18, 2011 10:04 am

Re: Tuning again

Post by Rebel » Tue Nov 01, 2011 11:55 pm

mcostalba wrote:Running games at fixed depth (especially so low like 8 plies) has some drawback,
Eval tuning I strictly do at fixed depth. I don't want external factors like time control or permanent brain to interfere. Enough volume will flatten all the horizon effects eventually, both sides.
running in a GUI like Arena has even more drawbacks, I'd suggest a command line tournament manager like cutechess-cli and run on time.
Downloaded...

I like Arena because it supports nodes-matches. IMO a better way to test search related changes than on time.
BTW your C is very assemblish, lovely stuff, really, no joking: it has a kind of vintage fashion.
All my engines were in assembler. I just can't get used to these brackets.

Code: Select all

&#123;  
   &#123; 
       &#123; 
           &#123;
           &#125;
       &#125;
   &#125;
&#125;
Things like that drives me crazy :wink:

mcostalba
Posts: 2681
Joined: Sat Jun 14, 2008 7:17 pm

Re: Tuning again

Post by mcostalba » Wed Nov 02, 2011 6:04 am

Rebel wrote: Eval tuning I strictly do at fixed depth. I don't want external factors like time control or permanent brain to interfere. Enough volume will flatten all the horizon effects eventually, both sides.
IMHO the main drawbacks are: impossible to test depth sensible stuff like king safety and artificial same depth for midgame and endgame. But I agree for some evaluation parameters could be good, actually I will give it a try.
All my engines were in assembler. I just can't get used to these brackets.

Code: Select all

&#123;  
   &#123; 
       &#123; 
           &#123;
           &#125;
       &#125;
   &#125;
&#125;
Things like that drives me crazy :wink:
Mee too :wink: The problem here is the excessive indentation level more than the brackets in itself.

Post Reply