Tuning again

[Rebel]

Joona post [ http://74.220.23.57/forum/viewtopic.php?t=40662 ] brings back sweet memories.

For the use of self-play I at the time wrote a small util (see below) that emulates a match between 2 equal engines in order to find out how many games it would take before every try (round) would give a reliable result. I consider a reliable result in the range of 49.9 - 50.1%

After all 1% is 6-7 elo points.

Running the utility shows that 10,000 games so now and then still may produce a 49-51% result so one is still left with an 6-7 elo error margin.

Only after 100,000 games things become stable.

Since I don't have the hardware to play 100,000 games I limit myself to 4000. When it shows an improvement I run it again with a different database. Kind of verification process. Then I make a decision.

Thoughts ?

The C-code then with apologies for the "goto" use, I am raised with that.

Ed

------------------------------------------------------------------

Code: Select all

#include <stdio.h>
#include <stdlib.h>

void main&#40;)            // emulate matches

&#123;       int r,x,max,c; float win,loss,draw,f1,f2,f3,f4; char w&#91;200&#93;; int rnd,d,e;

        srand&#40;rnd&#41;;

again&#58;  printf&#40;"Number of Games "); gets&#40;w&#41;; max=atoi&#40;w&#41;;

loop&#58;   x=0; win=0; loss=0; draw=0; printf&#40;"\n");

next&#58;   if &#40;x==max&#41; goto einde;

        r=rand&#40;); r=r&3; if &#40;r==0&#41; goto next;
        if &#40;r==1&#41; win++;
        if &#40;r==2&#41; loss++;
        if &#40;r==3&#41; draw++;
        x++; if &#40;x==&#40;max/4&#41;) goto disp;
             if &#40;x==&#40;max/2&#41;) goto disp;
             if &#40;x==&#40;max/4&#41;+&#40;max/2&#41;) goto disp;
             if &#40;x==max&#41; goto disp;
        goto next;


disp&#58;   f1=win+&#40;draw/2&#41;; f2=loss+&#40;draw/2&#41;; f4=x; f3=&#40;f1*100&#41;/f4; d=f1; e=f2;
        printf&#40;"%d-%d (%.1f%%)  ",d,e,f3&#41;;
        goto next;

einde&#58;  c=getch&#40;); if &#40;c=='q') return;
        if &#40;c=='a') &#123; printf&#40;"\n\n"); goto again; &#125;
        goto loop;

&#125;

[Edmund]

Going for a predefined LOS margin is much more accurate. Why are you tackling the problem from the other side?

[Rebel]

What's LOS ?

[zamar]

Since I don't have the hardware to play 100,000 games I limit myself to 4000.

Maybe time to buy a new machine? Let's say we use 5 seconds for one match.

100000 * 5s / (24 * 60 * 60) = 5.78 days.

This is approximately the time, it took to optimize one parameter set

[Rebel]

5 secs for a whole game? Never tried that This generation surely has a whole new elo explanation.

With Arena running on a quad I can do do 4 matches simultaneously at 8 ply producing 6000 games a day.

But I will try your 5 secs idea.

[zamar]

5 secs for a whole game? Never tried that

Well, to be more exact we used 5s+0.1s/move time controls but ran 4 matches in parallel, so it approximately resulted in 4games/20s.

[bob]

What's LOS ?

Likelihood Of Superiority

BayesElo will provide this.

[mcostalba]

5 secs for a whole game? Never tried that This generation surely has a whole new elo explanation.

With Arena running on a quad I can do do 4 matches simultaneously at 8 ply producing 6000 games a day.

But I will try your 5 secs idea.

Running games at fixed depth (especially so low like 8 plies) has some drawback, running in a GUI like Arena has even more drawbacks, I'd suggest a command line tournament manager like cutechess-cli and run on time.

BTW your C is very assemblish, lovely stuff, really, no joking: it has a kind of vintage fashion.

[Rebel]

Running games at fixed depth (especially so low like 8 plies) has some drawback,

Eval tuning I strictly do at fixed depth. I don't want external factors like time control or permanent brain to interfere. Enough volume will flatten all the horizon effects eventually, both sides.

running in a GUI like Arena has even more drawbacks, I'd suggest a command line tournament manager like cutechess-cli and run on time.

Downloaded...

I like Arena because it supports nodes-matches. IMO a better way to test search related changes than on time.

BTW your C is very assemblish, lovely stuff, really, no joking: it has a kind of vintage fashion.

All my engines were in assembler. I just can't get used to these brackets.

Code: Select all

{  
   { 
       { 
           {
           }
       }
   }
}

Things like that drives me crazy

[mcostalba]

Eval tuning I strictly do at fixed depth. I don't want external factors like time control or permanent brain to interfere. Enough volume will flatten all the horizon effects eventually, both sides.

IMHO the main drawbacks are: impossible to test depth sensible stuff like king safety and artificial same depth for midgame and endgame. But I agree for some evaluation parameters could be good, actually I will give it a try.

All my engines were in assembler. I just can't get used to these brackets.

Code: Select all

{  
   { 
       { 
           {
           }
       }
   }
}

Things like that drives me crazy

Mee too The problem here is the excessive indentation level more than the brackets in itself.