ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Tuning again
Goto page 1, 2, 3, 4  Next
 
Post new topic       TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions Threaded
View previous topic :: View next topic  
Author Message
Ed Schroder



Joined: 18 Aug 2011
Posts: 1615
PostPosted: Tue Nov 01, 2011 10:20 am    Post subject: Tuning again Reply to topic Reply with quote
Joona post [ http://74.220.23.57/forum/viewtopic.php?t=40662 ] brings back sweet memories.

For the use of self-play I at the time wrote a small util (see below) that emulates a match between 2 equal engines in order to find out how many games it would take before every try (round) would give a reliable result. I consider a reliable result in the range of 49.9 - 50.1%

After all 1% is 6-7 elo points.

Running the utility shows that 10,000 games so now and then still may produce a 49-51% result so one is still left with an 6-7 elo error margin.

Only after 100,000 games things become stable.

Since I don't have the hardware to play 100,000 games I limit myself to 4000. When it shows an improvement I run it again with a different database. Kind of verification process. Then I make a decision.

Thoughts ?

The C-code then with apologies for the "goto" use, I am raised with that.

Ed

------------------------------------------------------------------

Code:

#include <stdio.h>
#include <stdlib.h>

void main()            // emulate matches

{       int r,x,max,c; float win,loss,draw,f1,f2,f3,f4; char w[200]; int rnd,d,e;

        srand(rnd);

again:  printf("Number of Games "); gets(w); max=atoi(w);

loop:   x=0; win=0; loss=0; draw=0; printf("\n");

next:   if (x==max) goto einde;

        r=rand(); r=r&3; if (r==0) goto next;
        if (r==1) win++;
        if (r==2) loss++;
        if (r==3) draw++;
        x++; if (x==(max/4)) goto disp;
             if (x==(max/2)) goto disp;
             if (x==(max/4)+(max/2)) goto disp;
             if (x==max) goto disp;
        goto next;


disp:   f1=win+(draw/2); f2=loss+(draw/2); f4=x; f3=(f1*100)/f4; d=f1; e=f2;
        printf("%d-%d (%.1f%%)  ",d,e,f3);
        goto next;

einde:  c=getch(); if (c=='q') return;
        if (c=='a') { printf("\n\n"); goto again; }
        goto loop;

}

Back to top
View user's profile Send private message
Edmund Moshammer



Joined: 03 Dec 2007
Posts: 598
Location: St. Gallen, Switzerland
PostPosted: Tue Nov 01, 2011 11:42 am    Post subject: Re: Tuning again Reply to topic Reply with quote
Going for a predefined LOS margin is much more accurate. Why are you tackling the problem from the other side?
Back to top
View user's profile Send private message Visit poster's website
Ed Schroder



Joined: 18 Aug 2011
Posts: 1615
PostPosted: Tue Nov 01, 2011 4:13 pm    Post subject: Re: Tuning again Reply to topic Reply with quote
What's LOS ?
Back to top
View user's profile Send private message
Joona Kiiski



Joined: 18 Jan 2009
Posts: 546
PostPosted: Tue Nov 01, 2011 4:20 pm    Post subject: Re: Tuning again Reply to topic Reply with quote
Rebel wrote:
Since I don't have the hardware to play 100,000 games I limit myself to 4000.


Maybe time to buy a new machine? Let's say we use 5 seconds for one match.

100000 * 5s / (24 * 60 * 60) = 5.78 days.

This is approximately the time, it took to optimize one parameter set Smile
_________________
Joona Kiiski
Back to top
View user's profile Send private message
Ed Schroder



Joined: 18 Aug 2011
Posts: 1615
PostPosted: Tue Nov 01, 2011 6:11 pm    Post subject: Re: Tuning again Reply to topic Reply with quote
5 secs for a whole game? Never tried that Shocked This generation surely has a whole new elo explanation.

With Arena running on a quad I can do do 4 matches simultaneously at 8 ply producing 6000 games a day.

But I will try your 5 secs idea.
Back to top
View user's profile Send private message
Joona Kiiski



Joined: 18 Jan 2009
Posts: 546
PostPosted: Tue Nov 01, 2011 7:25 pm    Post subject: Re: Tuning again Reply to topic Reply with quote
Rebel wrote:
5 secs for a whole game? Never tried that Shocked


Well, to be more exact we used 5s+0.1s/move time controls but ran 4 matches in parallel, so it approximately resulted in 4games/20s.
_________________
Joona Kiiski
Back to top
View user's profile Send private message
Robert Hyatt



Joined: 27 Feb 2006
Posts: 15818
Location: Birmingham, AL
PostPosted: Tue Nov 01, 2011 8:04 pm    Post subject: Re: Tuning again Reply to topic Reply with quote
Rebel wrote:
What's LOS ?


Likelihood Of Superiority

BayesElo will provide this.
Back to top
View user's profile Send private message
Marco Costalba



Joined: 14 Jun 2008
Posts: 2090
PostPosted: Tue Nov 01, 2011 9:30 pm    Post subject: Re: Tuning again Reply to topic Reply with quote
Rebel wrote:
5 secs for a whole game? Never tried that Shocked This generation surely has a whole new elo explanation.

With Arena running on a quad I can do do 4 matches simultaneously at 8 ply producing 6000 games a day.

But I will try your 5 secs idea.


Running games at fixed depth (especially so low like 8 plies) has some drawback, running in a GUI like Arena has even more drawbacks, I'd suggest a command line tournament manager like cutechess-cli and run on time.

BTW your C is very assemblish, lovely stuff, really, no joking: it has a kind of vintage fashion.
Back to top
View user's profile Send private message
Ed Schroder



Joined: 18 Aug 2011
Posts: 1615
PostPosted: Tue Nov 01, 2011 11:55 pm    Post subject: Re: Tuning again Reply to topic Reply with quote
mcostalba wrote:
Running games at fixed depth (especially so low like 8 plies) has some drawback,

Eval tuning I strictly do at fixed depth. I don't want external factors like time control or permanent brain to interfere. Enough volume will flatten all the horizon effects eventually, both sides.

Quote:
running in a GUI like Arena has even more drawbacks, I'd suggest a command line tournament manager like cutechess-cli and run on time.

Downloaded...

I like Arena because it supports nodes-matches. IMO a better way to test search related changes than on time.

Quote:
BTW your C is very assemblish, lovely stuff, really, no joking: it has a kind of vintage fashion.

All my engines were in assembler. I just can't get used to these brackets.

Code:

{ 
   {
       {
           {
           }
       }
   }
}

Things like that drives me crazy Wink
Back to top
View user's profile Send private message
Marco Costalba



Joined: 14 Jun 2008
Posts: 2090
PostPosted: Wed Nov 02, 2011 6:04 am    Post subject: Re: Tuning again Reply to topic Reply with quote
Rebel wrote:
Eval tuning I strictly do at fixed depth. I don't want external factors like time control or permanent brain to interfere. Enough volume will flatten all the horizon effects eventually, both sides.


IMHO the main drawbacks are: impossible to test depth sensible stuff like king safety and artificial same depth for midgame and endgame. But I agree for some evaluation parameters could be good, actually I will give it a try.

Quote:
All my engines were in assembler. I just can't get used to these brackets.

Code:

{ 
   {
       {
           {
           }
       }
   }
}

Things like that drives me crazy Wink


Mee too Wink The problem here is the excessive indentation level more than the brackets in itself.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic       TalkChess.com Forum Index -> Computer Chess Club: Programming and Technical Discussions All times are GMT
Goto page 1, 2, 3, 4  Next 		Threaded
Page 1 of 4

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads