ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Who is stronger at chess? Computers or Humans?
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: General Topics Flat
View previous topic :: View next topic  

Who is stronger? Computer or Humans? How much?
Computer +500 elo
29%
 29%  [ 16 ]
Computer 301-500elo
27%
 27%  [ 15 ]
Computer 101-300 elo
32%
 32%  [ 18 ]
Computer 1-100 elo
3%
 3%  [ 2 ]
Equality
1%
 1%  [ 1 ]
Human 1-100 elo
3%
 3%  [ 2 ]
Human 101-300 elo
0%
 0%  [ 0 ]
Human 301-500 elo
0%
 0%  [ 0 ]
Human +500 elo
1%
 1%  [ 1 ]
Voted : 55
Total Votes : 55
- Poll expired -

Author Message
Adam Hair



Joined: 06 May 2009
Posts: 1637
Location: Fuquay-Varina, North Carolina
PostPost subject: Re: Who is stronger at chess? Computers or Humans?    Posted: Mon Feb 27, 2012 4:39 am Reply to topic Reply with quote
Milos wrote:
Adam Hair wrote:
There have been some hints from different authors that they were getting more than 70 Elo per doubling at shorter search times. I have seen a citation that Levy and Newborn stated in How Computers Play Chess of 50 to 70 Elo per doubling, but I have not seen that confirmed in recent times. So, I decided to see if it was true, for I needed to know for another test.

The following list is the result of my testing the approximate gain in Elo for each doubling of thinking time, which should roughly equal the same increase due to doubling speed:

[code]Base time control: 6 sec + 0.1 sec
(2) : 2 x (6 sec + 0.1 sec) = 12 sec + 0.2 sec
(4) : 24 sec + 0.4 sec
(8) : 48 sec + 0.8 sec
(16): 96 sec + 1.6 sec

QX6700 @ 3.05 GHz
100 positions per match, each position twice (reversed colors)

The mean Elo increase per doubling in time is 117.94 +/- 54. If the Elo increases for doubling from the base time to 12 sec + 0.2 sec is discarded, the increase per doubling is 108.16 +/- 38.52.

I will not claim that there are no problems with my testing methodology, but you will have to point out to me exactly what the problems are. I understand that these numbers do not correspond with your expectations, but this data does have a little bit of support in that it corresponds with the results found by the authors of Dirty and Spandrel.

If you do point out some plausible flaws, I would be willing to redo the study with corrections made to the methodology.

Ok, I was away for the weekend, so the answer comes with a bit of delay Smile.
I have 2 major problems with your testing method.
First and the most important is time controls. At least for the first 3 time controls (basically for any TC where average time per move is less than 1 sec) the Elo differences tend to be greatly exaggerated. This is a well-known fact. It's not due to search/evaluation algorithm weaknesses, but TC and interface problem differences which make blunders more often and therefore effectively lower Elo. You can also see then by the extremely low draw percentage. Btw. these extremely short TCs are used in engine tuning exactly because they exaggerate differences so any change you make to the engine is more easily recognized as positive or negative.


I used these time controls because I needed to know the change in Elo with thinking times in this range. I do not expect that the Elo increase holds for longer time controls. But this is what I had in mind when I said "shorter thinking times". However, due to search improvements and hardware improvements, I would assume that the depths reached at these time controls may not be inferior to those reached in the tests that determined the 50 to 70 Elo increase per doubling. Which I assumed occurred before How Computers Play Chess was published, where this "fact" was stated.

By the way, I think that extremely fast time controls are used in engine tuning in order to accumulate enough games in a reasonably amount of time in order to achieve some statistical significance. At least that is the rationale most authors use.

Milos wrote:

The second problem is related to the wide range of Elo differences in your list. I have no clue how you are selecting opponents, but when testing 1 engine with various TC's all opponents have to be the same, always with the same TC (for opponent, so there should be no same opponents with different TCs per opponent). Opponents should also be selected in a way that the weakest is slightly stronger than the tested engine at the shortest TC, while the strongest is slightly weaker then the tested engine at the longest TC and there should be no more than 3 doubling TCs for the tested engine since that would already give more than 200 Elo difference which is quite large.


In my tests, each engine played every other engine at the same time control. Based on those results, the opponents for a certain engine at a certain time control were determined. Those opponents were in a +/- 100 Elo interval from the intermediate rating for that engine/TC.

I agree that what you suggest would give more precise results, but I am not certain that this would give more accurate results. However, I am willing to conduct some tests in the manner you suggest. Let's decide upon the the time controls (something reasonable that takes less than one month to complete on a quad) and engines. What are your suggestions?
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Subject Author Date/Time
Who is stronger at chess? Computers or Humans? Maurizio Maglio Tue Feb 21, 2012 6:19 pm
      Re: Who is stronger at chess? Computers or Humans? Dr.Wael Deeb Tue Feb 21, 2012 6:28 pm
            Re: Who is stronger at chess? Computers or Humans? Maurizio Maglio Tue Feb 21, 2012 6:32 pm
      Re: Who is stronger at chess? Computers or Humans? Andrew R. Case Tue Feb 21, 2012 6:45 pm
            Re: Who is stronger at chess? Computers or Humans? Maurizio Maglio Tue Feb 21, 2012 6:50 pm
            Re: Who is stronger at chess? Computers or Humans? Julien MARCEL Tue Feb 21, 2012 7:01 pm
      Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Tue Feb 21, 2012 8:26 pm
            Re: Who is stronger at chess? Computers or Humans? Maurizio Maglio Tue Feb 21, 2012 8:45 pm
                  Re: Who is stronger at chess? Computers or Humans? Uri Zlatnik Tue Feb 21, 2012 10:08 pm
                        Re: Who is stronger at chess? Computers or Humans? Julien MARCEL Tue Feb 21, 2012 10:10 pm
                              Re: Who is stronger at chess? Computers or Humans? Uri Zlatnik Tue Feb 21, 2012 10:13 pm
                              Re: Who is stronger at chess? Computers or Humans? Dr.Wael Deeb Tue Feb 21, 2012 10:15 pm
                              Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Tue Feb 21, 2012 10:43 pm
                              Re: Who is stronger at chess? Computers or Humans? Uri Zlatnik Sun Feb 26, 2012 5:54 pm
                                    Re: Who is stronger at chess? Computers or Humans? Ignacio Garcia Mon Feb 27, 2012 2:03 am
                        Re: Who is stronger at chess? Computers or Humans? Dr.Wael Deeb Tue Feb 21, 2012 10:11 pm
                              Re: Who is stronger at chess? Computers or Humans? Uri Zlatnik Tue Feb 21, 2012 11:24 pm
                                    Re: Who is stronger at chess? Computers or Humans? Julien MARCEL Tue Feb 21, 2012 11:29 pm
                                    Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Tue Feb 21, 2012 11:32 pm
                                    Re: Who is stronger at chess? Computers or Humans? Maurizio Maglio Wed Feb 22, 2012 9:15 am
                                          Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Wed Feb 22, 2012 11:53 am
                                                Re: Who is stronger at chess? Computers or Humans? Uri Zlatnik Thu Feb 23, 2012 3:33 am
                                                      Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Thu Feb 23, 2012 8:49 am
                                                Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Thu Feb 23, 2012 5:14 pm
                                                      Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Thu Feb 23, 2012 7:43 pm
                                                            Re: Who is stronger at chess? Computers or Humans? Uri Zlatnik Thu Feb 23, 2012 11:15 pm
                                                            Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Fri Feb 24, 2012 4:09 am
                                                                  Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Fri Feb 24, 2012 10:48 am
                                                                        Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Fri Feb 24, 2012 1:10 pm
                                                                  Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Fri Feb 24, 2012 1:23 pm
                                                                        Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Fri Feb 24, 2012 1:51 pm
                                                                              Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Fri Feb 24, 2012 2:58 pm
                                                                                    Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Fri Feb 24, 2012 6:46 pm
                                                                  Re: Who is stronger at chess? Computers or Humans? Adam Hair Fri Feb 24, 2012 2:12 pm
                                                                        Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Fri Feb 24, 2012 2:39 pm
                                                                              Re: Who is stronger at chess? Computers or Humans? Adam Hair Fri Feb 24, 2012 5:10 pm
                                                                                    Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Fri Feb 24, 2012 10:25 pm
                                                                                          Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Sat Feb 25, 2012 1:17 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Sat Feb 25, 2012 2:57 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Sat Feb 25, 2012 9:42 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Adam Hair Sat Feb 25, 2012 3:34 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Sun Feb 26, 2012 11:28 pm
                                                                                          Re: Who is stronger at chess? Computers or Humans? Adam Hair Mon Feb 27, 2012 4:39 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Mon Feb 27, 2012 5:40 pm
                                                                                          Re: Who is stronger at chess? Computers or Humans? Adam Hair Tue Feb 28, 2012 2:04 am
                                                                                    Re: Who is stronger at chess? Computers or Humans? Norman Schmidt Sat Feb 25, 2012 3:36 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Adam Hair Sat Feb 25, 2012 3:46 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Norman Schmidt Sat Feb 25, 2012 3:52 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Norman Schmidt Sat Feb 25, 2012 4:05 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Adam Hair Mon Feb 27, 2012 4:49 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Ed Schroder Mon Feb 27, 2012 10:51 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Mon Feb 27, 2012 4:19 pm
                                                                                          Re: Who is stronger at chess? Computers or Humans? Ed Schroder Mon Feb 27, 2012 7:21 pm
                                                                                          Re: Who is stronger at chess? Computers or Humans? Carl Bicknell Mon Feb 27, 2012 10:26 pm
                                                                                          Re: Who is stronger at chess? Computers or Humans? Oliver Brausch Tue Feb 28, 2012 4:15 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Carl Bicknell Tue Feb 28, 2012 8:58 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Ricardo Barreira Tue Feb 28, 2012 9:29 am
                                                                                          Re: Who is stronger at chess? Computers or Humans? Uri Zlatnik Fri Mar 09, 2012 9:47 pm
                                          Re: Who is stronger at chess? Computers or Humans? Guy Marfan Mon Feb 27, 2012 2:52 pm
                                    Re: Who is stronger at chess? Computers or Humans? Dr.Wael Deeb Wed Feb 22, 2012 10:43 am
                                    Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Thu Feb 23, 2012 5:27 pm
                              Re: Who is stronger at chess? Computers or Humans? Ricardo Barreira Wed Feb 22, 2012 9:24 am
                  Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Tue Feb 21, 2012 11:15 pm
      Re: Who is stronger at chess? Computers or Humans? Sean Evans Sun Feb 26, 2012 12:00 am
            Re: Who is stronger at chess? Computers or Humans? Sedat Canbaz Sun Feb 26, 2012 10:07 am
      Re: Who is stronger at chess? Computers or Humans? Eduard Nemeth Tue Feb 28, 2012 2:12 am
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: General Topics

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads