TalkChess.com
Hosted by Your Move Chess & Games

Who is stronger? Computer or Humans? How much?
Computer +500 elo  29%  [ 16 ]
Computer 301-500elo  27%  [ 15 ]
Computer 101-300 elo  32%  [ 18 ]
Computer 1-100 elo  3%  [ 2 ]
Equality  1%  [ 1 ]
Human 1-100 elo  3%  [ 2 ]
Human 101-300 elo  0%  [ 0 ]
Human 301-500 elo  0%  [ 0 ]
Human +500 elo  1%  [ 1 ]
Voted : 55
- Poll expired -

Author Message
Milos Stanisavljevic

Joined: 25 Nov 2009
Posts: 1162

Post subject: Re: Who is stronger at chess? Computers or Humans?    Posted: Sun Feb 26, 2012 11:28 pm

 Adam Hair wrote: There have been some hints from different authors that they were getting more than 70 Elo per doubling at shorter search times. I have seen a citation that Levy and Newborn stated in How Computers Play Chess of 50 to 70 Elo per doubling, but I have not seen that confirmed in recent times. So, I decided to see if it was true, for I needed to know for another test. The following list is the result of my testing the approximate gain in Elo for each doubling of thinking time, which should roughly equal the same increase due to doubling speed: [code]Base time control: 6 sec + 0.1 sec (2) : 2 x (6 sec + 0.1 sec) = 12 sec + 0.2 sec (4) : 24 sec + 0.4 sec (8) : 48 sec + 0.8 sec (16): 96 sec + 1.6 sec QX6700 @ 3.05 GHz 100 positions per match, each position twice (reversed colors) The mean Elo increase per doubling in time is 117.94 +/- 54. If the Elo increases for doubling from the base time to 12 sec + 0.2 sec is discarded, the increase per doubling is 108.16 +/- 38.52. I will not claim that there are no problems with my testing methodology, but you will have to point out to me exactly what the problems are. I understand that these numbers do not correspond with your expectations, but this data does have a little bit of support in that it corresponds with the results found by the authors of Dirty and Spandrel. If you do point out some plausible flaws, I would be willing to redo the study with corrections made to the methodology.

Ok, I was away for the weekend, so the answer comes with a bit of delay .
I have 2 major problems with your testing method.
First and the most important is time controls. At least for the first 3 time controls (basically for any TC where average time per move is less than 1 sec) the Elo differences tend to be greatly exaggerated. This is a well-known fact. It's not due to search/evaluation algorithm weaknesses, but TC and interface problem differences which make blunders more often and therefore effectively lower Elo. You can also see then by the extremely low draw percentage. Btw. these extremely short TCs are used in engine tuning exactly because they exaggerate differences so any change you make to the engine is more easily recognized as positive or negative.

The second problem is related to the wide range of Elo differences in your list. I have no clue how you are selecting opponents, but when testing 1 engine with various TC's all opponents have to be the same, always with the same TC (for opponent, so there should be no same opponents with different TCs per opponent). Opponents should also be selected in a way that the weakest is slightly stronger than the tested engine at the shortest TC, while the strongest is slightly weaker then the tested engine at the longest TC and there should be no more than 3 doubling TCs for the tested engine since that would already give more than 200 Elo difference which is quite large.
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
Subject Author Date/Time
Maurizio Maglio Tue Feb 21, 2012 6:19 pm
Dr.Wael Deeb Tue Feb 21, 2012 6:28 pm
Maurizio Maglio Tue Feb 21, 2012 6:32 pm
Andrew R. Case Tue Feb 21, 2012 6:45 pm
Maurizio Maglio Tue Feb 21, 2012 6:50 pm
Julien MARCEL Tue Feb 21, 2012 7:01 pm
Sedat Canbaz Tue Feb 21, 2012 8:26 pm
Maurizio Maglio Tue Feb 21, 2012 8:45 pm
Uri Zlatnik Tue Feb 21, 2012 10:08 pm
Julien MARCEL Tue Feb 21, 2012 10:10 pm
Uri Zlatnik Tue Feb 21, 2012 10:13 pm
Dr.Wael Deeb Tue Feb 21, 2012 10:15 pm
Sedat Canbaz Tue Feb 21, 2012 10:43 pm
Uri Zlatnik Sun Feb 26, 2012 5:54 pm
Ignacio Garcia Mon Feb 27, 2012 2:03 am
Dr.Wael Deeb Tue Feb 21, 2012 10:11 pm
Uri Zlatnik Tue Feb 21, 2012 11:24 pm
Julien MARCEL Tue Feb 21, 2012 11:29 pm
Sedat Canbaz Tue Feb 21, 2012 11:32 pm
Maurizio Maglio Wed Feb 22, 2012 9:15 am
Sedat Canbaz Wed Feb 22, 2012 11:53 am
Uri Zlatnik Thu Feb 23, 2012 3:33 am
Sedat Canbaz Thu Feb 23, 2012 8:49 am
Milos Stanisavljevic Thu Feb 23, 2012 5:14 pm
Sedat Canbaz Thu Feb 23, 2012 7:43 pm
Uri Zlatnik Thu Feb 23, 2012 11:15 pm
Milos Stanisavljevic Fri Feb 24, 2012 4:09 am
Sedat Canbaz Fri Feb 24, 2012 10:48 am
Milos Stanisavljevic Fri Feb 24, 2012 1:10 pm
Sedat Canbaz Fri Feb 24, 2012 1:23 pm
Milos Stanisavljevic Fri Feb 24, 2012 1:51 pm
Sedat Canbaz Fri Feb 24, 2012 2:58 pm
Sedat Canbaz Fri Feb 24, 2012 6:46 pm
Adam Hair Fri Feb 24, 2012 2:12 pm
Milos Stanisavljevic Fri Feb 24, 2012 2:39 pm
Adam Hair Fri Feb 24, 2012 5:10 pm
Milos Stanisavljevic Fri Feb 24, 2012 10:25 pm
Sedat Canbaz Sat Feb 25, 2012 1:17 am
Milos Stanisavljevic Sat Feb 25, 2012 2:57 am
Sedat Canbaz Sat Feb 25, 2012 9:42 am
Adam Hair Sat Feb 25, 2012 3:34 am
Re: Who is stronger at chess? Computers or Humans? Milos Stanisavljevic Sun Feb 26, 2012 11:28 pm
Adam Hair Mon Feb 27, 2012 4:39 am
Milos Stanisavljevic Mon Feb 27, 2012 5:40 pm
Adam Hair Tue Feb 28, 2012 2:04 am
Norman Schmidt Sat Feb 25, 2012 3:36 am
Adam Hair Sat Feb 25, 2012 3:46 am
Norman Schmidt Sat Feb 25, 2012 3:52 am
Norman Schmidt Sat Feb 25, 2012 4:05 am
Adam Hair Mon Feb 27, 2012 4:49 am
Ed Schroder Mon Feb 27, 2012 10:51 am
Milos Stanisavljevic Mon Feb 27, 2012 4:19 pm
Ed Schroder Mon Feb 27, 2012 7:21 pm
Carl Bicknell Mon Feb 27, 2012 10:26 pm
Oliver Brausch Tue Feb 28, 2012 4:15 am
Carl Bicknell Tue Feb 28, 2012 8:58 am
Ricardo Barreira Tue Feb 28, 2012 9:29 am
Uri Zlatnik Fri Mar 09, 2012 9:47 pm
Guy Marfan Mon Feb 27, 2012 2:52 pm
Dr.Wael Deeb Wed Feb 22, 2012 10:43 am
Milos Stanisavljevic Thu Feb 23, 2012 5:27 pm
Ricardo Barreira Wed Feb 22, 2012 9:24 am
Sedat Canbaz Tue Feb 21, 2012 11:15 pm
Sean Evans Sun Feb 26, 2012 12:00 am
Sedat Canbaz Sun Feb 26, 2012 10:07 am
Eduard Nemeth Tue Feb 28, 2012 2:12 am

 Jump to: Select a forum Computer Chess Club Forums----------------Computer Chess Club: General TopicsComputer Chess Club: Tournaments and MatchesComputer Chess Club: Programming and Technical DiscussionsComputer Chess Club: Engine Origins Other Forums----------------Chess Thinkers ForumForum Help and Suggestions
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum