ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

POLL:Man vs Machine ?
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: General Topics Flat
View previous topic :: View next topic  
Author Message
Don Dailey



Joined: 29 Apr 2008
Posts: 4320
PostPost subject: Re: POLL:Man vs Machine ?    Posted: Tue Feb 21, 2012 6:11 pm Reply to topic Reply with quote
MM wrote:
Don wrote:
MM wrote:

Hi Don, your idea is interesting but i don't think it can work.

As u say, we are getting the point where we don't have rielable data on how strong computers are compared to human.

Whatever result could happen with your method, anybody could say'' but there was no 2800...Carlsen plays differently....handicap is in favour of the machine ...and many other complaints.



Science is all about making interpolations like this. People will complain that the test should have been like this or like that, but there is NO WAY to perfectly construct a test and most of their concerns are personal superstitions, not based on sound thinking.

You believe that there is something special and different about Carlsen that somehow invalidates any test with weaker players, but he does not play any differently from anyone else, he is just stronger. A 1900 is stronger than a 1700 player too, but there is nothing special that separates them other than ELO. The 1900 player is not made out of special stuff. There is really only ONE thing special about Carlsen, and that is that he is the strongest player in the world. That makes him special only because we attach a degree of significance to it. In fact I believe that Carlsen pretty much sucks. I am a big fan and he is my current favorite player and I admire him for being so strong so young, but he sucks! I am sure that he plays hundreds of ELO below optimal and there is nothing special about that. No only that but computers suck too and Carlsen is almost certainly at least 200 ELO below the best computer programs. So he cannot beat players that suck!

I say of this to illustrate that there nothing special that separates the top players from everyone else other than what separates YOU from players a few hundred ELO weaker.

So I think my test is perfect valid. If you get players within 400 ELO of the top players in strength and get a large enough sample of games at time odds you can get a very good estimate of how much stronger computer programs are. We can do that pretty easily because we know their ELO and we know how much the ELO increases with programs when time is added. It won't be perfect but it will much better than what we have now.

Quote:


At the time of the match Rybka 3 - Milov (2700), people were generally convinced that, even with Rybka Handicapped, Milov would have been crushed bad.
As you know Milov won that match (handicap match) and nothing changed.

There's a party pro-computer (very large) and a party pro-human (very little).

The error of Rybka team has been, in my point of view, to make an handicap match.

Wanna do a match? Ok, lets fight but no handicap.

I think Milov would have agreed.

Like Fern used to say...only my thoughts...

Best Regards


Hi Don, i didn't contest your idea, i like it but, speaking in general, i am a bit skeptical about the reaction of people, and not about ideas, at least this idea of yours, anyway..
please give an example to make me understand better.
If you found that on avarage of 2400 elo, humans score 50% against Houdini, having 6 times Houdini's time, what does it involves?

I mean, i'm not a math, i need an example how to translate a score from weaker players to Carlsen or high level player.

Thank you

Regards


I can make up a simple example. We take 4 fide players rated 2600 for example and they all play 100 games so that we have 400 games total. In order to get a lot of games we play matches at a fast time control, but not ridiculous. Let's say Fischer game in 15 minutes plus 10 or 15 second increment, something like that. We take a program such as Komodo and rig it to play at a level where it is roughly winning 50% of the games. I don't know what the level might be, but it might require some trial and error. Let's say it turns out that we need to play 10x faster - but Komodo is rigged to "pretend" to take more time for the benefit of the human player but in reality it's player 1.5 minutes + 1 second which is 10x faster. No pondering.

When the match is complete, we can easily estimate Komodo's rating at this time control under these conditions against human players.

Let's pretend that Komodo comes out winning 50% of the games and thus gets rating at 2600 ELO relatively to humans at this time control playing 10x faster on this hardware. So now the question is how strong would Komodo be had it played at the SAME time control which is 10x longer? That can be computed by simply running a series of automated tests. When it's done we have a rating estimate.

Larry Kaufman would probably suggest a rating adjustment based on computer vs computer play as the difference would be slightly overstated. I don't remember his formula but if it turned that Komodo plays at 3050 level Larry might calculate that it's overstated by 50 ELO (For example.) Since pondering is turned off we also have to ADD back a few ELO for that, perhaps 30 or 40 ELO. After these calculations we have a "reasonable" estimate of the ELO of computers vs humans but only at this exact time control.

I think this estimate could be directly compared to Carlsen or any other player and we would not be off by very much.

I think there is one more important element in this test. The human needs to be properly motivated just as he would be in any other rated match situation. The results could be widely advertised and that serves as motivation or he can be given monetary compensation for each win and draw - but the main point is that match conditions cannot be too relaxed or casual or experimental as that is likely to skew the results. It should not be played online with the master sitting at his computer and checking email and such.

Like any other kind of interpolation, it is desirable to minimize how much you have to interpolate! For example it is better to get the strongest players possible, if you have to settle for 2400 players instead of 2600 or 2700 players, the interpolation is stretched and the margin of error is likely to be wider. If the computer has to play too fast to make the match even, there is more error too. And of course the faster the human has to play (in order to get larger samples) the less relevant the result is to "match conditions." In fact the result is not accurate except for the time control chosen - so if you want to estimate how strong the computer is at longer time controls relative to humans you have to do another interpolation calculation that is subject to more error. We don't really have a good way to make that calculation - we just know that humans do better with longer time controls but there is no solid reliable data that will tell us how much to adjust the computer by for that.
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Subject Author Date/Time
POLL:Man vs Machine ? Sedat Canbaz Mon Feb 20, 2012 11:19 am
      Re: POLL:Man vs Machine ? Maurizio Maglio Mon Feb 20, 2012 12:22 pm
      Re: POLL:Man vs Machine ? Don Dailey Mon Feb 20, 2012 5:25 pm
            Re: POLL:Man vs Machine ? Maurizio Maglio Mon Feb 20, 2012 6:10 pm
                  Re: POLL:Man vs Machine ? Sedat Canbaz Mon Feb 20, 2012 6:27 pm
                        Re: POLL:Man vs Machine ? Don Dailey Mon Feb 20, 2012 6:35 pm
                              Re: POLL:Man vs Machine ? Sedat Canbaz Mon Feb 20, 2012 6:45 pm
                        Re: POLL:Man vs Machine ? Maurizio Maglio Mon Feb 20, 2012 6:49 pm
                              Re: POLL:Man vs Machine ? Sedat Canbaz Mon Feb 20, 2012 8:03 pm
                                    Re: POLL:Man vs Machine ? Maurizio Maglio Mon Feb 20, 2012 8:27 pm
                        Re: POLL:Man vs Machine ? Mel Cooper Mon Feb 20, 2012 7:04 pm
                              Re: POLL:Man vs Machine ? Maurizio Maglio Mon Feb 20, 2012 7:15 pm
                                    Re: POLL:Man vs Machine ? Mel Cooper Mon Feb 20, 2012 7:23 pm
                                          Re: POLL:Man vs Machine ? Maurizio Maglio Mon Feb 20, 2012 7:53 pm
      Re: POLL:Man vs Machine ? Michael Golub Mon Feb 20, 2012 10:21 pm
            Re: POLL:Man vs Machine ? Sedat Canbaz Mon Feb 20, 2012 10:45 pm
                  Re: POLL:Man vs Machine ? Maurizio Maglio Tue Feb 21, 2012 9:50 am
                        Re: POLL:Man vs Machine ? Sedat Canbaz Tue Feb 21, 2012 11:35 am
                              Re: POLL:Man vs Machine ? Thomas Mayer Tue Feb 21, 2012 11:41 am
                              Re: POLL:Man vs Machine ? Thomas Mayer Tue Feb 21, 2012 11:47 am
                                    Re: POLL:Man vs Machine ? Sedat Canbaz Tue Feb 21, 2012 1:30 pm
                              Re: POLL:Man vs Machine ? Maurizio Maglio Tue Feb 21, 2012 2:18 pm
                        Re: POLL:Man vs Machine ? Don Dailey Tue Feb 21, 2012 1:08 pm
                              Re: POLL:Man vs Machine ? Carl Bicknell Tue Feb 21, 2012 1:15 pm
                                    Re: POLL:Man vs Machine ? Don Dailey Tue Feb 21, 2012 2:09 pm
                                          Re: POLL:Man vs Machine ? Carl Bicknell Tue Feb 21, 2012 2:35 pm
                                                Re: POLL:Man vs Machine ? Uri Blass Thu Feb 23, 2012 10:02 am
                              Re: POLL:Man vs Machine ? Sedat Canbaz Tue Feb 21, 2012 2:38 pm
                                    Re: POLL:Man vs Machine ? Maurizio Maglio Tue Feb 21, 2012 3:10 pm
                                          Re: POLL:Man vs Machine ? Sedat Canbaz Tue Feb 21, 2012 3:43 pm
                                                Re: POLL:Man vs Machine ? Maurizio Maglio Tue Feb 21, 2012 4:19 pm
                                                      Re: POLL:Man vs Machine ? Sedat Canbaz Tue Feb 21, 2012 5:14 pm
                                    Re: POLL:Man vs Machine ? Don Dailey Tue Feb 21, 2012 3:17 pm
                                          Re: POLL:Man vs Machine ? Maurizio Maglio Tue Feb 21, 2012 3:58 pm
                                                Re: POLL:Man vs Machine ? Don Dailey Tue Feb 21, 2012 5:15 pm
                                                      Re: POLL:Man vs Machine ? Maurizio Maglio Tue Feb 21, 2012 5:42 pm
                                                            Re: POLL:Man vs Machine ? Don Dailey Tue Feb 21, 2012 6:11 pm
                                                                  Re: POLL:Man vs Machine ? Maurizio Maglio Tue Feb 21, 2012 6:27 pm
                                                                        Re: POLL:Man vs Machine ? Jean ahlbeck Wed Feb 22, 2012 2:20 am
                                                                              Re: POLL:Man vs Machine ? Sedat Canbaz Thu Feb 23, 2012 7:15 am
                                                                                    Re: POLL:Man vs Machine ? Uri Blass Thu Feb 23, 2012 10:08 am
                                                                                          Re: POLL:Man vs Machine ? Sedat Canbaz Thu Feb 23, 2012 11:44 am
                                                                              Re: POLL:Man vs Machine ? Sedat Canbaz Thu Feb 23, 2012 7:16 am
                                                                                    Re: POLL:Man vs Machine ? Sedat Canbaz Thu Feb 23, 2012 8:16 am
                                                                                          Re: POLL:Man vs Machine ? Jean ahlbeck Thu Feb 23, 2012 10:14 am
                                                                                          Re: POLL:Man vs Machine ? Sedat Canbaz Thu Feb 23, 2012 11:37 am
                                                                                          Re: POLL:Man vs Machine ? Jean ahlbeck Thu Feb 23, 2012 4:01 pm
                                                                                          Re: POLL:Man vs Machine ? Sedat Canbaz Thu Feb 23, 2012 7:32 pm
                                                                                          Re: POLL:Man vs Machine ? Dr.Wael Deeb Thu Feb 23, 2012 8:06 pm
                                                                                          Re: POLL:Man vs Machine ? Jean ahlbeck Fri Feb 24, 2012 1:28 am
                                                                                          Re: POLL:Man vs Machine ? Jean ahlbeck Fri Feb 24, 2012 1:52 am
                                                                                          Re: POLL:Man vs Machine ? Sedat Canbaz Fri Feb 24, 2012 11:20 am
                                                                                          Re: POLL:Man vs Machine ? Sedat Canbaz Wed Feb 29, 2012 11:19 pm
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: General Topics

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads