TalkChess.com
Hosted by Your Move Chess & Games

Author Message
Don Dailey

Joined: 29 Apr 2008
Posts: 4310

Post subject: Re: Number 1 engine on long time controls.    Posted: Tue Feb 28, 2012 12:30 am

That is very interesting stuff Jose!

Based on this idea and what Uri also suggested (which I think are the same basic concept) I started a new study - one that will probably take weeks to complete because it's being run on a slow quad laptop - a spare machine that I do not use much.

The new study combines both program from levels 00 to 08 and rates them all together in a massive round robin. The result will be plotted on a graph and I can handicap Houdini as Uri suggests by adjusting the x-axis, which is the equivalent of handicapping one of the programs by time. So if I add 0.5 to Houdini's X-AXIS for example it's like handicapping it by half a doubling or the same as giving Komodo 1.414 times more time.

The idea is to make an adjustment that causes the lines to intersect at some arbitrary point near the center. The intersection point will define the ELO rating for which they are are equivalent given the specified handicap.

It looks like I will only get 20 or 30 games per day per player (18 players) so it will be a few days before there is even enough data to produce relatively stable lines and a few weeks for the lines to really be very precise.

Ajedrecista wrote:
Hello:

Milos wrote:
 Uri Blass wrote: I disagree about the following: "if it is gaining more ELO per doubling that Houdni, then the logicial conclusion is that it WILL overtake it." One example may be enough to prove that the conclusion is wrong and I guess that houdini 32 bits gain more ELO per doubling than houdini 64 bits because of diminishing returns but houdini 32 bits is not going to overtake houdini 64 bits at long time control. I think that you need to start with not equal time control but with time control that gives result that is close to 50% If you find that the program that use more time earns more from doubling in this case then it is more logical to think that it can beat the stronger program at long time control and even in this case it is not something that I feel sure about it.

Finally someone who understands and doesn't write the usual nonsense.
I agree with everything you write.
Don is as usual writing a load of crap and pretending he doesn't understand (for marketing reasons).
He uses extremely short time controls where the initial difference between 2 engines is exaggerated and then tries to imply the conclusion on something that is more than 3 orders of magnitude longer TC. This is at least 10 doublings and just for fun I extrapolated his table to long TCs (40moves/120mins) using the simplest linear extrapolation.
So lets see (40moves/120mins is 3min/move and is approximately equivalent to 6000sec+100sec - 10 doublings - Level 10)
 Code: Level where 00 is 6 + 0.1 and each successive level is double.                                                                                                                                                                       Komodo                                                                                            HOUDINI   gains                                                                                            -------  ------                                                                                                 Level 00 -  +143.3                                                                                         Leval 01 -   +97.0   +46.3                                                                                 Leval 02 -   +74.6   +22.4                                                                                 Level 03 -   +52.8   +21.8                                                                                 Level 04 -   +39.5   +13.3                                                                                 Level 05 -   +27.0   +12.5                       Level 06 -   +14.5                                                             Leval 07 -   +2.0                                                                               Level 08 -   -10.5                                                                         Level 09 -   -23.0                                                                               Level 10 -   -35.5

At 40moves/120mins Komodo should be 35.5 Elo stronger than Houdini. (even with shape-preserving cubic extrapolation, at the end Komodo should still be 22Elo stronger!)
In reality Komodo is not stronger at any time control, eventually there might be a point where it becomes equal (asymptotic point with extremely large time per move).

So Don, please stop with nonsense, not all the ppl here are idiots to believe your marketing...

I have seen that Milos has done an extrapolation and I also want to post my own one. With the data provided by Don:

 Code: Level where 00 is 6 + 0.1 and each successive level is double.                                                                                                                                                                               Komodo                                                                                            HOUDINI   gains                                                                                            -------  ------                                                                                 Level 00 -  +143.3                                                                                         Leval 01 -   +97.0   +46.3                                                                                 Leval 02 -   +74.6   +22.4                                                                                 Level 03 -   +52.8   +21.8                                                                                 Level 04 -   +39.5   +13.3                                                                                 Level 05 -   +27.0   +12.5

I have done an adjust by least squares by hand, with the only help of a Casio calculator (I hope no errors in my calculations). I explain what I have done a little more:

 Code: x axis ---> the level (time control). y axis ---> Houdini Elo advantage over Komodo.

I did a rescale in the x axis in the following way: x axis is in logarithmic scale (basis e), and I add 1 to each level for avoid ln(0). So, when I put ln(1), it means Don's level 0, and so on; level 10 pointed out by Milos is ln(11) in my x axis. I calculated the least squares by hand with a system of equations, in this way:

 Code: Y(x) = m·ln(x) + n Y numbers go to 'HOUDINI' column. AX = B Symmetric matrix of size 2x2: A = [N, SUM(x_i); SUM(x_i), SUM((x_i)²)] Vector of size 2x1: X = [n, m] Vector of size 2x1: B = [SUM(y_i), SUM((x_i)·(y_i))] N is the number of (x, y) known data; here: N = 6. SUM(x_i) = ln(1) + ln(2) + ... + ln(6) ~ 6.5793 SUM((x_i)²) =[ln(1)]² + [ln(2)]² + ... + [ln(6)]² ~ 9.4099 SUM(y_i) = 143.3 + 97 + ... + 27 = 434.2 SUM(x_i·y_i) = ln(1)·143.3 + ln(2)·97 + ... + ln(6)·27 ~ 334.3384 Det.(A) = N·SUM((x_i)²) - [SUM(x_i)]² ~ 13.1729

Solving by Cramer, I got (more less, roundings included):

 Code: n ~ 143.1793 m ~ -64.5781

So, renaming the level names to their original names (from 0 to 10 instead from 1 to 11) and adding a couple of levels, this is what I get:

 Code: OWN ADJUST: =========== Level where 00 is 6 + 0.1 and each successive level is double.                                                                                                                                                                               Komodo                                                                                            HOUDINI   gains                                                                                            -------  ------                                                                                 Level 00 -  +143.2  Leval 01 -   +98.4   +44.8  Leval 02 -   +72.2   +26.2  Level 03 -   +53.7   +18.5  Level 04 -   +39.2   +14.5  Level 05 -   +27.5   +11.7  Level 06 -   +17.5   +10  Level 07 -    +8.9    +8.6  Level 08 -    +1.3    +7.6  Level 09 -    -5.5    +6.8  Level 10 -   -11.7    +6.2  Level 11 -   -17.3    +5.6  Level 12 -   -22.5    +5.2

So, I get smaller differences than Milos with my model. Given the fact that 2000 games are played in each level, I expect (at not very short time controls, where the draw ratio should rise with longer TC) uncertainties less than ±13 or ±14 Elo (with 95% confidence), that should not be forgotten. With the model I work: uncertainties are proportional to score(Houdini)*score(Komodo) - (draw_ratio)/4, so more draws mean less uncertainties. This model works fine in ranges of scores from 10% to 90%, which is the case here.

I wonder who of us (Milos and me) is more accurate in the extrapolation with the data given (I know that there are tons of ways of extrapolations). I see that in Milos' extrapolation, the missing numbers in 'Komodo gains' column are: {+12.5, +12.5, +12.5, +12.5, +12.5} (a very simple extrapolation, as Milos stated), which does not seem accurate for me because everyone can expect less gains with higher levels (in this sense, my extrapolation is a bit better). With cubic extrapolation, Milos gets -22 in 'HOUDINI' column (I assume that in level 10), which gets closer to my extrapolation, but is still a little far.

As each level doubles the previous level, maybe it could have more sense that my x axis was rescaled with logarithms of basis 2 (instead basis e), but it was much easier to me doing the math with natural logarithms. Comments, corrections, etc. are welcome, so please leave your impressions. Good luck to all the programmers with their engines!

Regards from Spain.

Ajedrecista.

_________________
"Your superior intellect is no match for our puny weapons." -Kang and Kodos
 Display posts from previous: All Posts1 Day7 Days2 Weeks1 Month3 Months6 Months1 Year Oldest FirstNewest First
Subject Author Date/Time
Teodoriu Catalin Fri Feb 24, 2012 7:17 pm
Uri Blass Fri Feb 24, 2012 7:52 pm
Teodoriu Catalin Sat Feb 25, 2012 8:02 am
Don Dailey Fri Feb 24, 2012 10:01 pm
Lonnie Cook Fri Feb 24, 2012 10:21 pm
Don Dailey Fri Feb 24, 2012 11:16 pm
Robert Houdart Fri Feb 24, 2012 11:40 pm
Don Dailey Fri Feb 24, 2012 11:51 pm
Robert Houdart Sat Feb 25, 2012 12:13 am
Ignacio Garcia Sat Feb 25, 2012 12:29 am
Don Dailey Sat Feb 25, 2012 12:30 am
Tom Giampietro Sat Feb 25, 2012 1:05 am
Don Dailey Sat Feb 25, 2012 1:40 am
Robert Houdart Sat Feb 25, 2012 1:52 am
Peter Skinner Sat Feb 25, 2012 5:54 am
Carl Bicknell Sat Feb 25, 2012 8:57 am
Ben-Hur Carlos Langoni Sat Feb 25, 2012 10:22 pm
Carl Bicknell Sun Feb 26, 2012 8:00 am
Ben-Hur Carlos Langoni Sun Feb 26, 2012 1:04 pm
George Speight Sat Feb 25, 2012 9:37 am
Uri Blass Sun Feb 26, 2012 10:43 am
Sedat Canbaz Sun Feb 26, 2012 11:54 am
Ben-Hur Carlos Langoni Sun Feb 26, 2012 1:17 pm
Sedat Canbaz Sun Feb 26, 2012 2:05 pm
Sedat Canbaz Sun Feb 26, 2012 2:57 pm
Don Dailey Sun Feb 26, 2012 1:57 pm
Uri Blass Sun Feb 26, 2012 2:33 pm
Ignacio Garcia Sun Feb 26, 2012 3:41 pm
Don Dailey Sun Feb 26, 2012 4:51 pm
Ignacio Garcia Sun Feb 26, 2012 5:34 pm
Ricardo Barreira Sun Feb 26, 2012 7:54 pm
Uri Blass Sun Feb 26, 2012 10:53 pm
Don Dailey Mon Feb 27, 2012 1:35 pm
Jouni Uski Mon Feb 27, 2012 4:18 pm
Robert Houdart Mon Feb 27, 2012 6:26 pm
Don Dailey Mon Feb 27, 2012 6:50 pm
Larry Kaufman Mon Feb 27, 2012 6:56 pm
Milos Stanisavljevic Mon Feb 27, 2012 5:04 pm
Milos Stanisavljevic Mon Feb 27, 2012 5:22 pm
Larry Kaufman Mon Feb 27, 2012 6:31 pm
Don Dailey Mon Feb 27, 2012 6:48 pm
Milos Stanisavljevic Mon Feb 27, 2012 4:52 pm
Larry Kaufman Mon Feb 27, 2012 5:11 pm
Milos Stanisavljevic Mon Feb 27, 2012 5:19 pm
Don Dailey Mon Feb 27, 2012 5:35 pm
Ignacio Garcia Mon Feb 27, 2012 5:58 pm
Don Dailey Mon Feb 27, 2012 6:38 pm
Ignacio Garcia Tue Feb 28, 2012 2:40 pm
Jesús Muñoz Tue Feb 28, 2012 3:06 pm
Don Dailey Tue Feb 28, 2012 9:06 pm
Ignacio Garcia Wed Feb 29, 2012 11:22 am
Jesús Muñoz Mon Feb 27, 2012 6:50 pm
Re: Number 1 engine on long time controls. Don Dailey Tue Feb 28, 2012 12:30 am
Jesús Muñoz Tue Feb 28, 2012 11:31 am
Don Dailey Tue Feb 28, 2012 11:57 am
Don Dailey Tue Feb 28, 2012 12:06 pm
Jouni Uski Tue Feb 28, 2012 2:10 pm
Don Dailey Tue Feb 28, 2012 8:31 pm
Bigler Tue Feb 28, 2012 4:52 pm
Larry Kaufman Tue Feb 28, 2012 4:58 pm
Bigler Tue Feb 28, 2012 5:03 pm
Larry Kaufman Tue Feb 28, 2012 5:46 pm
Terry McCracken Wed Feb 29, 2012 5:37 am
Carl Bicknell Wed Feb 29, 2012 9:19 am
Terry McCracken Wed Feb 29, 2012 5:49 pm
Dr.Wael Deeb Wed Feb 29, 2012 5:59 pm
Terry McCracken Thu Mar 01, 2012 4:52 am
Uri Blass Tue Feb 28, 2012 6:05 pm
Bram Mourik Tue Feb 28, 2012 9:01 pm
Don Dailey Tue Feb 28, 2012 9:30 pm
Dr.Wael Deeb Tue Feb 28, 2012 9:37 pm
Bram Mourik Wed Feb 29, 2012 5:58 pm
Don Dailey Wed Feb 29, 2012 6:25 pm
om prakash Thu Mar 01, 2012 7:00 am
enrico fagiuoli Wed Feb 29, 2012 8:43 pm
Leon Coleman Wed Feb 29, 2012 5:23 am
Don Dailey Wed Feb 29, 2012 12:47 pm
Vincent Diepeveen Thu Mar 01, 2012 11:44 am
Uri Blass Thu Mar 01, 2012 1:30 pm
Don Dailey Thu Mar 01, 2012 1:34 pm
enrico fagiuoli Thu Mar 01, 2012 2:57 pm
Don Dailey Thu Mar 01, 2012 9:48 pm
enrico fagiuoli Fri Mar 02, 2012 11:09 am
Vincent Diepeveen Fri Mar 02, 2012 12:20 pm
Don Dailey Fri Mar 02, 2012 12:26 pm
Vincent Diepeveen Fri Mar 02, 2012 12:51 pm
Dr.Wael Deeb Fri Mar 02, 2012 12:55 pm
Don Dailey Fri Mar 02, 2012 1:49 pm
Dr.Wael Deeb Fri Mar 02, 2012 3:10 pm
Don Dailey Fri Mar 02, 2012 3:30 pm
Dr.Wael Deeb Fri Mar 02, 2012 3:43 pm
Uri Blass Fri Mar 02, 2012 8:14 pm
Dr.Wael Deeb Fri Mar 02, 2012 8:31 pm
Uri Blass Fri Mar 02, 2012 8:45 pm
Larry Kaufman Fri Mar 02, 2012 8:58 pm
Dr.Wael Deeb Fri Mar 02, 2012 9:26 pm
Dr.Wael Deeb Fri Mar 02, 2012 9:23 pm
Carl Bicknell Fri Mar 02, 2012 5:41 pm
Larry Kaufman Fri Mar 02, 2012 6:59 pm
Dr.Wael Deeb Fri Mar 02, 2012 7:10 pm
Carl Bicknell Sat Mar 03, 2012 3:49 pm
Dr.Wael Deeb Sun Mar 04, 2012 8:36 am
Robert Flesher Fri Mar 02, 2012 9:28 pm
Dr.Wael Deeb Fri Mar 02, 2012 9:55 pm
Jon Dart Sat Feb 25, 2012 9:09 pm
Andrew R. Case Sat Feb 25, 2012 9:14 pm
Gabor Szots Thu Mar 01, 2012 2:19 pm
Larry Kaufman Thu Mar 01, 2012 3:21 pm
om prakash Thu Mar 01, 2012 4:16 pm
Larry Kaufman Thu Mar 01, 2012 4:51 pm
om prakash Thu Mar 01, 2012 4:54 pm
George Speight Sun Mar 04, 2012 9:42 am
Jouni Uski Sun Mar 04, 2012 11:35 am
Maurizio Maglio Sun Mar 04, 2012 11:41 am
Don Dailey Fri Mar 02, 2012 1:34 pm

 Jump to: Select a forum Computer Chess Club Forums----------------Computer Chess Club: General TopicsComputer Chess Club: Tournaments and MatchesComputer Chess Club: Programming and Technical DiscussionsComputer Chess Club: Engine Origins Other Forums----------------Chess Thinkers ForumForum Help and Suggestions
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum