Nakamura vs. Komodo

[JJJ]

If komodo is 2900 elo human rating at 1 sec / move, it could be the challenge I d like to see

2900 is close to the highest elo of Magnus Carslen. Could be interesting to see strong GM ( around 2700 ) play some against it.

[duncan]

Nakamura played well. So, now we know that draw is best thing to expect at these handicap, will you change them Larry ?

What is the currently elo of komodo based only with his game against human ? Can you determine it ?

Well, the handicap obviously depends on the opponent. I don't know when or if we'll have another chance to play a 2800 level opponent. I'll try to think of some new ideas for matches with "ordinary" GMs.

I plan to calculate Komodo's performance rating at each handicap so far. The question is how much to add to those numbers for the handicap. Based on both my own tests and those of Kai, we have some idea, but it's not very exact. Anyway, I'll post something on this once I have the time.

Thanks Larry and Mark for this match. By now you are able pick the right handicaps in order to have balanced matches, as the last matches had shown. People are getting pretty good too at predicting the result, 2.5/4 for Komodo was the central value in the poll answers.

Naka played very well, congratulations.

The overall performance of Komodo versus Naka is probably somewhere around 3200 ELO. Also, this performance is much more stable than computer ratings. An improvement of 200 ELO points of Komodo on CCRL will be reflected only in some minor increase of the balanced handicap against top human. This "compression" of improvement comes from the dilation of the handicap with the strength, engine improves not much faster than the same handicap increases too in its ELO value with the strength improvement of the engine. Probably Naka can draw at f7 + c7 odds even against the perfect engine, because there are only 1000-1200 more ELO points to Komodo, and the value of the handicap against top human increases too to huge value at this level of engine strength.

Other ways of giving odds seem less spectacular. Fixed depth is meaningless, a 1992 Mephisto is probably not far away from Komodo at fixed depth (in fact I don't know which is stronger). Time odds: not intuitive for most chess players, not clear in its magnitude even to many CC specialists. That Russian GM commentary on twitch was very interesting, at one time he asked something like: "do computers really need time to think? I don't see any difference in the level of playing whether it thinks for 1 second or 1 minute". Time odds would surely not impress him. At 1 second/move Komodo is roughly 2900 or so in human ratings, at 1 minute/move maybe 3100, both beyond the usual human play. Maybe "taking back" is something new to consider, as humans, even top ones, are particularly vulnerable to some "stupid" blunders, which ruin almost all good intentions.

Thanks again for this entertainment!

PS What was the value of the contempt in these games?

if nakamura had had normal time controls, while komodo keeps his, do you think he would have done better ?

[whereagles]

Komodo at four ply would probably be something like 1800 elo or so, I would guess.

That seems plausible, if not optimistic.

I played a few games at 3 moves (6 plies) with the Mac OS simple chess engine (Sjeng-based?). I should be around 2000-2200 ELO and at those settings I could beat it easily. Moving the setting to 3 sec/move totally turns the tables though.. lol.

[Laskos]

Nakamura played well. So, now we know that draw is best thing to expect at these handicap, will you change them Larry ?

What is the currently elo of komodo based only with his game against human ? Can you determine it ?

Well, the handicap obviously depends on the opponent. I don't know when or if we'll have another chance to play a 2800 level opponent. I'll try to think of some new ideas for matches with "ordinary" GMs.

I plan to calculate Komodo's performance rating at each handicap so far. The question is how much to add to those numbers for the handicap. Based on both my own tests and those of Kai, we have some idea, but it's not very exact. Anyway, I'll post something on this once I have the time.

Thanks Larry and Mark for this match. By now you are able pick the right handicaps in order to have balanced matches, as the last matches had shown. People are getting pretty good too at predicting the result, 2.5/4 for Komodo was the central value in the poll answers.

Naka played very well, congratulations.

The overall performance of Komodo versus Naka is probably somewhere around 3200 ELO. Also, this performance is much more stable than computer ratings. An improvement of 200 ELO points of Komodo on CCRL will be reflected only in some minor increase of the balanced handicap against top human. This "compression" of improvement comes from the dilation of the handicap with the strength, engine improves not much faster than the same handicap increases too in its ELO value with the strength improvement of the engine. Probably Naka can draw at f7 + c7 odds even against the perfect engine, because there are only 1000-1200 more ELO points to Komodo, and the value of the handicap against top human increases too to huge value at this level of engine strength.

Other ways of giving odds seem less spectacular. Fixed depth is meaningless, a 1992 Mephisto is probably not far away from Komodo at fixed depth (in fact I don't know which is stronger). Time odds: not intuitive for most chess players, not clear in its magnitude even to many CC specialists. That Russian GM commentary on twitch was very interesting, at one time he asked something like: "do computers really need time to think? I don't see any difference in the level of playing whether it thinks for 1 second or 1 minute". Time odds would surely not impress him. At 1 second/move Komodo is roughly 2900 or so in human ratings, at 1 minute/move maybe 3100, both beyond the usual human play. Maybe "taking back" is something new to consider, as humans, even top ones, are particularly vulnerable to some "stupid" blunders, which ruin almost all good intentions.

Thanks again for this entertainment!

PS What was the value of the contempt in these games?

if nakamura had had normal time controls, while komodo keeps his, do you think he would have done better ?

Yes, definitely so, although might not be clear in 1-2 games. Komodo can play too in 2 hours, doesn't matter too much.

[Laskos]

If komodo is 2900 elo human rating at 1 sec / move, it could be the challenge I d like to see

2900 is close to the highest elo of Magnus Carslen. Could be interesting to see strong GM ( around 2700 ) play some against it.

A strong GM would probably lose against 24 core machine at 1 s/move. Better play against a smartphone at 1 s/move. Do you think the chess community would be impressed by such time-odds match? I don't think so.

[duncan]

Nakamura played well. So, now we know that draw is best thing to expect at these handicap, will you change them Larry ?

What is the currently elo of komodo based only with his game against human ? Can you determine it ?

Well, the handicap obviously depends on the opponent. I don't know when or if we'll have another chance to play a 2800 level opponent. I'll try to think of some new ideas for matches with "ordinary" GMs.

I plan to calculate Komodo's performance rating at each handicap so far. The question is how much to add to those numbers for the handicap. Based on both my own tests and those of Kai, we have some idea, but it's not very exact. Anyway, I'll post something on this once I have the time.

Thanks Larry and Mark for this match. By now you are able pick the right handicaps in order to have balanced matches, as the last matches had shown. People are getting pretty good too at predicting the result, 2.5/4 for Komodo was the central value in the poll answers.

Naka played very well, congratulations.

The overall performance of Komodo versus Naka is probably somewhere around 3200 ELO. Also, this performance is much more stable than computer ratings. An improvement of 200 ELO points of Komodo on CCRL will be reflected only in some minor increase of the balanced handicap against top human. This "compression" of improvement comes from the dilation of the handicap with the strength, engine improves not much faster than the same handicap increases too in its ELO value with the strength improvement of the engine. Probably Naka can draw at f7 + c7 odds even against the perfect engine, because there are only 1000-1200 more ELO points to Komodo, and the value of the handicap against top human increases too to huge value at this level of engine strength.

Other ways of giving odds seem less spectacular. Fixed depth is meaningless, a 1992 Mephisto is probably not far away from Komodo at fixed depth (in fact I don't know which is stronger). Time odds: not intuitive for most chess players, not clear in its magnitude even to many CC specialists. That Russian GM commentary on twitch was very interesting, at one time he asked something like: "do computers really need time to think? I don't see any difference in the level of playing whether it thinks for 1 second or 1 minute". Time odds would surely not impress him. At 1 second/move Komodo is roughly 2900 or so in human ratings, at 1 minute/move maybe 3100, both beyond the usual human play. Maybe "taking back" is something new to consider, as humans, even top ones, are particularly vulnerable to some "stupid" blunders, which ruin almost all good intentions.

Thanks again for this entertainment!

PS What was the value of the contempt in these games?

if nakamura had had normal time controls, while komodo keeps his, do you think he would have done better ?

Yes, definitely so, although might not be clear in 1-2 games. Komodo can play too in 2 hours, doesn't matter too much.

so do you think it will be possible in the future for kommodo to give pawn and move odds and beat the world champion under normal game conditions. would 200-300 extra elo be sufficient ?

[lkaufman]

Nakamura played well. So, now we know that draw is best thing to expect at these handicap, will you change them Larry ?

What is the currently elo of komodo based only with his game against human ? Can you determine it ?

Well, the handicap obviously depends on the opponent. I don't know when or if we'll have another chance to play a 2800 level opponent. I'll try to think of some new ideas for matches with "ordinary" GMs.

I plan to calculate Komodo's performance rating at each handicap so far. The question is how much to add to those numbers for the handicap. Based on both my own tests and those of Kai, we have some idea, but it's not very exact. Anyway, I'll post something on this once I have the time.

Thanks Larry and Mark for this match. By now you are able pick the right handicaps in order to have balanced matches, as the last matches had shown. People are getting pretty good too at predicting the result, 2.5/4 for Komodo was the central value in the poll answers.

Naka played very well, congratulations.

The overall performance of Komodo versus Naka is probably somewhere around 3200 ELO. Also, this performance is much more stable than computer ratings. An improvement of 200 ELO points of Komodo on CCRL will be reflected only in some minor increase of the balanced handicap against top human. This "compression" of improvement comes from the dilation of the handicap with the strength, engine improves not much faster than the same handicap increases too in its ELO value with the strength improvement of the engine. Probably Naka can draw at f7 + c7 odds even against the perfect engine, because there are only 1000-1200 more ELO points to Komodo, and the value of the handicap against top human increases too to huge value at this level of engine strength.

Other ways of giving odds seem less spectacular. Fixed depth is meaningless, a 1992 Mephisto is probably not far away from Komodo at fixed depth (in fact I don't know which is stronger). Time odds: not intuitive for most chess players, not clear in its magnitude even to many CC specialists. That Russian GM commentary on twitch was very interesting, at one time he asked something like: "do computers really need time to think? I don't see any difference in the level of playing whether it thinks for 1 second or 1 minute". Time odds would surely not impress him. At 1 second/move Komodo is roughly 2900 or so in human ratings, at 1 minute/move maybe 3100, both beyond the usual human play. Maybe "taking back" is something new to consider, as humans, even top ones, are particularly vulnerable to some "stupid" blunders, which ruin almost all good intentions.

Thanks again for this entertainment!

PS What was the value of the contempt in these games?

40

[lkaufman]

Nakamura played well. So, now we know that draw is best thing to expect at these handicap, will you change them Larry ?

What is the currently elo of komodo based only with his game against human ? Can you determine it ?

Well, the handicap obviously depends on the opponent. I don't know when or if we'll have another chance to play a 2800 level opponent. I'll try to think of some new ideas for matches with "ordinary" GMs.

I plan to calculate Komodo's performance rating at each handicap so far. The question is how much to add to those numbers for the handicap. Based on both my own tests and those of Kai, we have some idea, but it's not very exact. Anyway, I'll post something on this once I have the time.

Thanks Larry and Mark for this match. By now you are able pick the right handicaps in order to have balanced matches, as the last matches had shown. People are getting pretty good too at predicting the result, 2.5/4 for Komodo was the central value in the poll answers.

Naka played very well, congratulations.

The overall performance of Komodo versus Naka is probably somewhere around 3200 ELO. Also, this performance is much more stable than computer ratings. An improvement of 200 ELO points of Komodo on CCRL will be reflected only in some minor increase of the balanced handicap against top human. This "compression" of improvement comes from the dilation of the handicap with the strength, engine improves not much faster than the same handicap increases too in its ELO value with the strength improvement of the engine. Probably Naka can draw at f7 + c7 odds even against the perfect engine, because there are only 1000-1200 more ELO points to Komodo, and the value of the handicap against top human increases too to huge value at this level of engine strength.

Other ways of giving odds seem less spectacular. Fixed depth is meaningless, a 1992 Mephisto is probably not far away from Komodo at fixed depth (in fact I don't know which is stronger). Time odds: not intuitive for most chess players, not clear in its magnitude even to many CC specialists. That Russian GM commentary on twitch was very interesting, at one time he asked something like: "do computers really need time to think? I don't see any difference in the level of playing whether it thinks for 1 second or 1 minute". Time odds would surely not impress him. At 1 second/move Komodo is roughly 2900 or so in human ratings, at 1 minute/move maybe 3100, both beyond the usual human play. Maybe "taking back" is something new to consider, as humans, even top ones, are particularly vulnerable to some "stupid" blunders, which ruin almost all good intentions.

Thanks again for this entertainment!

PS What was the value of the contempt in these games?

if nakamura had had normal time controls, while komodo keeps his, do you think he would have done better ?

Yes, definitely so, although might not be clear in 1-2 games. Komodo can play too in 2 hours, doesn't matter too much.

so do you think it will be possible in the future for kommodo to give pawn and move odds and beat the world champion under normal game conditions. would 200-300 extra elo be sufficient ?

Well, it's far from clear who would win a match tomorrow at f7 odds between Nakamura and Komodo at 45' + 15", since they only played one game and it was a draw. Carlsen might be a bit tougher, and doubling the time limit would favor him, but I think in a few years the top engine would win such a match, though most of the games would be draws.

[Laskos]

so do you think it will be possible in the future for kommodo to give pawn and move odds and beat the world champion under normal game conditions. would 200-300 extra elo be sufficient ?

To beat the champion with say 70-80% probability (to call it consistent beating) at regular TC with f7 odds would require probably more, maybe 400-600 extra CCRL ELO. I am not sure how Komodo stands today on this handicap against a 2800 GM, and it depends also on individual GM too.

[Laskos]

I remember when simulating these handicaps that the rule of thumb would be that for human against computer "material odds beat positional odds" (for the same nominal value of odds).

was the human a super grand-master?

I simulated the top GM too, the situation is the same, humans would better choose material handicap instead of a positional one for the same nominal value in centipawns of handicap. I can show the results, although it is lengthy:

The performance of Machine (Komodo LTC) versus Human (Komodo STC) from regular starting positions (openings 2moves_v1.pgn):

Code: Select all

Score of Machine vs Human: 1532 - 29 - 439  [0.876] 2000
ELO difference: 339
Finished match

The following two opening positions give almost identical evaluation of the handicap:

Positional:
[D]rnbqkbnr/pppppppp/8/8/4PP2/5N2/PPPP2PP/RNBQKB1R w KQkq -

Code: Select all

Engine: Komodo 9.3 64-bit (4096 MB)
by Don Dailey, Larry Kaufman, Mark Lefler

29.01  3:16   +0.92    1.c4 c5 2.d4 cxd4 3.Nxd4 e6 4.Nc3 Nc6 
                       5.Be3 Nh6 6.Be2 Be7 7.O-O O-O 8.Qd3 f6 
                       9.Rad1 b6 10.a3 Nf7 11.b4 Nxd4 
                       12.Bxd4 Bb7 13.Be3 d6 14.Nb5 (804.547.930) 4090 

30.01  3:42   +0.90    1.c4 e6 2.Nc3 c5 3.d4 cxd4 4.Nxd4 Nc6 
                       5.Be3 Nh6 6.Be2 Be7 7.O-O O-O 8.Qd3 f6 
                       9.Rad1 b6 10.a3 Nf7 11.b4 Nxd4 
                       12.Bxd4 Bb7 13.Be3 d6 14.Rd2 (909.886.536) 4093 

31.01  5:37   +0.97++  1.c4 e6 2.Nc3 (1.380.858.073) 4092 

31.01  6:01   +0.90--  1.c4 e6 (1.477.978.172) 4088 

31.01  6:31   +0.91    1.c4 e6 2.Nc3 c5 3.d4 cxd4 4.Nxd4 Nc6 
                       5.Be3 Nh6 6.Be2 Be7 7.O-O O-O 8.Qd3 f6 
                       9.a3 b6 10.Rad1 Nf7 11.b4 Nxd4 
                       12.Bxd4 Bb7 13.Be3 d6 14.Nb5 (1.603.084.173) 4092 

32.01  8:06   +0.98++  1.c4 e6 2.Nc3 (1.991.904.940) 4092

Material:
[D]rnbqkbnr/p1pppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq -

Code: Select all

Engine: Komodo 9.3 64-bit (4096 MB)
by Don Dailey, Larry Kaufman, Mark Lefler

28.01  3:16   +1.01    1.Nf3 Nf6 2.d4 Ba6 3.Nbd2 e6 4.c4 Bb7 
                       5.e3 c5 6.Bd3 d6 7.O-O Nbd7 8.Qc2 h6 
                       9.b3 Be7 10.Bb2 O-O 11.h3 Rb8 12.Ne4 Qc7 
                       13.Nc3 a6 14.Qd2 (756.606.512) 3844 

29.01  4:03   +0.99    1.Nf3 Nf6 2.d4 Ba6 3.Nbd2 e6 4.c4 Bb7 
                       5.e3 c5 6.Bd3 Be7 7.O-O O-O 8.b3 d6 
                       9.h3 Nbd7 10.Qc2 h6 11.Bb2 a5 12.Bc3 Qc7 
                       13.Rfe1 Rab8 14.Ne4 (938.374.833) 3856 

30.01  5:37   +0.96    1.Nf3 Nf6 2.d4 Ba6 3.Nbd2 e6 4.c4 Bb7 
                       5.e3 Be7 6.Bd3 c5 7.O-O O-O 8.b3 d6 
                       9.h3 a5 10.Bb2 Nbd7 11.Qc2 h6 12.Ne4 Nxe4 
                       13.Bxe4 Bxe4 14.Qxe4 (1.305.455.827) 3870 

30.02  8:17   +1.01    1.d4 Nf6 2.Nf3 Ba6 3.Nbd2 e6 4.c4 Bb7 
                       5.e3 Be7 6.Bd3 c5 7.O-O O-O 8.b3 d6 
                       9.Bb2 Nbd7 10.h3 h6 11.Qc2 Qc7 
                       12.Ne4 Nxe4 13.Bxe4 Bxe4 14.Qxe4 (1.952.013.201) 3924 

31.01  11:38  +0.95    1.d4 Nf6 2.Nf3 e6 3.c4 Bb7 4.a3 Be7 
                       5.Nc3 Ne4 6.Nxe4 Bxe4 7.e3 O-O 8.Bd3 d5 
                       9.O-O Nd7 10.b4 c6 11.c5 h6 12.a4 a6 
                       13.Bb2 Qb8 14.Bc3 (2.791.841.185) 3995

Performance of Human versus Machine positional odds:

Code: Select all

    Program                             Score     %    Av.Op.  Elo    +   -    Draws

  1 Machine                        : 1484.5/2000  74.2    -92     92   13  13   34.8 %
  2 Human                          :  515.5/2000  25.8     92    -92   13  13   34.8 %

Human gain from positional odds: 155 ELO points

Performance of Human versus Machine material odds:

Code: Select all

    Program                            Score     %    Av.Op.  Elo    +   -    Draws

  1 Machine                        : 1319.5/2000  66.0    -58     58   12  11   43.9 %
  2 Human                          :  680.5/2000  34.0     58    -58   11  12   43.9 %

Human gain from material odds: 223 ELO points

For the same nominal value (centipawns), human gains about 40% more from material odds than from positional odds. Therefore the handicap in game 4 was 70cp (instead of apparent 100cp by the eval) in terms of material, the smallest of the 4 games Naka played. I don't think he lost accidentally the fourth game, in fact I predicted that he will have problems in this game.