Me. Gideon Pro (1993) fairly equal to Komodo 9.2: 45.0-55.0

[Laskos]

Thanks to Ed Schroder for enabling Mephisto Gideon Pro (1993) with Winboard and UCI compatibility. This top engine of 1993 is the oldest UCI engine I have. The port allows to test it in various conditions.

It seems since 1993 the main strength transition is from minutes to seconds. Mephisto Gideon Pro at 40 moves in 2 minutes ( 40/2') is fairly equal to Komodo 9.2 at 40 moves in 2 seconds (40/2'').

Code: Select all

Games Completed = 100 of 100 (Avg game length = 214.449 sec)
Settings = Gauntlet/32MB/120000ms in 40 moves/M 800000cp for 1000 moves, D 25000 moves/PGN:2moves_v2.pgn(32000)
Time = 3287 sec elapsed, 0 sec remaining
 1.  Mephisto Gideon Pro (1993)  45.0/100	38-48-14  	(L: m=48 t=0 i=0 a=0)	(D: r=8 i=3 f=2 s=1 a=0)	(tpm=2709.5 d=9.95 nps=246638359)
 2.  Komodo 9.2 64-bit        	55.0/100	48-38-14  	(L: m=38 t=0 i=0 a=0)	(D: r=8 i=3 f=2 s=1 a=0)	(tpm=39.9 d=14.01 nps=1648576)

It is a factor of 60 time control handicap, but one must keep in mind that this facotr increases to longer time controls. On CEGT rating list, used preferentially over CCRL for large ELO spans because CEGT uses uncompressed ELO scale, one can infere that on equal hardware Gideon is about 970 ELO points weaker than Komodo at 40/4'.

Some properties of Gideon, top engine of 1993, compared to Komodo 9.2, top engine of 2015:

Effective Branching Factor:

Gideon

Code: Select all

  n/s: 1.296.602  
  TotTime: 111:49m    SolTime: 111:49m
  Ply: 9   Positions: 30   Avg Nodes: 5847624   Branching = 3.84
  Ply:10   Positions: 30   Avg Nodes:20184746   Branching = 3.45
  Ply:11   Positions: 30   Avg Nodes:69769347   Branching = 3.46

Average EBF at relevant plies: 3.58

Komodo

Code: Select all

  n/s: 1.488.637  
  TotTime: 45:37m    SolTime: 45:37m
  Ply:18   Positions: 30   Avg Nodes: 6304278   Branching = 1.65
  Ply:19   Positions: 30   Avg Nodes:10764642   Branching = 1.71
  Ply:20   Positions: 30   Avg Nodes:18700347   Branching = 1.74
  Ply:21   Positions: 30   Avg Nodes:31329107   Branching = 1.68
  Ply:22   Positions: 30   Avg Nodes:51579137   Branching = 1.65
  Ply:23   Positions: 30   Avg Nodes:81829576   Branching = 1.59

Average EBF at relevant plies: 1.67


The gain from doubling in time at blitz time control is comparable, slightly higher in Komodo:

Gideon: 79 ELO points from 1'+1'' to 2'+2''
Komodo: 103 ELO points from 1'+1'' to 2'+2''
One must remember that the top hardware of 1993 was a x486 at 50MHz, roughly 100 slower than a today core, so this time control is really long time control for Gideon.

Artistic impression: Gideon won all 6-7 games at loosely blitz time control against me, and it seems somewhat more tactical than modern engines. The search depth at blitz in the opening was 7-9 plies for Gideon and roughly double, 16-19 plies for Komodo on one core. Not the same plies, though. Gideon plies are a bit "thicker" ELO-wise.

[lkaufman]

Thanks to Ed Schroder for enabling Mephisto Gideon Pro (1993) with Winboard and UCI compatibility. This top engine of 1993 is the oldest UCI engine I have. The port allows to test it in various conditions.

It seems since 1993 the main strength transition is from minutes to seconds. Mephisto Gideon Pro at 40 moves in 2 minutes ( 40/2') is fairly equal to Komodo 9.2 at 40 moves in 2 seconds (40/2'').

Code: Select all

Games Completed = 100 of 100 (Avg game length = 214.449 sec)
Settings = Gauntlet/32MB/120000ms in 40 moves/M 800000cp for 1000 moves, D 25000 moves/PGN:2moves_v2.pgn(32000)
Time = 3287 sec elapsed, 0 sec remaining
 1.  Mephisto Gideon Pro (1993)  45.0/100	38-48-14  	(L: m=48 t=0 i=0 a=0)	(D: r=8 i=3 f=2 s=1 a=0)	(tpm=2709.5 d=9.95 nps=246638359)
 2.  Komodo 9.2 64-bit        	55.0/100	48-38-14  	(L: m=38 t=0 i=0 a=0)	(D: r=8 i=3 f=2 s=1 a=0)	(tpm=39.9 d=14.01 nps=1648576)

It is a factor of 60 time control handicap, but one must keep in mind that this facotr increases to longer time controls. On CEGT rating list, used preferentially over CCRL for large ELO spans because CEGT uses uncompressed ELO scale, one can infere that on equal hardware Gideon is about 970 ELO points weaker than Komodo at 40/4'.

Some properties of Gideon, top engine of 1993, compared to Komodo 9.2, top engine of 2015:

Effective Branching Factor:

Gideon

Code: Select all

  n/s: 1.296.602  
  TotTime: 111:49m    SolTime: 111:49m
  Ply: 9   Positions: 30   Avg Nodes: 5847624   Branching = 3.84
  Ply:10   Positions: 30   Avg Nodes:20184746   Branching = 3.45
  Ply:11   Positions: 30   Avg Nodes:69769347   Branching = 3.46

Average EBF at relevant plies: 3.58

Komodo

Code: Select all

  n/s: 1.488.637  
  TotTime: 45:37m    SolTime: 45:37m
  Ply:18   Positions: 30   Avg Nodes: 6304278   Branching = 1.65
  Ply:19   Positions: 30   Avg Nodes:10764642   Branching = 1.71
  Ply:20   Positions: 30   Avg Nodes:18700347   Branching = 1.74
  Ply:21   Positions: 30   Avg Nodes:31329107   Branching = 1.68
  Ply:22   Positions: 30   Avg Nodes:51579137   Branching = 1.65
  Ply:23   Positions: 30   Avg Nodes:81829576   Branching = 1.59

Average EBF at relevant plies: 1.67


The gain from doubling in time at blitz time control is comparable, slightly higher in Komodo:

Gideon: 79 ELO points from 1'+1'' to 2'+2''
Komodo: 103 ELO points from 1'+1'' to 2'+2''
One must remember that the top hardware of 1993 was a x486 at 50MHz, roughly 100 slower than a today core, so this time control is really long time control for Gideon.

Artistic impression: Gideon won all 6-7 games at loosely blitz time control against me, and it seems somewhat more tactical than modern engines. The search depth at blitz in the opening was 7-9 plies for Gideon and roughly double, 16-19 plies for Komodo on one core. Not the same plies, though. Gideon plies are a bit "thicker" ELO-wise.

Note that at 40 moves in two seconds, which works out to 50 milliseconds per move, Komodo's default overhead of 30 milliseconds is a huge factor. You could probably lower that to 10 milliseconds in Linux, but I think that most Windows GUIs need something like our default value. So to minimize the overhead factor, you might try rerunning at 40 in five seconds vs 40 in five minutes. I think Komodo would do much better.

[Laskos]

Thanks to Ed Schroder for enabling Mephisto Gideon Pro (1993) with Winboard and UCI compatibility. This top engine of 1993 is the oldest UCI engine I have. The port allows to test it in various conditions.

It seems since 1993 the main strength transition is from minutes to seconds. Mephisto Gideon Pro at 40 moves in 2 minutes ( 40/2') is fairly equal to Komodo 9.2 at 40 moves in 2 seconds (40/2'').

Code: Select all

Games Completed = 100 of 100 (Avg game length = 214.449 sec)
Settings = Gauntlet/32MB/120000ms in 40 moves/M 800000cp for 1000 moves, D 25000 moves/PGN:2moves_v2.pgn(32000)
Time = 3287 sec elapsed, 0 sec remaining
 1.  Mephisto Gideon Pro (1993)  45.0/100	38-48-14  	(L: m=48 t=0 i=0 a=0)	(D: r=8 i=3 f=2 s=1 a=0)	(tpm=2709.5 d=9.95 nps=246638359)
 2.  Komodo 9.2 64-bit        	55.0/100	48-38-14  	(L: m=38 t=0 i=0 a=0)	(D: r=8 i=3 f=2 s=1 a=0)	(tpm=39.9 d=14.01 nps=1648576)

It is a factor of 60 time control handicap, but one must keep in mind that this facotr increases to longer time controls. On CEGT rating list, used preferentially over CCRL for large ELO spans because CEGT uses uncompressed ELO scale, one can infere that on equal hardware Gideon is about 970 ELO points weaker than Komodo at 40/4'.

Some properties of Gideon, top engine of 1993, compared to Komodo 9.2, top engine of 2015:

Effective Branching Factor:

Gideon

Code: Select all

  n/s: 1.296.602  
  TotTime: 111:49m    SolTime: 111:49m
  Ply: 9   Positions: 30   Avg Nodes: 5847624   Branching = 3.84
  Ply:10   Positions: 30   Avg Nodes:20184746   Branching = 3.45
  Ply:11   Positions: 30   Avg Nodes:69769347   Branching = 3.46

Average EBF at relevant plies: 3.58

Komodo

Code: Select all

  n/s: 1.488.637  
  TotTime: 45:37m    SolTime: 45:37m
  Ply:18   Positions: 30   Avg Nodes: 6304278   Branching = 1.65
  Ply:19   Positions: 30   Avg Nodes:10764642   Branching = 1.71
  Ply:20   Positions: 30   Avg Nodes:18700347   Branching = 1.74
  Ply:21   Positions: 30   Avg Nodes:31329107   Branching = 1.68
  Ply:22   Positions: 30   Avg Nodes:51579137   Branching = 1.65
  Ply:23   Positions: 30   Avg Nodes:81829576   Branching = 1.59

Average EBF at relevant plies: 1.67


The gain from doubling in time at blitz time control is comparable, slightly higher in Komodo:

Gideon: 79 ELO points from 1'+1'' to 2'+2''
Komodo: 103 ELO points from 1'+1'' to 2'+2''
One must remember that the top hardware of 1993 was a x486 at 50MHz, roughly 100 slower than a today core, so this time control is really long time control for Gideon.

Artistic impression: Gideon won all 6-7 games at loosely blitz time control against me, and it seems somewhat more tactical than modern engines. The search depth at blitz in the opening was 7-9 plies for Gideon and roughly double, 16-19 plies for Komodo on one core. Not the same plies, though. Gideon plies are a bit "thicker" ELO-wise.

Note that at 40 moves in two seconds, which works out to 50 milliseconds per move, Komodo's default overhead of 30 milliseconds is a huge factor. You could probably lower that to 10 milliseconds in Linux, but I think that most Windows GUIs need something like our default value. So to minimize the overhead factor, you might try rerunning at 40 in five seconds vs 40 in five minutes. I think Komodo would do much better.

Yes, Komodo will improve for two reasons: overhead and because doubling at very short tc for Komodo is worth much more ELO than doubling at longer tc for Gideon. I have a result of 65:35 for Komodo at 40/4 seconds versus Gideon 40/8 minutes, a factor of 120 handicap in time. Next I will play Komodo 40/10 seconds versus Gideon 40/1 hour, my guess it will be pretty balanced, and a factor of 360 handicap in time to equal Gideon at fairly regular tournament time control.

[ernest]

Hi Kai,

I wonder what happens on test positions.

Given 10 positions Komodo solves in around 1 minute, I doubt very much that Gideon solves them in around 1 hour or so (I think that many of them Gideon will never solve...).

Progress in chess software is not only a matter of speed.

[lkaufman]

Hi Kai,

I wonder what happens on test positions.

Given 10 positions Komodo solves in around 1 minute, I doubt very much that Gideon solves them in around 1 hour or so (I think that many of them Gideon will never solve...).

Progress in chess software is not only a matter of speed.

Probably if you give Komodo one minute per move for a game and you give Gideon one hour per move for a game, Komodo will also score overwhelmingly. It's not so much the difference between problems and play, as the fact (as Kai pointed out) that Komodo will scale way better than the old engine given more time.

[Rebel]

Hi Kai,

I wonder what happens on test positions.

Given 10 positions Komodo solves in around 1 minute, I doubt very much that Gideon solves them in around 1 hour or so (I think that many of them Gideon will never solve...).

Progress in chess software is not only a matter of speed.

Probably if you give Komodo one minute per move for a game and you give Gideon one hour per move for a game, Komodo will also score overwhelmingly. It's not so much the difference between problems and play, as the fact (as Kai pointed out) that Komodo will scale way better than the old engine given more time.

True of course. See the EBF's of both. In those days the power of reductions still had to be discovered. OTOH, I could count the number of plies for a given position to find the right move. Back then, a ply was a real ply

[Laskos]

Hi Kai,

I wonder what happens on test positions.

Given 10 positions Komodo solves in around 1 minute, I doubt very much that Gideon solves them in around 1 hour or so (I think that many of them Gideon will never solve...).

Progress in chess software is not only a matter of speed.

Sure there are many differences, but I wrote that factor of 60 when comparing purely time handicap is not a stable quantity. For example, you gave 1 minute per move for Komodo. I expect Gideon to be the strength of Komodo not at 60 minutes per move, but maybe at 10,000 minutes per move. This factor increases rapidly with time control. Doubling in time for Komodo at fast controls is much larger ELO-wise than doubling for Gideon at long and very long controls, where we match them. So, it is expected that many things solved by Komodo in 1 minute are never practically solved by Gideon.

[Laskos]

Note that at 40 moves in two seconds, which works out to 50 milliseconds per move, Komodo's default overhead of 30 milliseconds is a huge factor. You could probably lower that to 10 milliseconds in Linux, but I think that most Windows GUIs need something like our default value. So to minimize the overhead factor, you might try rerunning at 40 in five seconds vs 40 in five minutes. I think Komodo would do much better.

Yes, Komodo will improve for two reasons: overhead and because doubling at very short tc for Komodo is worth much more ELO than doubling at longer tc for Gideon. I have a result of 65:35 for Komodo at 40/4 seconds versus Gideon 40/8 minutes, a factor of 120 handicap in time. Next I will play Komodo 40/10 seconds versus Gideon 40/1 hour, my guess it will be pretty balanced, and a factor of 360 handicap in time to equal Gideon at fairly regular tournament time control.

I seem to have overshoot a bit. Komodo at 40 moves/10 seconds beat 31.5-8.5 Gideon at 40 moves/3600 seconds. Probably 7-8 seconds were enough for Komodo to be equal, and it seems a factor of 500 would be a fair handicap at this tournament time control of Gideon.

[fern]

Kai, as looks like you are really an experts in test and technicalities, let me ask you how much a program that is 1880 Elo with a 486-33 MHz becomes when running a 3 giga ram and 3 giga processor....

Fern

[Laskos]

Kai, as looks like you are really an experts in test and technicalities, let me ask you how much a program that is 1880 Elo with a 486-33 MHz becomes when running a 3 giga ram and 3 giga processor....

Fern

Close to 200 factor speed improvement in hardware, thus almost 8 doublings. 60 points per doubling, so ~480 ELO points improvement, or 2360. That is in computer ELO terms. If 1880 ELO was against humans, then the rule of thumb is human ELO differences ~70% of computer ELO differences, therefore the improvement in human ELO would be 0.7*480~330 ELO points, therefore now the former 1880 human ELO software is 1880+330=2210 human ELO (on a new core). That probably can be tested against some anchor engines related to human play, from SSDF for example.