Houdini 3 running for the IPON

[lkaufman]

OK, so 40/20 CEGT (no PB) is too close to IPON levels to provide scaling info, so the CCRL 40/40 will be the first substantial test that sheds light on scaling.

that's correct, our 40/20PB=off (which effectively is 40/10 or less) is rather close to IPON. But our 40/20PB=on whithout adapting, so it is 40/20 effectively, is ~ 5 times slower than IPON. Mean game length is around 80 minutes, IPON 16 minutes. Hope to have first results next week.

Our 40/120 results will take some time as on one quad a 50-game match lasts 3-4 days when Shredder Classic interface is started four times (12 to 16 games per day). So it will take at least one month to have 400-500 games.

I note that your 40/20 PBON games average 80 minutes. This means they average at least 80 moves. Most games are effectively decided in well under 80 moves, but the draws have to be played out until the bitter end. If we look at time spent on the first 40 moves, where the majority of games are probably effectively decided, your figure would be 40 minutes, and IPON probably something like 13 minutes. So the "real" ratio would be around 3 to 1 rather than 5. Anyway, this shows how much more efficient increment testing is compared to repeating time control. If you switched to increment testing with the IPON 100 to 1 ratio, you could probably play 50% more games in the same time with comparable average quality. No one plays human tournaments at 40/40 minutes or 40/20 or 40/5, and even 40/100' plus increment plus 30' for the rest of the game is non-repeating and is only used to give the players a bathroom break at move 40. I think it's time for CEGT and CCRL to seriously consider switching to increment play. Repeating time control play is obsolete for both humans and computers.

[Modern Times]

I think it's time for CEGT and CCRL to seriously consider switching to increment play. Repeating time control play is obsolete for both humans and computers.

When we started our list there were engines that didn't support this sort of time control. Repeating time control was the most compatible over a wide range of engines. That was why it was chosen mainly, although there were other reasons. I don't know if that is the case today.

Many CCRL members actually prefer fischer time control, but it means starting all over again from scratch. We don't have resource for an extra list, so it would require agreement to cease an existing one and start afresh. I think that is unlikely.

[IWB]

I think it's time for CEGT and CCRL to seriously consider switching to increment play. Repeating time control play is obsolete for both humans and computers.

Ahhh ... !!! Repeating time controls are played because analog clocks could not add increments. Nowadays it a a crusted tradition which is played because people in chess clubs play it because when they entered the club it was played like this ... it as a bad habit which is hard to wipe out!

Thx for your statement
Ingo

[Adam Hair]

Hi Uri,

....
Initial CEGT results suggest 70 elo improvement relative to houdini2
and they use 40/20 and 4 cpu.

"40/20" is a theoretical value, because at our 40/20-list the real time control is adapted to hardware power by a benchmark. Real 40/20 would only be played on an AMD-X2-4200 @ 2,4 GHZ (our hardware reference) but the hardware used by Werner and Johan is very much faster. IIRC they play with 40/10 (Werner) and 40/8 (?!, Johan), but I don't know exactly.

This is slower than IPON but still far away from "real slow" time controls.

These will be played at:

- CEGT 40/20 with PB = on (starts as soon as Critter 1.6 test is finished, approx. beginning/middle next week)
- CEGT 40/120 (starts this evening)
and CCRL 40/40

But this will take some time.

and don't forget: only 140 (!!) games played so far! Next weekend we will have relatively reliable ratings at least for 40/3 and hopefully for 40/"20" too.

OK, so 40/20 CEGT (no PB) is too close to IPON levels to provide scaling info, so the CCRL 40/40 will be the first substantial test that sheds light on scaling.

40/40 testing with 1 CPU and 4 CPU has started. I will start sharing my 40/40 1 CPU results this evening. My actual time control is 40 moves in 25 minutes and 50 seconds, repeating. The testing is occurring on a QX6700, so no POPCNT.

[lkaufman]

I think it's time for CEGT and CCRL to seriously consider switching to increment play. Repeating time control play is obsolete for both humans and computers.

When we started our list there were engines that didn't support this sort of time control. Repeating time control was the most compatible over a wide range of engines. That was why it was chosen mainly, although there were other reasons. I don't know if that is the case today.

Many CCRL members actually prefer fischer time control, but it means starting all over again from scratch. We don't have resource for an extra list, so it would require agreement to cease an existing one and start afresh. I think that is unlikely.

I think you only need to agree to start using increment time controls as of a certain date. For example, the blitz list could use 5' +3" (adapted). All the existing data would be kept. You already combine AMD with Intel, SSE4 with nonSSE4, etc., and you already made a major change to your blitz criteria when you adopted a minimum of 40/3 even when adaption would call for a much faster time limit. Switching to increment might favor some engines over others by a few elo compared to repeating time controls, but probably no more so than the decision to require a 40/3 absolute minimum. The 40/3 min. was presumably done to raise the quality of the games, and this would apply equally to a switch to increment play. After a few months, everyone would agree that the lists were more relevant than they would have been without the change.

[Modern Times]

you already made a major change to your blitz criteria when you adopted a minimum of 40/3 even when adaption would call for a much faster time limit.

No we did not do that. I don't know where you got that idea from. I think CEGT mentioned it.

[lkaufman]

you already made a major change to your blitz criteria when you adopted a minimum of 40/3 even when adaption would call for a much faster time limit.

No we did not do that. I don't know where you got that idea from. I think CEGT mentioned it.

Oh, I thought you both did that. So you run your 40/4 games at 40/2 or even faster if that is what the adjustment rules call for?

[Modern Times]

We don't have any machines that are faster than 40/2.

But of course you don't always need to round. ChessGUI and Winboard for example let you specify minutes and seconds, which is exactly what I do. e.g. 40 moves in 2mins 15secs on this machine.

[IWB]

Last test for now:

I tested H3-tactical vs the for strongest opponents:

Code: Select all

Houdini 3 STD             : 600 (+263,=264,-73), 65.8 %

Stockfish 2.2.2 JA            : 150 (+ 75,= 59,- 16), 69.7 %
Deep Rybka 4.1                : 150 (+ 70,= 63,- 17), 67.7 %
Critter 1.4a                  : 150 (+ 57,= 73,- 20), 62.3 %
Komodo 5                      : 150 (+ 61,= 69,- 20), 63.7 %


Houdini 3 STD-T           : 600 (+220,=275,-105), 59.6 %

Stockfish 2.2.2 JA            : 150 (+ 50,= 69,- 31), 56.3 %
Deep Rybka 4.1                : 150 (+ 68,= 59,- 23), 65.0 %
Critter 1.4a                  : 150 (+ 56,= 75,- 19), 62.3 %
Komodo 5                      : 150 (+ 46,= 72,- 32), 54.7 %



Houdini 2.0 STD           : 5850 (+198,=267,-135) 55.3 %

Stockfish 2.2.2 JA            : 150 (+ 54,= 63,- 33), 57.0 %
Deep Rybka 4.1                : 150 (+ 54,= 67,- 29), 58.3 %
Critter 1.4a                  : 150 (+ 42,= 74,- 34), 52.7 %
Komodo 5                      : 150 (+ 48,= 63,- 39), 53.0 %

6.2% difference, but better than H2.

Bye
Ingo

[Graham Banks]

I think it's time for CEGT and CCRL to seriously consider switching to increment play. Repeating time control play is obsolete for both humans and computers.

If you're only interested in compiling a rating list with the top or most recent engines, then incremental time controls are fine.
If you want to test a wide diversity of engines, including older engines and weaker engines, using incremental time controls would mean excluding too many because they don't support that form of time control.

Repeating time controls may be obsolete for human play, but no such argument can be applied to engine play as the same factors and reasoning don't apply.
It's entirely a matter of taste.
In my case I prefer repeating time controls because it gives a consistency in use of time and strength of play through all stages of a game.

Nobody is right here and nobody is wrong.