Rybka 4 Q6600 - Houdini 1.03a Q6600 35.5-24.5

[Jeroen]

Match played under the following conditions:

Hardware = Q6600
60 games, using Noomen Testsuite 2008
Both play 30 positions, alternating white and black
Both use 4 cores
Tempo = game in 60 mins
Ponder = off

Result: Rybka 4 - Houdini 1.03a 35.5 - 24.5 (+13 -2 =45) or +64 elo

My own observations:

1. Only 2 wins in 60 games for Houdini, profiting from (severe) time trouble by Rybka (that spoilt 2 equal positions)
2. Rybka won 13 times, clearly seeing more and outplaying Houdini in those games
3. Houdini is unable to outplay Rybka 4 when the latter avoids time trouble, while the opposite happened many times
4. In this match using this playing tempo and the 30 match positions, Rybka 4 shows itself to be better than Houdini

Some examples:

[Event "Blitz:60'"]
[Site "Q6600 2.4 GHz"]
[Date "2010.10.09"]
[White "Deep Rybka 4 x64"]
[Black "Houdini 1.03a x64 4_CPU"]
[Result "1-0"]
[ECO "B81"]
[PlyCount "81"]

1. e4 c5 2. Nf3 e6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 d6 6. g4 h6 7. h4 Nc6 8. Rg1 h5 9. gxh5 Nxh5 10. Bg5 Nf6 11. Qd2 Qb6 12. Nb3 a6 13. O-O-O Bd7 14. Kb1 Ne5 15. Nd4 Nc6 16. Nxc6 Bxc6 17. Bd3 Qc7 18. Qe3 Qa5 19. a3 Be7 20. Bc4 Rc8 21. Bb3 Qc5 22. Qe2 Bb5 23. Qe1 Bc4 24. Bxc4 Qxc4 25. f4 b5 26. e5 dxe5 27. fxe5 Nd5 28. Nxd5 Bxg5 29. Rxg5 Qxc2+ 30. Ka1 exd5 31. e6 O-O 32. Qg1 Qh7 33. e7 Rfe8 34. Qg4 f5 35. Qe2 Rc7 36. Qe6+ Kh8 37. Rxd5 Qh6 38. Qxf5 Kg8 39. Rd7 Rc1+ 40. Ka2 Re1 41. Rg6 1-0

[Event "Blitz:60'"]
[Site "Q6600 2.4 GHz"]
[Date "2010.10.09"]
[White "Deep Rybka 4 x64"]
[Black "Houdini 1.03a x64 4_CPU"]
[Result "1-0"]
[ECO "B67"]
[PlyCount "119"]

1. e4 c5 2. Nf3 Nc6 3. d4 cxd4 4. Nxd4 Nf6 5. Nc3 d6 6. Bg5 e6 7. Qd2 a6 8. O-O-O Bd7 9. f4 b5 10. Bxf6 gxf6 11. Kb1 Qb6 12. Nce2 Rc8 13. f5 h5 14. Nxc6 Qxc6 15. Nd4 Qxe4 16. fxe6 fxe6 17. Bd3 Qg4 18. Rhe1 Kf7 19. Qf2 h4 20. Be4 Be7 21. Rd3 Qg5 22. Qe2 e5 23. Nf3 Qh5 24. Ra3 Rc5 25. Rxa6 Rhc8 26. Bd3 R8c7 27. Rf1 Kg7 28. g4 hxg3 29. Rg1 f5 30. Rxg3+ Kf6 31. Rg5 Qh8 32. Qf2 Kf7 33. Rxf5+ Bxf5 34. Bxf5 Qf6 35. Be4 Rc8 36. Qg1 Kf8 37. a3 Qf7 38. Qg4 Qf4 39. Qg6 Qf6 40. Qh5 Qf7 41. Qh3 R5c7 42. Rb6 Rd8 43. Rxb5 Bf6 44. Nh4 Bxh4 45. Qxh4 Rdc8 46. Rb3 Qe6 47. Qh5 Ke7 48. Bf5 Qf7 49. Qxf7+ Kxf7 50. Bxc8 Rxc8 51. a4 Rc5 52. Ra3 e4 53. b4 Rf5 54. a5 Rf3 55. c3 e3 56. Kc2 Rh3 57. a6 Rh8 58. a7 Ra8 59. b5
Ke6 60. b6 1-0

[Event "Blitz:60'"]
[Site "Q6600 2.4 GHz"]
[Date "2010.10.11"]
[White "Houdini 1.03a x64 4_CPU"]
[Black "Deep Rybka 4 x64"]
[Result "0-1"]
[ECO "B12"]
[PlyCount "132"]

1. e4 c6 2. d4 d5 3. e5 Bf5 4. Nc3 e6 5. g4 Bg6 6. Nge2 c5 7. Be3 Nc6 8. dxc5 h5 9. Nd4 a6 10. Nxc6 bxc6 11. Bd3 Ne7 12. g5 Bxd3 13. Qxd3 Nf5 14. O-O-O Qa5 15. g6 fxg6 16. Rhg1 Kf7 17. Qf1 Bxc5 18. Qg2 Ne7 19. Kb1 Bxe3 20. Qf3+ Ke8 21. Qxe3 Rb8 22. b3 Rf8 23. Ne2 c5 24. Rg2 Qb4 25. Qd3 a5 26. Rxg6 c4 27. Qg3 Nxg6 28. Qxg6+ Kd7 29. f4 Qe7 30. Qxh5 a4 31. Qg4 axb3 32. axb3 cxb3 33. Rxd5+ Ke8 34. Rd6 bxc2+ 35. Kxc2 Kf7 36. Nd4 Rb6 37. Rxe6 Rxe6 38. Qf5+ Rf6 39. exf6 Qxf6 40. Qd5+ Kg6 41. f5+ Kh7 42. Qe4 Rd8 43. Kd3 Qa6+ 44. Kd2 Kg8 45. Qe5 Qd6 46. Qxd6 Rxd6 47. Ke3 Kf7 48. Nb5 Rd5 49. Nd4 Kf6 50. Ke4 Ra5 51. Kf3 Ra2 52. Kg3 Ra4 53. Nf3 Kxf5 54. Nh4+ Kf6 55. h3 g5 56. Ng2 Kg6 57. Ne1 Kf5 58. Nc2 Ke4 59. Ne1 Ra3+ 60. Kg4 Ra2 61. Kg3 Kf5 62. Nf3 Ra3 63. Kg2 Kf4 64. Nd4 Ra2+ 65. Kf1 Kg3 66. Ne6 Ra5 0-1

[Event "Blitz:60'"]
[Site "Q6600 2.4 GHz"]
[Date "2010.10.15"]
[White "Deep Rybka 4 x64"]
[Black "Houdini 1.03a x64 4_CPU"]
[Result "1-0"]
[ECO "E35"]
[PlyCount "97"]

1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. Qc2 d5 5. cxd5 exd5 6. Bg5 h6 7. Bxf6 Qxf6 8. a3 Bxc3+ 9. Qxc3 O-O 10. e3 c6 11. Ne2 Re8 12. Ng3 g6 13. Bd3 h5 14. Qc2 a5 15. Ne2 Bg4 16. O-O Bxe2 17. Bxe2 Nd7 18. b4 b5 19. bxa5 Rxa5 20. Rfc1 Rc8 21. a4 b4 22. Bd1 Qd6 23. Rab1 Rc7 24. Qd2 Rb7 25. Rb2 Nb8 26. Bb3 Rc7 27. g3 Rb7 28. Rbc2 Kg7 29. Kg2 Ra6 30. Qd3 Re7 31. Qd1 Ra5 32. Qd2 Rb7 33. Qe1 Qe6 34. f3 Qd6 35. Rd2 f5 36. e4 fxe4 37. fxe4 dxe4 38. Re2 Re7 39. Rxe4 Raa7 40. Rc5 Rxe4 41. Qxe4 Kf8 42. Re5 Re7 43. Bc4 Ke8 44. a5 Kd8 45. a6 Nxa6 46. Bxa6 Rxe5 47. dxe5 Qd7 48. Bc4 g5 49. Qe3 1-0

[Event "Blitz:60'"]
[Site "Q6600 2.4 GHz"]
[Date "2010.10.15"]
[White "Houdini 1.03a x64 4_CPU"]
[Black "Deep Rybka 4 x64"]
[Result "0-1"]
[ECO "E35"]
[PlyCount "104"]

1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. Qc2 d5 5. cxd5 exd5 6. Bg5 h6 7. Bxf6 Qxf6 8. a3 Bxc3+ 9. Qxc3 O-O 10. e3 c6 11. Rc1 Bf5 12. Qb4 Nd7 13. Qxb7 Rab8 14. Qxc6 Rxb2 15. Ne2 Rfb8 16. Qxf6 Nxf6 17. Nc3 Ne4 18. Nd1 Rb1 19. Rxb1 Rxb1 20. Be2 Ra1 21. a4 Nc3 22. a5 Be4 23. f3 Nxe2 24. Kxe2 Ra2+ 25. Ke1 Bc2 26. Rf1 Bxd1 27. Kxd1 Rxg2 28. h3 Kf8 29. e4 Ra2 30. Re1 Ke7 31. exd5+ Kd7 32. Re5 Rxa5 33. Ke2 Ra1 34. Kd3 a5 35. Kc4 a4 36. Rf5 f6 37. d6 Kxd6 38. Ra5 g5 39. Ra6+ Ke7 40. Kd5 Kf7 41. Rc6 a3 42. Rc3 h5 43. Rc2 Rb1 44. Ra2 Rb3 45. Rf2 Rb2 46. Rf1 a2 47. Ra1 f5 48. Kc4 g4 49. Kc3 Rf2 50. hxg4 hxg4 51. Kd3 g3 52. Ke3 f4+ 0-1

As I can see from its main lines, eval and behaviour that Houdini is clearly a Rybka derivative, I am going to delete it from my hard disk. I am not interested in derivatives .

[Martin Thoresen]

Thanks for your report, Jeroen.

But why are the games played at a sudden death time control?

[gerold]

Why don't the Rybka team fix the TC problem. At fast TC Houdini has
proven over and over to be stronger.
On one of the rating list at longer TC with 1000+ games the two are
within 5elo of each other.
Best,
Gerold.

P.S. I still use Rybka even if it is a proven clone of Fruit.

[Günther Höhne]

Thanks Jeroen, nice test! And my approval in the assessment of Houdini.

Regards
Günther

[mwyoung]

I am not buying what you are trying to sell. Houdini has be out for a long time now. Many of us has been able to play 100's of games with Houdini. It is clear Houdini is faster then Rybka, and has a better endgame evaluation.

Rybka 4 +64 elo stronger then Houdini.....

[IWB]

Rybka 4 +64 elo stronger then Houdini.....

Why not, I would not draw any conclusion out of 60 games, but that is as good as playing 100 blitz games and conclude that the Littos are 50, 80 or 100 Elos better than Rybka 3/4.

The only conclusion which is really sure is, that you should not make a statement about a playing strength of an enigne because of a comparision of TWO enignes. Regardless what you make out of that it is nonsens!

Bye
Ingo

[beram]

Match played under the following conditions:

Code: Select all

Hardware = Q6600
60 games, using Noomen Testsuite 2008
Both play 30 positions, alternating white and black
Both use 4 cores
Tempo = game in 60 mins
Ponder = off

Result: Rybka 4 - Houdini 1.03a  35.5 - 24.5  (+13 -2 =45) or +64 elo

My own observations:

1. Only 2 wins in 60 games for Houdini, profiting from (severe) time trouble by Rybka (that spoilt 2 equal positions)
2. Rybka won 13 times, clearly seeing more and outplaying Houdini in those games
3. Houdini is unable to outplay Rybka 4 when the latter avoids time trouble, while the opposite happened many times
4. In this match using this playing tempo and the 30 match positions, Rybka 4 shows itself to be better than Houdini

As I watched the other Very LongTC games between Houdini 1.02 and R4 in the match of Martin Thoresen. I agree with the above observations point 2. and 4.

Code: Select all

As I can see from its main lines, eval and behaviour that Houdini is clearly a Rybka derivative, I am going to delete it from my hard disk. I am not interested in derivatives ;-).[/quote]

For this there is no proof in the eval's, haven't you read BB's report than?
http://www.open-chess.org/download/file.php?id=13
Most programmers agree on that report stating that there is no proof for derivative, e.g. to quote Chris Wittington for this:

..."1. there's no re-compilation of disassembled code here, too much stuff is different and it looks like the ideas in IPPOLIT are implemented such that the programmer absolutely has to fully understand what he is doing. the many listed differences require far too much skill to leave to a non-understanding re-compilation. there doesn't even seem to be evidence of fragments of re-compiled or stolen material, the closest I read was in piece tables but they're not that close, have substantial differences anyway, and the possible 'close' matches are explainable and probably repeatable in many programs piece tables.."

Kind regards Bram

[Uri Blass]

I think that it is also possible that rybka4 is simply better in the noomen positions and with a different set of positions Houdini can score better.

I saw in another thread that Rybka4 played very poor in the king indian defence and people claimed that Rybka3 human is better.

I think that it may be interesting to decide about target(A is better than B and later build positions that a match based on them prove it for different time controls not only based on direct match but also based on games against different opponents)

We can learn from it about the relative weaknesses of different chess engines.

I would like to see 30 positions that we are going to call
1)Rybkafavourite_testsuite.pgn.
2)Stockfishfavourite_testsuite.pgn
3)Naumfavourite_testsuite.pgn
4)Houdinifavourite_testsuite.pgn

The idea is that you search for positions when Rybka4 is going to be number 1 by a big margin in test 1 and stockfish or Naum or houdini are going to be number 1 in test 2,test 3 and test 4.

[peter]

I would like to see 30 positions that we are going to call
1)Rybkafavourite_testsuite.pgn.
2)Stockfishfavourite_testsuite.pgn
3)Naumfavourite_testsuite.pgn
4)Houdinifavourite_testsuite.pgn

The idea is that you search for positions when Rybka4 is going to be number 1 by a big margin in test 1 and stockfish or Naum or houdini are going to be number 1 in test 2,test 3 and test 4.

Hi!
I fully agree with you, whether you were serious about it or not.

I'd welcome testsuites like the ones you gave, if there were Shredderfavourite, Critterfavourite and Fritzfavourite ones too and the community able to interpret them all right because of the one and only meaningful question together with all these testsuites:
Why were positions of a certain kind better for engines like for example fruit and others for example better for fritz.
The question you could answer in this way (and in this way only), which positions to deal with which engine best, is the only one of interest for the chess player, beeing such I'd like to ask from a software engineer programming chess playing software too, wouldn't I?
Taking it seriously again: Jeroen Noomen's testset 2008 for me is a good one, just because the positions are a little bit different from the one and only basic position in chess. There are several that don't seem fully balanced to engines neither to human players, some are connected with certain kind of middlegame play and the engine has to deal with that too, not only with 5 moves only books and well selected positions to start games with to be as sure as possible, there wasn't any kind of advantage neither to the one nor to the other side.
Such well selected postions used for most rating lists of common interest (I don't say anything against this interest but it simply isn't the only one by far) are for nowadays-engines much to near to basic position itself.
No matter of the numbers of "doublets" you produce by matching many engines just without any kind of books at all, this still is the one and only other way of testing engines against each others. If you want to rate by their ability of dealing with the basic postion and positions very near to it in character and number of moves out of book, you have to match from basic positon only, which I fo myself are rarely interested in at all.
If you want to test abilities besides this very special one, you have to select other positions as for meanings of chess, not of accident.
Accident cannot answer your questions of chess, the illusion of double-blind studies in chess ratings are simple self-deception as for my personal point of view