First, I adjusted time controls for each engine, so that they become pretty equal in strength. The time controls varied, from 60ms/move for Houdini to 600ms/move for Junior. The strength adjusted results using Frank Quisinsky 5,000+ opening 8 mover positions are:
Code: Select all
Program Score % Elo + - Draws
1 Junior 13 : 2428.5/4620 52.6 3015 9 9 27.5 %
2 Houdini 3 : 2407.5/4630 52.0 3012 8 8 36.1 %
3 Hiarcs 14 : 2386.5/4629 51.6 3009 9 9 27.4 %
4 Critter 1.6 : 2339.0/4590 51.0 3006 8 8 33.3 %
5 Komodo 5 : 2348.5/4622 50.8 3005 8 8 31.4 %
6 Rybka 4.1 : 2152.0/4523 47.6 2986 8 8 33.1 %
7 Stockfish 2.3.1 : 2048.0/4606 44.5 2967 8 8 31.8 %
420 games for each opening, 120 games per engine, errors 2SD are 50-55 Elo points.
Grunfeld
[D]rnbqkb1r/ppp1pp1p/5np1/3p4/2PP4/2N5/PP2PPPP/R1BQKBNR w KQkq - 0 4[/D]
Code: Select all
Program Score % Elo Performance
1 Stockfish 2.3.1 : 79.5/120 66.2 3100 +133
2 Critter 1.6 : 71.5/120 59.6 3057 +51
3 Hiarcs 14 : 61.0/119 51.3 3008 -1
4 Houdini 3 : 57.5/120 47.9 2987 -25
5 Komodo 5 : 56.5/120 47.1 2982 -23
6 Junior 13 : 49.5/120 41.2 2947 -68
7 Rybka 4.1 : 43.5/119 36.6 2918 -68
Dutch - Staunton-Gambit
[D]rnbqkbnr/ppppp1pp/8/5p2/3PP3/8/PPP2PPP/RNBQKBNR b KQkq - 0 2[/D]
Code: Select all
Program Score % Elo Performance
1 Critter 1.6 : 70.5/120 58.8 3053 +47
2 Komodo 5 : 69.5/120 57.9 3048 +43
3 Hiarcs 14 : 61.5/120 51.2 3007 -2
4 Stockfish 2.3.1 : 60.5/119 50.8 3005 +38
5 Houdini 3 : 58.0/119 48.7 2992 -20
6 Rybka 4.1 : 51.5/120 42.9 2958 -28
7 Junior 13 : 47.5/120 39.6 2937 -78
Nimzo-Indian
[D]rnbqk2r/pppp1ppp/4pn2/8/1bPP4/2N5/PP2PPPP/R1BQKBNR w KQkq - 0 4[/D]
Code: Select all
Program Score % Elo Performance
1 Critter 1.6 : 69.0/119 58.0 3048 +42
2 Komodo 5 : 65.0/119 54.6 3028 +23
3 Stockfish 2.3.1 : 64.5/120 53.8 3022 +55
4 Houdini 3 : 62.0/119 52.1 3013 +1
5 Hiarcs 14 : 60.0/120 50.0 3000 -9
6 Rybka 4.1 : 48.0/117 41.0 2946 -40
7 Junior 13 : 48.5/120 40.4 2942 -73
Queensgambit
[D]rnbqkbnr/ppp1pppp/8/8/2pP4/8/PP2PPPP/RNBQKBNR w KQkq - 0 3[/D]
Code: Select all
Program Score % Elo Performance
1 Critter 1.6 : 68.0/119 57.1 3043 +37
2 Stockfish 2.3.1 : 65.0/119 54.6 3028 +61
3 Houdini 3 : 60.5/120 50.4 3003 -9
4 Komodo 5 : 60.0/120 50.0 3000 -5
5 Hiarcs 14 : 59.0/120 49.2 2995 -14
6 Junior 13 : 54.5/120 45.4 2973 -42
7 Rybka 4.1 : 50.0/116 43.1 2958 -28
Sicilian
[D]rnbqkb1r/pp2pppp/3p1n2/8/3NP3/2N5/PPP2PPP/R1BQKB1R b KQkq - 0 5[/D]
Code: Select all
Program Score % Elo Performance
1 Houdini 3 : 69.5/119 58.4 3051 +39
2 Critter 1.6 : 64.5/119 54.2 3025 +19
3 Komodo 5 : 61.5/119 51.7 3010 +5
4 Hiarcs 14 : 62.0/120 51.7 3010 +1
5 Stockfish 2.3.1 : 56.0/120 46.7 2980 +13
6 Junior 13 : 55.5/119 46.6 2980 -35
7 Rybka 4.1 : 47.0/116 40.5 2942 -44
English
[D]r1bqk1nr/pp1p1pbp/2n3p1/2p1p3/2P1P3/2N3P1/PP1P1PBP/R1BQK1NR w KQkq - 0 6[/D]
Code: Select all
Program Score % Elo Performance
1 Critter 1.6 : 68.5/120 57.1 3043 +37
2 Komodo 5 : 68.0/120 56.7 3040 +35
3 Houdini 3 : 66.0/119 55.5 3033 +21
4 Stockfish 2.3.1 : 63.5/120 52.9 3018 +51
5 Hiarcs 14 : 57.5/120 47.9 2988 -21
6 Junior 13 : 50.5/120 42.1 2953 -62
7 Rybka 4.1 : 45.0/119 37.8 2926 -60
Caro-Kann
[D]rn1qkbnr/pp2pppp/2p5/3pPb2/3P4/8/PPP2PPP/RNBQKBNR w KQkq - 0 4[/D]
Code: Select all
Program Score % Elo Performance
1 Critter 1.6 : 72.0/119 60.5 3063 +57
2 Komodo 5 : 67.5/120 56.2 3037 +32
3 Houdini 3 : 58.5/119 49.2 2995 -17
4 Stockfish 2.3.1 : 56.5/118 47.9 2988 +21
5 Hiarcs 14 : 57.0/120 47.5 2985 -24
6 Junior 13 : 55.0/120 45.8 2975 -40
7 Rybka 4.1 : 49.5/116 42.7 2956 -30
Kings-Indian
[D]rnbq1rk1/ppp1ppbp/3p1np1/8/2PPP3/2N2N2/PP2BPPP/R1BQK2R b KQ - 0 6[/D]
Code: Select all
Program Score % Elo Performance
1 Critter 1.6 : 75.5/120 62.9 3079 +73
2 Houdini 3 : 68.0/120 56.7 3040 +28
3 Stockfish 2.3.1 : 67.5/120 56.2 3037 +70
4 Komodo 5 : 61.5/119 51.7 3011 +6
5 Junior 13 : 55.5/120 46.2 2978 -37
6 Hiarcs 14 : 49.0/120 40.8 2945 -64
7 Rybka 4.1 : 42.0/119 35.3 2910 -76
French Defence
[D]rnbqk1nr/ppp2ppp/4p3/3pP3/1b1P4/2N5/PPP2PPP/R1BQKBNR b KQ - 0 4[/D]
Code: Select all
Program Score % Elo Performance
1 Houdini 3 : 72.5/120 60.4 3063 +51
2 Critter 1.6 : 68.5/119 57.6 3046 +40
3 Stockfish 2.3.1 : 61.5/120 51.2 3007 +40
4 Junior 13 : 59.5/120 49.6 2997 -22
5 Komodo 5 : 57.5/120 47.9 2988 -17
6 Hiarcs 14 : 53.5/120 44.6 2968 -41
7 Rybka 4.1 : 46.0/119 38.7 2931 -55
Spanish
[D]r1bqk2r/1pppbppp/p1n2n2/4p3/B3P3/5N2/PPPP1PPP/RNBQ1RK1 w kq - 0 6[/D]
Code: Select all
Program Score % Elo
1 Critter 1.6 : 77.0/120 64.2 3087 +81
2 Komodo 5 : 64.5/120 53.8 3022 +17
3 Stockfish 2.3.1 : 62.5/118 53.0 3019 +52
4 Hiarcs 14 : 61.5/119 51.7 3011 +2
5 Houdini 3 : 59.5/119 50.0 3001 -11
6 Junior 13 : 52.0/120 43.3 2960 -55
7 Rybka 4.1 : 39.0/116 33.6 2899 -87
Engines outperform and underperform significantly (with more than 50 Elo points or 2 standard deviations) as follows:
Houdini 3:
Outperforms: French Defence
Underperforms: None
Komodo 5:
Outperforms: None
Underperforms: None
Critter 1.6:
Outperforms: Grunfeld, Caro-Kann, Kings-Indian, Spanish
Underperforms: None
Stockfish 2.3.1:
Outperforms: Grunfeld, Nimzo-Indian, Qeensgambit, English, Kings-Indian, Spanish
Underperforms: None
Rybka 4.1:
Outperforms: None
Underperforms: Grunfeld, English, Kings-Indian, French Defence, Spanish
Hiarcs 14:
Outperforms: None
Underperforms: Kings-Indian
Junior 13:
Outperforms: None
Underperforms: Grunfeld, Dutch - Staunton-Gambit, Nimzo-Indian, English, Spanish
Komodo, Houdini, Hiarcs have no preferences, Stockfish and Critter play better prefered openings, Rybka and Junior play badly without books.
Kai