ChessUSA.com TalkChess.com
Hosted by Your Move Chess & Games
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Engines performance for selected openings
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: General Topics Flat
View previous topic :: View next topic  
Author Message
Kai Laskos



Joined: 26 Jul 2006
Posts: 6782

PostPost subject: Engines performance for selected openings    Posted: Sun Dec 23, 2012 5:00 pm Reply to topic Reply with quote

As an offshoot of the thread http://talkchess.com/forum/viewtopic.php?t=46398&start=70 I am posting here some results of 7 engines (Houdini 3, Komodo 5, Critter 1.6, Stockfish 2.3.1, Rybka 4.1, Hiarcs 14, Junior 13) for various, common opening positions 2-6 moves deep.

First, I adjusted time controls for each engine, so that they become pretty equal in strength. The time controls varied, from 60ms/move for Houdini to 600ms/move for Junior. The strength adjusted results using Frank Quisinsky 5,000+ opening 8 mover positions are:

Code:
    Program                            Score     %       Elo    +   -    Draws

  1 Junior 13                      : 2428.5/4620  52.6   3015    9   9   27.5 %
  2 Houdini 3                      : 2407.5/4630  52.0   3012    8   8   36.1 %
  3 Hiarcs 14                      : 2386.5/4629  51.6   3009    9   9   27.4 %
  4 Critter 1.6                    : 2339.0/4590  51.0   3006    8   8   33.3 %
  5 Komodo 5                       : 2348.5/4622  50.8   3005    8   8   31.4 %
  6 Rybka 4.1                      : 2152.0/4523  47.6   2986    8   8   33.1 %
  7 Stockfish 2.3.1                : 2048.0/4606  44.5   2967    8   8   31.8 %


I then took 10 common openings from the CSS rating list of Klaus Wlotzka and using these 7 engines, I compared their scores with that obtained previously, when I used 5,000+ opening 8 mover positions. We can see if engines overperform or underperform in these short (2-6 moves) openings.

420 games for each opening, 120 games per engine, errors 2SD are 50-55 Elo points.


Grunfeld
[D]rnbqkb1r/ppp1pp1p/5np1/3p4/2PP4/2N5/PP2PPPP/R1BQKBNR w KQkq - 0 4[/D]
BR BN BB BQ BK BB    BR
BP BP BP    BP BP :: BP
   ::    ::    BN BP ::
::    :: BP ::    ::    
   :: WP WP    ::    ::
::    WN    ::    ::    
WP WP    :: WP WP WP WP
WR    WB WQ WK WB WN WR


Code:
    Program                            Score     %     Elo   Performance 

  1 Stockfish 2.3.1                :  79.5/120  66.2   3100    +133 
  2 Critter 1.6                    :  71.5/120  59.6   3057     +51
  3 Hiarcs 14                      :  61.0/119  51.3   3008      -1
  4 Houdini 3                      :  57.5/120  47.9   2987     -25
  5 Komodo 5                       :  56.5/120  47.1   2982     -23
  6 Junior 13                      :  49.5/120  41.2   2947     -68
  7 Rybka 4.1                      :  43.5/119  36.6   2918     -68




Dutch - Staunton-Gambit
[D]rnbqkbnr/ppppp1pp/8/5p2/3PP3/8/PPP2PPP/RNBQKBNR b KQkq - 0 2[/D]
BR BN BB BQ BK BB BN BR
BP BP BP BP BP    BP BP
   ::    ::    ::    ::
::    ::    :: BP ::    
   ::    WP WP ::    ::
::    ::    ::    ::    
WP WP WP ::    WP WP WP
WR WN WB WQ WK WB WN WR


Code:
    Program                            Score     %     Elo   Performance

  1 Critter 1.6                    :  70.5/120  58.8   3053     +47 
  2 Komodo 5                       :  69.5/120  57.9   3048     +43
  3 Hiarcs 14                      :  61.5/120  51.2   3007      -2
  4 Stockfish 2.3.1                :  60.5/119  50.8   3005     +38
  5 Houdini 3                      :  58.0/119  48.7   2992     -20
  6 Rybka 4.1                      :  51.5/120  42.9   2958     -28
  7 Junior 13                      :  47.5/120  39.6   2937     -78




Nimzo-Indian
[D]rnbqk2r/pppp1ppp/4pn2/8/1bPP4/2N5/PP2PPPP/R1BQKBNR w KQkq - 0 4[/D]
BR BN BB BQ BK ::    BR
BP BP BP BP :: BP BP BP
   ::    :: BP BN    ::
::    ::    ::    ::    
   BB WP WP    ::    ::
::    WN    ::    ::    
WP WP    :: WP WP WP WP
WR    WB WQ WK WB WN WR


Code:
    Program                            Score     %     Elo   Performance

  1 Critter 1.6                    :  69.0/119  58.0   3048     +42
  2 Komodo 5                       :  65.0/119  54.6   3028     +23
  3 Stockfish 2.3.1                :  64.5/120  53.8   3022     +55
  4 Houdini 3                      :  62.0/119  52.1   3013      +1
  5 Hiarcs 14                      :  60.0/120  50.0   3000      -9
  6 Rybka 4.1                      :  48.0/117  41.0   2946     -40
  7 Junior 13                      :  48.5/120  40.4   2942     -73



Queensgambit
[D]rnbqkbnr/ppp1pppp/8/8/2pP4/8/PP2PPPP/RNBQKBNR w KQkq - 0 3[/D]
BR BN BB BQ BK BB BN BR
BP BP BP    BP BP BP BP
   ::    ::    ::    ::
::    ::    ::    ::    
   :: BP WP    ::    ::
::    ::    ::    ::    
WP WP    :: WP WP WP WP
WR WN WB WQ WK WB WN WR


Code:
    Program                            Score     %     Elo   Performance

  1 Critter 1.6                    :  68.0/119  57.1   3043     +37
  2 Stockfish 2.3.1                :  65.0/119  54.6   3028     +61
  3 Houdini 3                      :  60.5/120  50.4   3003      -9
  4 Komodo 5                       :  60.0/120  50.0   3000      -5
  5 Hiarcs 14                      :  59.0/120  49.2   2995     -14
  6 Junior 13                      :  54.5/120  45.4   2973     -42
  7 Rybka 4.1                      :  50.0/116  43.1   2958     -28
 



Sicilian
[D]rnbqkb1r/pp2pppp/3p1n2/8/3NP3/2N5/PPP2PPP/R1BQKB1R b KQkq - 0 5[/D]
BR BN BB BQ BK BB    BR
BP BP ::    BP BP BP BP
   ::    BP    BN    ::
::    ::    ::    ::    
   ::    WN WP ::    ::
::    WN    ::    ::    
WP WP WP ::    WP WP WP
WR    WB WQ WK WB :: WR


Code:
    Program                            Score     %     Elo   Performance

  1 Houdini 3                      :  69.5/119  58.4   3051     +39
  2 Critter 1.6                    :  64.5/119  54.2   3025     +19
  3 Komodo 5                       :  61.5/119  51.7   3010      +5
  4 Hiarcs 14                      :  62.0/120  51.7   3010      +1
  5 Stockfish 2.3.1                :  56.0/120  46.7   2980     +13
  6 Junior 13                      :  55.5/119  46.6   2980     -35
  7 Rybka 4.1                      :  47.0/116  40.5   2942     -44



English
[D]r1bqk1nr/pp1p1pbp/2n3p1/2p1p3/2P1P3/2N3P1/PP1P1PBP/R1BQK1NR w KQkq - 0 6[/D]
BR :: BB BQ BK :: BN BR
BP BP :: BP :: BP BB BP
   :: BN ::    :: BP ::
::    BP    BP    ::    
   :: WP :: WP ::    ::
::    WN    ::    WP    
WP WP    WP    WP WB WP
WR    WB WQ WK    WN WR


Code:
    Program                            Score     %     Elo   Performance

  1 Critter 1.6                    :  68.5/120  57.1   3043     +37
  2 Komodo 5                       :  68.0/120  56.7   3040     +35
  3 Houdini 3                      :  66.0/119  55.5   3033     +21
  4 Stockfish 2.3.1                :  63.5/120  52.9   3018     +51
  5 Hiarcs 14                      :  57.5/120  47.9   2988     -21
  6 Junior 13                      :  50.5/120  42.1   2953     -62
  7 Rybka 4.1                      :  45.0/119  37.8   2926     -60



Caro-Kann
[D]rn1qkbnr/pp2pppp/2p5/3pPb2/3P4/8/PPP2PPP/RNBQKBNR w KQkq - 0 4[/D]
BR BN    BQ BK BB BN BR
BP BP ::    BP BP BP BP
   :: BP ::    ::    ::
::    :: BP WP BB ::    
   ::    WP    ::    ::
::    ::    ::    ::    
WP WP WP ::    WP WP WP
WR WN WB WQ WK WB WN WR


Code:

    Program                            Score     %     Elo   Performance

  1 Critter 1.6                    :  72.0/119  60.5   3063     +57
  2 Komodo 5                       :  67.5/120  56.2   3037     +32
  3 Houdini 3                      :  58.5/119  49.2   2995     -17
  4 Stockfish 2.3.1                :  56.5/118  47.9   2988     +21
  5 Hiarcs 14                      :  57.0/120  47.5   2985     -24
  6 Junior 13                      :  55.0/120  45.8   2975     -40
  7 Rybka 4.1                      :  49.5/116  42.7   2956     -30



Kings-Indian
[D]rnbq1rk1/ppp1ppbp/3p1np1/8/2PPP3/2N2N2/PP2BPPP/R1BQK2R b KQ - 0 6[/D]
BR BN BB BQ    BR BK ::
BP BP BP    BP BP BB BP
   ::    BP    BN BP ::
::    ::    ::    ::    
   :: WP WP WP ::    ::
::    WN    :: WN ::    
WP WP    :: WB WP WP WP
WR    WB WQ WK    :: WR


Code:
    Program                            Score     %     Elo   Performance

  1 Critter 1.6                    :  75.5/120  62.9   3079     +73
  2 Houdini 3                      :  68.0/120  56.7   3040     +28
  3 Stockfish 2.3.1                :  67.5/120  56.2   3037     +70
  4 Komodo 5                       :  61.5/119  51.7   3011      +6
  5 Junior 13                      :  55.5/120  46.2   2978     -37
  6 Hiarcs 14                      :  49.0/120  40.8   2945     -64
  7 Rybka 4.1                      :  42.0/119  35.3   2910     -76



French Defence
[D]rnbqk1nr/ppp2ppp/4p3/3pP3/1b1P4/2N5/PPP2PPP/R1BQKBNR b KQ - 0 4[/D]
BR BN BB BQ BK :: BN BR
BP BP BP    :: BP BP BP
   ::    :: BP ::    ::
::    :: BP WP    ::    
   BB    WP    ::    ::
::    WN    ::    ::    
WP WP WP ::    WP WP WP
WR    WB WQ WK WB WN WR


Code:
    Program                            Score     %     Elo   Performance

  1 Houdini 3                      :  72.5/120  60.4   3063     +51
  2 Critter 1.6                    :  68.5/119  57.6   3046     +40
  3 Stockfish 2.3.1                :  61.5/120  51.2   3007     +40
  4 Junior 13                      :  59.5/120  49.6   2997     -22
  5 Komodo 5                       :  57.5/120  47.9   2988     -17
  6 Hiarcs 14                      :  53.5/120  44.6   2968     -41
  7 Rybka 4.1                      :  46.0/119  38.7   2931     -55



Spanish
[D]r1bqk2r/1pppbppp/p1n2n2/4p3/B3P3/5N2/PPPP1PPP/RNBQ1RK1 w kq - 0 6[/D]
BR :: BB BQ BK ::    BR
:: BP BP BP BB BP BP BP
BP :: BN ::    BN    ::
::    ::    BP    ::    
WB ::    :: WP ::    ::
::    ::    :: WN ::    
WP WP WP WP    WP WP WP
WR WN WB WQ :: WR WK    


Code:
    Program                            Score     %     Elo

  1 Critter 1.6                    :  77.0/120  64.2   3087     +81
  2 Komodo 5                       :  64.5/120  53.8   3022     +17
  3 Stockfish 2.3.1                :  62.5/118  53.0   3019     +52
  4 Hiarcs 14                      :  61.5/119  51.7   3011      +2
  5 Houdini 3                      :  59.5/119  50.0   3001     -11
  6 Junior 13                      :  52.0/120  43.3   2960     -55
  7 Rybka 4.1                      :  39.0/116  33.6   2899     -87
 




Engines outperform and underperform significantly (with more than 50 Elo points or 2 standard deviations) as follows:

Houdini 3:
Outperforms: French Defence
Underperforms: None

Komodo 5:
Outperforms: None
Underperforms: None

Critter 1.6:
Outperforms: Grunfeld, Caro-Kann, Kings-Indian, Spanish
Underperforms: None

Stockfish 2.3.1:
Outperforms: Grunfeld, Nimzo-Indian, Qeensgambit, English, Kings-Indian, Spanish
Underperforms: None

Rybka 4.1:
Outperforms: None
Underperforms: Grunfeld, English, Kings-Indian, French Defence, Spanish

Hiarcs 14:
Outperforms: None
Underperforms: Kings-Indian

Junior 13:
Outperforms: None
Underperforms: Grunfeld, Dutch - Staunton-Gambit, Nimzo-Indian, English, Spanish


Komodo, Houdini, Hiarcs have no preferences, Stockfish and Critter play better prefered openings, Rybka and Junior play badly without books.

Kai
Back to top
View user's profile Send private message
Display posts from previous:   
Subject Author Date/Time
Engines performance for selected openings Kai Laskos Sun Dec 23, 2012 5:00 pm
      Re: Engines performance for selected openings Don Dailey Sun Dec 23, 2012 8:10 pm
      Re: Engines performance for selected openings om prakash Mon Dec 24, 2012 1:23 am
            Re: Engines performance for selected openings Kah Huat, Koh Mon Dec 24, 2012 2:26 am
                  Re: Engines performance for selected openings Kai Laskos Mon Dec 24, 2012 11:21 am
                        Re: Engines performance for selected openings Vincent Lejeune Mon Dec 24, 2012 12:45 pm
                              Re: Engines performance for selected openings. Jesús Muñoz Mon Dec 24, 2012 1:03 pm
                        Re: Engines performance for selected openings Don Dailey Mon Dec 24, 2012 1:25 pm
                              Re: Engines performance for selected openings Kai Laskos Mon Dec 24, 2012 3:09 pm
                                    Re: Engines performance for selected openings Don Dailey Mon Dec 24, 2012 3:44 pm
                                          Re: Engines performance for selected openings Kai Laskos Mon Dec 24, 2012 5:51 pm
                                                Re: Engines performance for selected openings Don Dailey Mon Dec 24, 2012 6:05 pm
            Re: Engines performance for selected openings Kai Laskos Mon Dec 24, 2012 11:07 am
                  Re: Engines performance for selected openings om prakash Mon Dec 24, 2012 11:55 am
                        Re: Engines performance for selected openings Kah Huat, Koh Mon Dec 24, 2012 12:05 pm
Post new topic    TalkChess.com Forum Index -> Computer Chess Club: General Topics

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum




Powered by phpBB © 2001, 2005 phpBB Group
Enhanced with Moby Threads