Don wrote:Laskos wrote:Kohflote wrote:Hi,
Firstly, thank you for the info and is definitely helpful.
Can you please clarify the elo and performance of each opening that you have given, does that mean is the elo and performance of the engine playing black?
What about Pirc Defence?
Best wishes,
Koh, Kah Huat
The performances for selected openings are compared with the base performance in 16,000+ games of adjusted for strength engines given in the first table. The engines use different time controls, so that their strength is comparable. These 16,000+ games were played using 5,000+ PGN opening 8-move positions by Frank Quisinsky, they are pretty balanced, and not too deep, similar to a generic book. Each of the selected openings is played by all engines with both white and black, so the performances of engines are combined white/black performances for each opening.
I will post the results for Pirc Defence soon.
Kai
So you are saying that how well an engine does in one particular opening is determined by it's performance on both sides of that opening.
I am wondering if it was be more useful to break this down into defense and offense. In real games a program may stronger prefer defending and opening it would never choose. Humans wont' play openings they think are inferior but would prefer to defend them (for that same reason.)
A question I have always had is whether (in general) a program will play an opening better if it "likes" the opening. Each program seems to have openings that due to their evaluation function they are "happy" with compared to others. Does that mean they will also play those openings better or could it even mean they will play them worse? Presumably there is a chance they are mis-evalating an opening that they prefer over others and that is why the opening is getting a higher score than the others.
Don
Yes, on both sides of opening. I am aware that this has its flaws, my assumption is that if an engine "understands" the opening, and this is a mainstream, balanced opening without short-term hits, then it will play it better both sides. Also, I am not very skilled in separating the PGN file in White and Black performances for each engine, I would rather play gauntlets with fixed colours, a very lengthy process. Then I would have to compare to general White and Black performances in that 16,000 games file.
I agree that an engine could play even worse the balanced openings it "likes", I do not know if it's more mis-evaluation or more a feature of an engine, and that would be interesting to study (my general opinion is mixed on that).
I put 3 more openings to test, the Slav Defence and the Catalan are to follow:
Pirc Defence
[D]rnbqkb1r/ppp1pp1p/3p1np1/8/3PP3/2N5/PPP2PPP/R1BQKBNR w KQkq - 0 4[/D]
Code: Select all
Program Score % Elo Performance
1 Junior 13 : 65.0/118 55.1 3030 +15
2 Houdini 3 : 63.0/120 52.5 3015 +3
3 Komodo 5 : 61.5/120 51.2 3008 +3
4 Rybka 4.1 : 59.0/118 50.0 3000 +14
5 Hiarcs 14 : 58.5/120 48.8 2993 -16
6 Critter 1.6 : 58.0/120 48.3 2990 -16
7 Stockfish 2.3.1 : 53.0/120 44.2 2965 -2
Four Knights
[D]r1bqkb1r/pppp1ppp/2n2n2/4p3/4P3/2N2N2/PPPP1PPP/R1BQKB1R w KQkq - 0 4[/D]
Code: Select all
Program Score % Elo Performance
1 Critter 1.6 : 64.0/120 53.3 3020 +14
2 Hiarcs 14 : 62.0/119 52.1 3013 +4
3 Rybka 4.1 : 58.5/115 50.9 3005 +19
4 Komodo 5 : 60.0/118 50.8 3005 0
5 Houdini 3 : 59.0/117 50.4 3002 -10
6 Junior 13 : 59.0/119 49.6 2998 -17
7 Stockfish 2.3.1 : 51.5/120 42.9 2958 -9
No engine under- or over-performs in these 2 openings beyond error margins (50 Elo points)
Semi-Slav Defence
[D]rnbqkb1r/pp3ppp/2p1pn2/3p4/2PP4/2N2N2/PP2PPPP/R1BQKB1R w KQkq - 0 5[/D]
Code: Select all
Program Score % Elo Performance
1 Komodo 5 : 69.5/119 58.4 3055 +50
2 Critter 1.6 : 66.5/119 55.9 3035 +29
3 Junior 13 : 61.0/120 50.8 3005 -10
4 Rybka 4.1 : 57.0/116 49.1 2995 +9
5 Stockfish 2.3.1 : 55.5/118 47.0 2982 +15
6 Houdini 3 : 55.5/119 46.6 2980 -32
7 Hiarcs 14 : 50.0/119 42.0 2952 -57
Komodo overperforms, Hiarcs underperforms.