Komodo 3 v Strelka 5 -- 100 games 3 minutes

[PawnStormZ]

       Hi all.

       I recently ran a 100 game match between the 64-bit versions of Strelka 5 and Komodo 3.

       Match conditions: Intel core I7 920 at 2.66GHz using 4 cores HT off; Win 7 64 Home Premimum; 6G RAM; pc doing nothing but the match while a game is being played.

       Each engine uses default settings; no large pages; no end-game tablebases; 512 hash each; 1 core each; ponder = ON.

       Time control is game in 3 minutes. I am using the excellent "ChessGUI" to run the match, set to give each engine a 1/10th second "overstep" margin to help avoid any time losses.

       The openings are played using random choices from a set of epd positions after 8 moves in games of high-rated players; each position is played as Black and White by each engine.

       I decided to have ponder=ON since with the 4 cores there will not be a penalty to either engine by letting the other think while not "on the move".
   
                       Results:                games are here... http://www.datafilehost.com/download-5e439b6e.html
                                  
                               Strelka 5          57.5    + 35    = 45    - 20
                               Komodo 3        42.5    + 20    = 45    - 35

       

[ernest]

ponder = ON.
       Time control is game in 3 minutes. I am using the excellent "ChessGUI" to run the match, set to give each engine a 1/10th second "overstep" margin to help avoid any time losses.

Hi Pal,

Amazing!!!!!!!!!!!!!!!!
ponder = ON and no crashes with Strelka 5!
(with Fritz and Shredder GUI, it's unplayable, ask me and Ingo Bauer, with Arena I don't know..., and Strelka 5.1 is not much better)

There has to be an explanation!

[Matthias Gemuh]

ponder = ON.
       Time control is game in 3 minutes. I am using the excellent "ChessGUI" to run the match, set to give each engine a 1/10th second "overstep" margin to help avoid any time losses.

Hi Pal,

Amazing!!!!!!!!!!!!!!!!
ponder = ON and no crashes with Strelka 5!
(with Fritz and Shredder GUI, it's unplayable, ask me and Ingo Bauer, with Arena I don't know..., and Strelka 5.1 is not much better)

There has to be an explanation!

The best explanation is that the move that an UCI engine must send to GUI when there is a ponder miss is always discarded.
So ChessGUI does not penalize an engine if it "forgets" to send it.

[IWB]

Hello Matthias,

The best explanation is that the move that an UCI engine must send to GUI when there is a ponder miss is always discarded.

What exactly do you mean by " move ... that an UCI engine must send to GUI when ... ponder miss"? I am a bit puzzled by this as there is nothing official in the protocoll for a "ponder miss" but just for a "ponderhit" where the engine only get a ponderhit and
Quote: " ... This will be sent if the engine was told to ponder on the same move the user has played. The engine should continue searching but switch from pondering to normal search."
When there is a pondermiss the engine usually only should get a position and a go ... ( position [fen <fenstring> | startpos ] moves <move1> .... <movei>) (go start calculating on the current position set up with the "position" command)
I am not sure if I understand what you want to say. Please help.

So ChessGUI does not penalize an engine if it "forgets" to send it.

That sounds a bit like you are working around bad implementations of the UCI-protocol in engines. If so, this will not help to improve the engines ...

Maybe I completly missunderstood something here.

Bye
Ingo

[Matthias Gemuh]

Hello Matthias,

The best explanation is that the move that an UCI engine must send to GUI when there is a ponder miss is always discarded.

What exactly do you mean by " move ... that an UCI engine must send to GUI when ... ponder miss"? I am a bit puzzled by this as there is nothing official in the protocoll for a "ponder miss" but just for a "ponderhit" where the engine only get a ponderhit and
Quote: " ... This will be sent if the engine was told to ponder on the same move the user has played. The engine should continue searching but switch from pondering to normal search."
When there is a pondermiss the engine usually only should get a position and a go ... ( position [fen <fenstring> | startpos ] moves <move1> .... <movei>) (go start calculating on the current position set up with the "position" command)
I am not sure if I understand what you want to say. Please help.

So ChessGUI does not penalize an engine if it "forgets" to send it.

That sounds a bit like you are working around bad implementations of the UCI-protocol in engines. If so, this will not help to improve the engines ...

Maybe I completly missunderstood something here.

Bye
Ingo

Hi Ingo,

every "go" must be answered sooner or later by sending a bestmove, whether search was a complete one or just interrupted by a "stop" as in the case of a ponder miss.

20% of all WB engines would not work if GUI authors don't break protocol.
That is why I don't penalize the few UCI engines that misbehave.

Matthias.

[Graham Banks]

Hi Ingo,

every "go" must be answered sooner or later by sending a bestmove, whether search was a complete one or just interrupted by a "stop" as in the case of a ponder miss.

20% of all WB engines would not work if GUI authors don't break protocol.
That is why I don't penalize the few UCI engines that misbehave.

Matthias.

Well Matthias - at least you know that those of us who ChessGUI appreciate all that you've done to make engine v engine testing much more pleasurable.

[Matthias Gemuh]

Hi Ingo,

every "go" must be answered sooner or later by sending a bestmove, whether search was a complete one or just interrupted by a "stop" as in the case of a ponder miss.

20% of all WB engines would not work if GUI authors don't break protocol.
That is why I don't penalize the few UCI engines that misbehave.

Matthias.

Well Matthias - at least you know that those of us who ChessGUI appreciate all that you've done to make engine v engine testing much more pleasurable.

Some of the guys who break the WB protocol (to different extents) are :
Martin, HGM, Matthias, Ilari, Gunnar, etc.

[IWB]

Hello Mathias,

Hi Ingo,

every "go" must be answered sooner or later by sending a bestmove, whether search was a complete one or just interrupted by a "stop" as in the case of a ponder miss.

Ahh now I got you, you mean that more in general as discarding bestmoves from a non ponderhit is mandatory for every GUI.

E.g:
...
Mon Nov 07 06:52:06 2011: from Deep Shredder 12 x64 (0): bestmove e7e5 ponder c2c4
Mon Nov 07 06:52:06 2011: to Deep Shredder 12 x64 (0): position startpos moves h2h3 e7e5 c2c4
Mon Nov 07 06:52:06 2011: to Deep Shredder 12 x64 (0): go ponder wtime 306000 btime 290134 winc 3000 binc 3000
....
Mon Nov 07 06:52:15 2011: to Deep Shredder 12 x64 (0): stop
Mon Nov 07 06:52:15 2011: from Deep Shredder 12 x64 (0): info string Shredder got command stop
Mon Nov 07 06:52:15 2011: from Deep Shredder 12 x64 (0): info refutation e8e7
Mon Nov 07 06:52:15 2011: from Deep Shredder 12 x64 (0): info nps 3165068 cpuload 3650 nodes 29672516 hashfull 36
Mon Nov 07 06:52:15 2011: from Deep Shredder 12 x64 (0): bestmove b8c6
Mon Nov 07 06:52:15 2011: to Deep Shredder 12 x64 (0): position startpos moves h2h3 e7e5 a2a3
Mon Nov 07 06:52:15 2011: to Deep Shredder 12 x64 (0): go wtime 299625 btime 290134 winc 3000 binc 3000
...
As you can see the bestmove @ the 5th 6.52.15 is simply ignored (what else to do?).
I was mainly confused because your proposal is not solving the problem of Strelka!

20% of all WB engines would not work if GUI authors don't break protocol.
That is why I don't penalize the few UCI engines that misbehave.

Matthias.

I know that the UCI protocoll asks to ignore wrong things, but to assume something right when the implementation is bluntly wrong might not be a solution!
And regarding your 20% which would not work ... maybe WB-protocoll would be more accepted by the users if all engines would behave identical and the configuration would be equal (... wait, its called UCI then) but because the GUI programmers work araund the problems the engine programmers become lazy.

I personaly think that if there is a standard in case of a breaking of the rules it should be enforced by the GUI (or the engine). I see different interpretations of the protocoll in GUIs and personaly thing that this is not good for the end user!

Have a nice week
Ingo

[Graham Banks]

.......I see different interpretations of the protocoll in GUIs and personaly thing that this is not good for the end user!

Have a nice week
Ingo

Really depends what the user is looking for. Under ChessGUI, I can use more engines reliably than in any other GUI, plus I get very few crashes and time losses. When I do strike problems, I can send the automatically produced debug files to either Matthias or the engine author to help remedy any issues. The key is helping the engine authors to improve their engines by doing so.

For engine v engine testing, I'd have to say that ChessGUI is now my preferred GUI.
However, different users have different wants and preferences and there's nothing wrong with that.

Graham.

[IWB]

Hi Graham,

I dont agree. Beeing very broad in a GUI protocoll implementation means that enignes evolves into anything but not UCI anymore. The more liberal a GUI is the more the programmers get "lazy" the more other GUIs have problems (maybe that is the reason why WB engines differ so much).

I can not tell you how often I have read (e.g. in the last weeks with Strelka) in the german fora the sentence "but it runs here, use a proper GUI" despite the fact that the problem is 100% in the engine ... I know that the end user dont care and dont know better, but on the long run it might end like in WB (where you never know if the running configuration you have is really the best possible one)
I favor a strict implementation of protocolls (regardless which one) and as people blame the GUI regardless of the real problem I can only call for more "solidarity" from the GUI programmer side!

Bye and a nice week for you
Ingo