Peculiarity of Komodo 5.1MP

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
Laskos
Posts: 9441
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Peculiarity of Komodo 5.1MP

Post by Laskos » Wed Jun 19, 2013 11:48 am

I tested the latest Komodo, Houdini and Stockfish for MP implementation. I put play Engine 4 threads against the same Engine on 1 thread, both to fixed depth 11 for Komodo and Houdini, depth 12 for Stockfish.
4 physical i7 cores.

Code: Select all

    Program                            Score     %     Elo

  1 Komodo 4 threads               : 183.0/300  61.0   3039
  2 Komodo 1 thread                : 117.0/300  39.0   2961


    Program                            Score     %     Elo

  1 Houdini 4 threads              : 181.0/360  50.3   3001
  2 Houdini 1 thread               : 179.0/360  49.7   2999


    Program                            Score     %     Elo 

  1 SF 4 threads                   : 221.5/434  51.0   3004
  2 SF 1 thread                    : 212.5/434  49.0   2996
As it can be seen Houdini and Stockfish are within error margins equal to constant depth on 4 and 1 thread. However Komodo 5.1MP shows +80 points increase for 4 threads compared to 1 thread to fixed depth 11. So, time to depth is an incorrect way of calculating Komodo's MP efficiency. It seems that it increases the width of the tree as much as it increases the depth with number of threads.

yanquis1972
Posts: 1762
Joined: Tue Jun 02, 2009 10:14 pm

Re: Peculiarity of Komodo 5.1MP

Post by yanquis1972 » Wed Jun 19, 2013 12:09 pm

interesting...is this a unique (in the literal sense) implementation of MP in a chess engine?

User avatar
Laskos
Posts: 9441
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Peculiarity of Komodo 5.1MP

Post by Laskos » Wed Jun 19, 2013 12:16 pm

yanquis1972 wrote:interesting...is this a unique (in the literal sense) implementation of MP in a chess engine?
First time I encounter, but I only tested very few engines. Some folks like Bob Hyatt even stated bluntly that time-to-depth is the only way to measure MP efficiency. Not for Komodo.

lkaufman
Posts: 3722
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Peculiarity of Komodo 5.1MP

Post by lkaufman » Wed Jun 19, 2013 2:08 pm

Yes, the implementation of MP in Komodo is nothing like that in other top chess engines, and the observation is correct that Komodo MP is much stronger at the same depth than SP, but the speedup is also much less than for other engines so the end result is fairly close. I suspect that our method would be useless for a traditional full-width engine but works for highly selective engines like ours. I'll leave it to Don to decide what we wants to say about MP implementation.

Daniel Shawul
Posts: 3757
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: Peculiarity of Komodo 5.1MP

Post by Daniel Shawul » Wed Jun 19, 2013 2:53 pm

This is misleading because komodo probably compensates for poor parallel implementation by searching wider, otherwise it shouldn't be getting any elos for fixed depth test. You should do the test with time and see how much each gain.

User avatar
Laskos
Posts: 9441
Joined: Wed Jul 26, 2006 8:21 pm
Full name: Kai Laskos

Re: Peculiarity of Komodo 5.1MP

Post by Laskos » Wed Jun 19, 2013 3:14 pm

Daniel Shawul wrote:This is misleading because komodo probably compensates for poor parallel implementation by searching wider, otherwise it shouldn't be getting any elos for fixed depth test. You should do the test with time and see how much each gain.
I already wrote that it probably searches wider with the number of threads. But it's different from other top engines, which to given depth are pretty much the same strength on different number of threads. And the rule (as stated by some) that time-to-depth is determining MP efficiency does not apply to Komodo.
I will leave to test groups and individuals to test with time, there are plenty of volunteers. I was just curious about this aspect.

Uri Blass
Posts: 8586
Joined: Wed Mar 08, 2006 11:37 pm
Location: Tel-Aviv Israel

Re: Peculiarity of Komodo 5.1MP

Post by Uri Blass » Wed Jun 19, 2013 3:18 pm

Daniel Shawul wrote:This is misleading because komodo probably compensates for poor parallel implementation by searching wider, otherwise it shouldn't be getting any elos for fixed depth test. You should do the test with time and see how much each gain.
"poor parallel implementation?"

The target of parallel implementation is not to get bigger depth but to play better.

If the implementation is good in helping komodo to play better than I think that it is wrong to call it poor.

Maybe it is the opposite and komodo compensates for poor pruning of lines that it should not prune by parallel implementation that prevent it to prune good moves in the relevant lines.

Daniel Shawul
Posts: 3757
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: Peculiarity of Komodo 5.1MP

Post by Daniel Shawul » Wed Jun 19, 2013 3:20 pm

Laskos wrote:
Daniel Shawul wrote:This is misleading because komodo probably compensates for poor parallel implementation by searching wider, otherwise it shouldn't be getting any elos for fixed depth test. You should do the test with time and see how much each gain.
I already wrote that it probably searches wider with the number of threads. But it's different from other top engines, which to given depth are pretty much the same strength on different number of threads. And the rule (as stated by some) that time-to-depth is determining MP efficiency does not apply to Komodo.
I will leave to test groups and individuals to test with time, there are plenty of volunteers. I was just curious about this aspect.
I saw it but I felt like stating the obvious once again should help consumers in making decisions.
I said you should do a timed test not time-to-depth. Simply play a game of 4 threads vs 1 thread for say 40/1. That should really tell how much each engine gain from parallelization.

Daniel Shawul
Posts: 3757
Joined: Tue Mar 14, 2006 10:34 am
Location: Ethiopia
Contact:

Re: Peculiarity of Komodo 5.1MP

Post by Daniel Shawul » Wed Jun 19, 2013 3:23 pm

Uri Blass wrote:
Daniel Shawul wrote:This is misleading because komodo probably compensates for poor parallel implementation by searching wider, otherwise it shouldn't be getting any elos for fixed depth test. You should do the test with time and see how much each gain.
"poor parallel implementation?"

The target of parallel implementation is not to get bigger depth but to play better.

If the implementation is good in helping komodo to play better than I think that it is wrong to call it poor.

Maybe it is the opposite and komodo compensates for poor pruning of lines that it should not prune by parallel implementation that prevent it to prune good moves in the relevant lines.
Uri, I don't think you understood the test well. It was a _fixed depth_ test that is meaningless unless both parallel and serial versions search more or less similar tree.
If i decided to use alpha-beta only for the parallel search while using lmr+nullmove+other pruning for the sequential search who would you think would win for a fixed depth test. Get it?

User avatar
michiguel
Posts: 6388
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: Peculiarity of Komodo 5.1MP

Post by michiguel » Wed Jun 19, 2013 3:51 pm

Laskos wrote:
yanquis1972 wrote:interesting...is this a unique (in the literal sense) implementation of MP in a chess engine?
First time I encounter, but I only tested very few engines. Some folks like Bob Hyatt even stated bluntly that time-to-depth is the only way to measure MP efficiency. Not for Komodo.
I remember the thread and I did not agree with that. What matters, for any engine, it is the elo gained. Time to depth for a selected number of representative (are they?) positions may or may not be an accurate way to predict how much strength is gained, for several reasons. There are too many assumptions in the process.

Miguel

Post Reply