The scaling of Houdini 3 and Komodo 5

[Laskos]

We talk very much about Elo differences at different time controls to infer the scaling, but I think that the scaling behavior is better seen as time handicap factor needed for a stronger engine to become equal in strength to a weaker engine. And this is equivalent to a hardware handicap (assuming we know how the engines behave on several cores, etc.).

For short time controls I found the following:

Komodo at 100ms time/move (hyper-bullet) needs 2.2~2.3 time handicap versus Houdini 3. I played several direct matches with Houdini 3 being handicapped (shorter time control), and found time/move values of 100ms for Komodo 5 and 44ms for Houdini 3 to have a very balanced result of 2974 Houdini vs 3026 Komodo. They are equal at these time controls within error margins, LOS is 78%, inconclusive.

Then, I increased both TC by a factor of 10 (bullet), Komodo 5 at 1000ms/move, Houdini 3 at 440ms/move. If they scale identically, the match result should be inconclusive. But Houdini 3 won the match 324 to 276 with a LOS of 98%. From this result we can get the handicap needed (assuming 70-80 Elo points for doubling at blitz) as 2.5~2.6 factor at this 10x times longer control.

So, there is 95-98% confidence that Houdini 3 scales better than Komodo 5 from hyper-bullet to blitz, handicap time factors or hardware needed rising from 2.2~2.3 to 2.5~2.6 respectively. If we assume that long time controls need a further factor of 100 in time, then by extrapolating we get a handicap factor (for time or hardware) of 3.1~3.4 at long time controls. For long time control it is an extrapolation, and uncertainties are still large. Basically, my prediction is that at hyper-bullet, two core Komodo 5 is pretty equal to single core Houdini 3, at very long TC Komodo 5 on 4 cores is equal to single core Houdini 3. Still speculative for long TC, though.

Kai

[Don]

I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable.

However I usually do scaling experiments at multiple time controls for both programs - then you can simply see the slope of the lines. If the lines are not parallel then the two programs scale differently. Note that both programs must be run at different time controls for this to be clear unless the different is large enough that the line "crosses."

Graphing woks very well because there is automatic interpolation, you can predict the crossover point very accurately without actually running the exact level. I generally use gnuplot which make creating a graph trivial.

Most people only view program strength locally, in other words, "how good does it play on my current machine at a few seconds per move" because that is the most relevant definition for them, that's what everyone seems to care about. That is a good definition for an engineer but a scientist usually thinks differently and applies a more general or global definition.

You can take almost ANY reasonably well written program and make it stronger than Komodo or Houdini in a local sense if you can make it run fast enough. But this is equivalent to running it on faster hardware, it does not change the core program or how well it scales. It's a practical consideration but not a theoretical one.

As far as your study of Houdini 3 scalability I cannot comment on that since I have no data of my own. We allocate only a small fraction of our time studying the behavior of other programs but we do spend some time on things like that.

Don

[beram]

I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...

Don

Thx Don (and Kai), for explaining this further.

But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?

Because I hope Komodo MP will come out soon

[Don]

I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...

Don

Thx Don (and Kai), for explaining this further.

But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?

Because I hope Komodo MP will come out soon

I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start. I have at least 4 reasons:

1. I want to learn c++
2. I simulate features easier to do in c++
3. current komodo not well designed for MP.
4. current code has accumulated years of cruft.


I originally did design Komodo with MP in mind but with age there have been numerous changes that make it more difficult. Different bits and pieces of Komodo assume SP. None of the above require a re-write, I could always just take some time to clean up the code, etc ... but together there is a pretty compelling case for me.

Also, Komodo was never designed for performance, I had always intended for that the be the last stop (so to speak) but it's past due for that. I never expected Komodo to get this strong without the optimizations. I now think of the current code base as a proof of concept prototype - which is really what it was always intended to be.

A few months ago I did experiment with some simple optimizations and was able to achieve a lot of performance with just a day of work so that was pretty encouraging. I thank Richard Vida for giving me some suggestions and I implemented some of them as well as others. So I am pretty excited about getting into this - you are going to LOVE the final result!

[Graham Banks]

I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start..........you are going to LOVE the final result!

Looking forward to it!

[beram]

I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...

Don

Thx Don (and Kai), for explaining this further.

But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?

Because I hope Komodo MP will come out soon

I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start. I have at least 4 reasons:

1. I want to learn c++
2. I simulate features easier to do in c++
3. current komodo not well designed for MP.
4. current code has accumulated years of cruft.


I originally did design Komodo with MP in mind but with age there have been numerous changes that make it more difficult. Different bits and pieces of Komodo assume SP. None of the above require a re-write, I could always just take some time to clean up the code, etc ... but together there is a pretty compelling case for me.

Also, Komodo was never designed for performance, I had always intended for that the be the last stop (so to speak) but it's past due for that. I never expected Komodo to get this strong without the optimizations. I now think of the current code base as a proof of concept prototype - which is really what it was always intended to be.

A few months ago I did experiment with some simple optimizations and was able to achieve a lot of performance with just a day of work so that was pretty encouraging. I thank Richard Vida for giving me some suggestions and I implemented some of them as well as others. So I am pretty excited about getting into this - you are going to LOVE the final result!

Sounds promising, but probably this will take you quite more time than just a few months. Nevertheless looking forward to it.

grts bram

[Dr.Wael Deeb]

I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...

Don

Thx Don (and Kai), for explaining this further.

But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?

Because I hope Komodo MP will come out soon

I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start. I have at least 4 reasons:

1. I want to learn c++
2. I simulate features easier to do in c++
3. current komodo not well designed for MP.
4. current code has accumulated years of cruft.


I originally did design Komodo with MP in mind but with age there have been numerous changes that make it more difficult. Different bits and pieces of Komodo assume SP. None of the above require a re-write, I could always just take some time to clean up the code, etc ... but together there is a pretty compelling case for me.

Also, Komodo was never designed for performance, I had always intended for that the be the last stop (so to speak) but it's past due for that. I never expected Komodo to get this strong without the optimizations. I now think of the current code base as a proof of concept prototype - which is really what it was always intended to be.

A few months ago I did experiment with some simple optimizations and was able to achieve a lot of performance with just a day of work so that was pretty encouraging. I thank Richard Vida for giving me some suggestions and I implemented some of them as well as others. So I am pretty excited about getting into this - you are going to LOVE the final result!

Good luck Don with your project....

I'll keep my fingers crossed for you regards,
Dr.D

[hammerklavier]

Hi Don!
Komodo was written in C ?

is possible implement this? ---->http://en.wikipedia.org/wiki/Cilk

excuse me my ignorance.

[Don]

Hi Don!
Komodo was written in C ?

is possible implement this? ---->http://en.wikipedia.org/wiki/Cilk

excuse me my ignorance.

I was a part of the Cilk team long ago. However I want Komodo to be portable for any conceivable situation including cluster usage eventually.

Also, Cilk has lost some features which made it especially convenient for chess. However it would still be a good choice.

Don

[Werewolf]

Hi Don!
Komodo was written in C ?

is possible implement this? ---->http://en.wikipedia.org/wiki/Cilk

excuse me my ignorance.

I was a part of the Cilk team long ago. However I want Komodo to be portable for any conceivable situation including cluster usage eventually.

Also, Cilk has lost some features which made it especially convenient for chess. However it would still be a good choice.

Don

Hi Don,
How many NPS did Cilk Chess examine? I heard it was very fast for the time. Did it have specialist hardware?