We talk very much about Elo differences at different time controls to infer the scaling, but I think that the scaling behavior is better seen as time handicap factor needed for a stronger engine to become equal in strength to a weaker engine. And this is equivalent to a hardware handicap (assuming we know how the engines behave on several cores, etc.).
For short time controls I found the following:
Komodo at 100ms time/move (hyper-bullet) needs 2.2~2.3 time handicap versus Houdini 3. I played several direct matches with Houdini 3 being handicapped (shorter time control), and found time/move values of 100ms for Komodo 5 and 44ms for Houdini 3 to have a very balanced result of 2974 Houdini vs 3026 Komodo. They are equal at these time controls within error margins, LOS is 78%, inconclusive.
Then, I increased both TC by a factor of 10 (bullet), Komodo 5 at 1000ms/move, Houdini 3 at 440ms/move. If they scale identically, the match result should be inconclusive. But Houdini 3 won the match 324 to 276 with a LOS of 98%. From this result we can get the handicap needed (assuming 70-80 Elo points for doubling at blitz) as 2.5~2.6 factor at this 10x times longer control.
So, there is 95-98% confidence that Houdini 3 scales better than Komodo 5 from hyper-bullet to blitz, handicap time factors or hardware needed rising from 2.2~2.3 to 2.5~2.6 respectively. If we assume that long time controls need a further factor of 100 in time, then by extrapolating we get a handicap factor (for time or hardware) of 3.1~3.4 at long time controls. For long time control it is an extrapolation, and uncertainties are still large. Basically, my prediction is that at hyper-bullet, two core Komodo 5 is pretty equal to single core Houdini 3, at very long TC Komodo 5 on 4 cores is equal to single core Houdini 3. Still speculative for long TC, though.
Kai
The scaling of Houdini 3 and Komodo 5
Moderators: hgm, Rebel, chrisw
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
-
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: The scaling of Houdini 3 and Komodo 5
I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable.
However I usually do scaling experiments at multiple time controls for both programs - then you can simply see the slope of the lines. If the lines are not parallel then the two programs scale differently. Note that both programs must be run at different time controls for this to be clear unless the different is large enough that the line "crosses."
Graphing woks very well because there is automatic interpolation, you can predict the crossover point very accurately without actually running the exact level. I generally use gnuplot which make creating a graph trivial.
Most people only view program strength locally, in other words, "how good does it play on my current machine at a few seconds per move" because that is the most relevant definition for them, that's what everyone seems to care about. That is a good definition for an engineer but a scientist usually thinks differently and applies a more general or global definition.
You can take almost ANY reasonably well written program and make it stronger than Komodo or Houdini in a local sense if you can make it run fast enough. But this is equivalent to running it on faster hardware, it does not change the core program or how well it scales. It's a practical consideration but not a theoretical one.
As far as your study of Houdini 3 scalability I cannot comment on that since I have no data of my own. We allocate only a small fraction of our time studying the behavior of other programs but we do spend some time on things like that.
Don
However I usually do scaling experiments at multiple time controls for both programs - then you can simply see the slope of the lines. If the lines are not parallel then the two programs scale differently. Note that both programs must be run at different time controls for this to be clear unless the different is large enough that the line "crosses."
Graphing woks very well because there is automatic interpolation, you can predict the crossover point very accurately without actually running the exact level. I generally use gnuplot which make creating a graph trivial.
Most people only view program strength locally, in other words, "how good does it play on my current machine at a few seconds per move" because that is the most relevant definition for them, that's what everyone seems to care about. That is a good definition for an engineer but a scientist usually thinks differently and applies a more general or global definition.
You can take almost ANY reasonably well written program and make it stronger than Komodo or Houdini in a local sense if you can make it run fast enough. But this is equivalent to running it on faster hardware, it does not change the core program or how well it scales. It's a practical consideration but not a theoretical one.
As far as your study of Houdini 3 scalability I cannot comment on that since I have no data of my own. We allocate only a small fraction of our time studying the behavior of other programs but we do spend some time on things like that.
Don
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
- Posts: 1187
- Joined: Wed Jan 06, 2010 3:11 pm
Re: The scaling of Houdini 3 and Komodo 5
Thx Don (and Kai), for explaining this further.Don wrote:I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...
Don
But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?
Because I hope Komodo MP will come out soon
-
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: The scaling of Houdini 3 and Komodo 5
I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start. I have at least 4 reasons:beram wrote:Thx Don (and Kai), for explaining this further.Don wrote:I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...
Don
But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?
Because I hope Komodo MP will come out soon
1. I want to learn c++
2. I simulate features easier to do in c++
3. current komodo not well designed for MP.
4. current code has accumulated years of cruft.
I originally did design Komodo with MP in mind but with age there have been numerous changes that make it more difficult. Different bits and pieces of Komodo assume SP. None of the above require a re-write, I could always just take some time to clean up the code, etc ... but together there is a pretty compelling case for me.
Also, Komodo was never designed for performance, I had always intended for that the be the last stop (so to speak) but it's past due for that. I never expected Komodo to get this strong without the optimizations. I now think of the current code base as a proof of concept prototype - which is really what it was always intended to be.
A few months ago I did experiment with some simple optimizations and was able to achieve a lot of performance with just a day of work so that was pretty encouraging. I thank Richard Vida for giving me some suggestions and I implemented some of them as well as others. So I am pretty excited about getting into this - you are going to LOVE the final result!
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
- Posts: 41461
- Joined: Sun Feb 26, 2006 10:52 am
- Location: Auckland, NZ
Re: The scaling of Houdini 3 and Komodo 5
Looking forward to it!Don wrote:I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start..........you are going to LOVE the final result!
gbanksnz at gmail.com
-
- Posts: 1187
- Joined: Wed Jan 06, 2010 3:11 pm
Re: The scaling of Houdini 3 and Komodo 5
Sounds promising, but probably this will take you quite more time than just a few months. Nevertheless looking forward to it.Don wrote:I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start. I have at least 4 reasons:beram wrote:Thx Don (and Kai), for explaining this further.Don wrote:I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...
Don
But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?
Because I hope Komodo MP will come out soon
1. I want to learn c++
2. I simulate features easier to do in c++
3. current komodo not well designed for MP.
4. current code has accumulated years of cruft.
I originally did design Komodo with MP in mind but with age there have been numerous changes that make it more difficult. Different bits and pieces of Komodo assume SP. None of the above require a re-write, I could always just take some time to clean up the code, etc ... but together there is a pretty compelling case for me.
Also, Komodo was never designed for performance, I had always intended for that the be the last stop (so to speak) but it's past due for that. I never expected Komodo to get this strong without the optimizations. I now think of the current code base as a proof of concept prototype - which is really what it was always intended to be.
A few months ago I did experiment with some simple optimizations and was able to achieve a lot of performance with just a day of work so that was pretty encouraging. I thank Richard Vida for giving me some suggestions and I implemented some of them as well as others. So I am pretty excited about getting into this - you are going to LOVE the final result!
grts bram
-
- Posts: 9773
- Joined: Wed Mar 08, 2006 8:44 pm
- Location: Amman,Jordan
Re: The scaling of Houdini 3 and Komodo 5
Don wrote:I am working on a complete rewrite of Komodo in c++11 for a couple of reasons and it will be MP from the start. I have at least 4 reasons:beram wrote:Thx Don (and Kai), for explaining this further.Don wrote:I agree with this. It's not enough to see a program "catching" another program from behind to prove it's scalable...
Don
But may I ask you another question: how far are you and Larry with your work on "scaling" from Komodo SP to MP ?
Because I hope Komodo MP will come out soon
1. I want to learn c++
2. I simulate features easier to do in c++
3. current komodo not well designed for MP.
4. current code has accumulated years of cruft.
I originally did design Komodo with MP in mind but with age there have been numerous changes that make it more difficult. Different bits and pieces of Komodo assume SP. None of the above require a re-write, I could always just take some time to clean up the code, etc ... but together there is a pretty compelling case for me.
Also, Komodo was never designed for performance, I had always intended for that the be the last stop (so to speak) but it's past due for that. I never expected Komodo to get this strong without the optimizations. I now think of the current code base as a proof of concept prototype - which is really what it was always intended to be.
A few months ago I did experiment with some simple optimizations and was able to achieve a lot of performance with just a day of work so that was pretty encouraging. I thank Richard Vida for giving me some suggestions and I implemented some of them as well as others. So I am pretty excited about getting into this - you are going to LOVE the final result!
Good luck Don with your project....
I'll keep my fingers crossed for you regards,
Dr.D
_No one can hit as hard as life.But it ain’t about how hard you can hit.It’s about how hard you can get hit and keep moving forward.How much you can take and keep moving forward….
-
- Posts: 113
- Joined: Mon Nov 28, 2011 10:17 am
Re: The scaling of Houdini 3 and Komodo 5
Hi Don!
Komodo was written in C ?
is possible implement this? ---->http://en.wikipedia.org/wiki/Cilk
excuse me my ignorance.
Komodo was written in C ?
is possible implement this? ---->http://en.wikipedia.org/wiki/Cilk
excuse me my ignorance.
-
- Posts: 5106
- Joined: Tue Apr 29, 2008 4:27 pm
Re: The scaling of Houdini 3 and Komodo 5
I was a part of the Cilk team long ago. However I want Komodo to be portable for any conceivable situation including cluster usage eventually.hammerklavier wrote:Hi Don!
Komodo was written in C ?
is possible implement this? ---->http://en.wikipedia.org/wiki/Cilk
excuse me my ignorance.
Also, Cilk has lost some features which made it especially convenient for chess. However it would still be a good choice.
Don
Capital punishment would be more effective as a preventive measure if it were administered prior to the crime.
-
- Posts: 1797
- Joined: Thu Sep 18, 2008 10:24 pm
Re: The scaling of Houdini 3 and Komodo 5
Hi Don,Don wrote:I was a part of the Cilk team long ago. However I want Komodo to be portable for any conceivable situation including cluster usage eventually.hammerklavier wrote:Hi Don!
Komodo was written in C ?
is possible implement this? ---->http://en.wikipedia.org/wiki/Cilk
excuse me my ignorance.
Also, Cilk has lost some features which made it especially convenient for chess. However it would still be a good choice.
Don
How many NPS did Cilk Chess examine? I heard it was very fast for the time. Did it have specialist hardware?