Houdini with a six point lead near the halfway point of TCEC

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Nordlandia
Posts: 2477
Joined: Fri Sep 25, 2015 7:38 pm
Location: Sortland, Norway

Re: Houdini with a six point lead near the halfway point of

Post by Nordlandia » Fri Dec 01, 2017 7:24 pm

Komodo is partially crippled if speed is 22% less than usual.

Modern Times
Posts: 2420
Joined: Thu Jun 07, 2012 9:02 pm

Re: Houdini with a six point lead near the halfway point of

Post by Modern Times » Fri Dec 01, 2017 7:46 pm

syzygy wrote: Obviously there is still a lot of room for discussion on whether the exact number is 22% or 23% or maybe 22.78%.
Or even 8% as Mark calculated on his own 22-core machine. General observations aren't enough. The only way is after the tournament to run the exe on some identical positions on infinite analysis and compare, and perhaps run the internal benchmark as well. Until that is done the 23% number is pure speculation.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: Houdini with a six point lead near the halfway point of

Post by syzygy » Fri Dec 01, 2017 8:55 pm

Modern Times wrote:
syzygy wrote: Obviously there is still a lot of room for discussion on whether the exact number is 22% or 23% or maybe 22.78%.
Or even 8% as Mark calculated on his own 22-core machine. General observations aren't enough. The only way is after the tournament to run the exe on some identical positions on infinite analysis and compare, and perhaps run the internal benchmark as well. Until that is done the 23% number is pure speculation.
If you care much about 22.78% versus 23%, then yes, it is pure speculation as you say.

I think most people will happily accept that there is a substantial slowdown of around 23% by simply inspecting the nps numbers reported by Komodo over a series of positions in a number of games.

But feel free to attack Mark by arguing that he cannot know that it is 23% and not 22.78% :roll: :roll: :roll:

Modern Times
Posts: 2420
Joined: Thu Jun 07, 2012 9:02 pm

Re: Houdini with a six point lead near the halfway point of

Post by Modern Times » Sat Dec 02, 2017 8:04 am

No attacks on Mark, to the contrary his 8% number is the only verifiable one at the moment.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

User avatar
Graham Banks
Posts: 33251
Joined: Sun Feb 26, 2006 9:52 am
Location: Auckland, NZ

Re: Houdini with a six point lead near the halfway point of

Post by Graham Banks » Sat Dec 02, 2017 8:59 am

From games 25-67, the scores have been level.
Those five unanswered losses in the first 24 games really hurt Komodo.
My email addresses:
gbanksnz at gmail.com
gbanksnz at yahoo.co.nz

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: Houdini with a six point lead near the halfway point of

Post by syzygy » Sat Dec 02, 2017 10:49 am

Modern Times wrote:No attacks on Mark, to the contrary his 8% number is the only verifiable one at the moment.
Do you realise the 23% number comes from him?

And do you understand how he got to that number? (Hint: not by "extrapolating" the 8% number.)

Isaac
Posts: 265
Joined: Sat Feb 22, 2014 7:37 pm

Re: Houdini with a six point lead near the halfway point of

Post by Isaac » Sat Dec 02, 2017 1:27 pm

Why are people comparing the nps of Komodo in stage 2 (different engine), with Komodo of another stage?
SF dev versions have very different nps from a version to the next. After applying several many patches you cannot expect the nps to roughly stay the same. I am not saying it is meaningless to do so, but it isn't meaningul either. It cannot give much information. What count is elo/strength, not raw nps.
If the bug is only a slowdown (not a functional change), then the correct way to estimate the slowdown is to let the 2 engines, normal and bugged, on many different positions and report the average nps or something like that. Just like the bench command of stockfish and other engines. Comparing the nps between 2 differently functional engines doesn't give as much information as I think people are attributing.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: Houdini with a six point lead near the halfway point of

Post by syzygy » Sat Dec 02, 2017 1:36 pm

Isaac wrote:Why are people comparing the nps of Komodo in stage 2 (different engine), with Komodo of another stage?
If an internal change in Komodo had explained the nps difference, we trust that Mark would have said so.

Here we have a case where there is probably no slowdown on 1 core, an 8% slowdown with 24 cores and a 23% slowdown with 43 cores. That cannot be explained by a regular change in the eval or search code. Of course it is possible that Mark overhauled the threading code, but it is not very likely.
SF dev versions have very different nps from a version to the next. After applying several many patches you cannot expect the nps to roughly stay the same.
SF's nps roughly does stay the same over many patches.
I am not saying it is meaningless to do so, but it isn't meaningul either.
If Mark thinks it is meaningful, who are we to judge differently?

Isaac
Posts: 265
Joined: Sat Feb 22, 2014 7:37 pm

Re: Houdini with a six point lead near the halfway point of

Post by Isaac » Sat Dec 02, 2017 2:08 pm

syzygy wrote:SF's nps roughly does stay the same over many patches.
Ok thanks, I didn't know this. I thought there was a great variability between each dev versions, even though the average nps on the bench positions stays around a mean that doesn't change much with time anymore.

syzygy
Posts: 4458
Joined: Tue Feb 28, 2012 10:56 pm

Re: Houdini with a six point lead near the halfway point of

Post by syzygy » Sat Dec 02, 2017 4:24 pm

Isaac wrote:
syzygy wrote:SF's nps roughly does stay the same over many patches.
Ok thanks, I didn't know this. I thought there was a great variability between each dev versions, even though the average nps on the bench positions stays around a mean that doesn't change much with time anymore.
The bench number itself (total nodes) varies a lot from patch to patch in a completely unpredictable manner.

Post Reply