Houdini with a six point lead near the halfway point of TCEC

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
IanO
Posts: 478
Joined: Wed Mar 08, 2006 8:45 pm
Location: Portland, OR
Contact:

Re: Houdini with a six point lead near the halfway point of

Post by IanO » Wed Nov 29, 2017 9:15 pm

Amusing that Komodo made back two points since this post, including the first black victory in the match! As we should all be aware, don't count your chickens until they've hatched.

egiovannotti
Posts: 36
Joined: Wed Oct 31, 2012 8:28 am

Re: Houdini with a six point lead near the halfway point of

Post by egiovannotti » Thu Nov 30, 2017 5:33 pm

For me, the progress of Houdini goes hand in hand with the Stockfish and if the latter does not have significant progress there is very little to improve in Houdini and rather, with so little time, Houdart runs the risk of introducing a bug.

kk
Posts: 10
Joined: Wed Nov 23, 2016 7:23 am
Location: UK

Re: Houdini with a six point lead near the halfway point of

Post by kk » Thu Nov 30, 2017 7:36 pm

Not quite sure how Houdini only progresses when Stockfish progresses when Houdini is getting superior results to Stockfish? Looks to me like Stockfish has hit a wall anD development improvement has slowed somewhat.

User avatar
velmarin
Posts: 1600
Joined: Mon Feb 21, 2011 8:48 am

Re: Houdini with a six point lead near the halfway point of

Post by velmarin » Thu Nov 30, 2017 8:02 pm

egiovannotti wrote:For me, the progress of Houdini goes hand in hand with the Stockfish and if the latter does not have significant progress there is very little to improve in Houdini and rather, with so little time, Houdart runs the risk of introducing a bug.
Your comment is OK.
No one can doubt Houdart's talent,
Perhaps he has managed to introduce things of Houdini_4 into the current Houdini, together with a good testing frame.

TommyTC
Posts: 18
Joined: Thu Mar 30, 2017 6:52 am

Re: Houdini with a six point lead near the halfway point of

Post by TommyTC » Thu Nov 30, 2017 9:09 pm

syzygy wrote:
Modern Times wrote:
syzygy wrote:
velmarin wrote:Houdart is right to decline a change.
Yes, and in my view the request to replace Komodo should have been rejected without consulting Houdart.
I agree also, but I guess there was a slight chance that he would have agreed to it so they put the question.
And exactly that is why, in my view, they should not have put the question to him.

I think all agree that 23% lower nps is not a play-limiting bug "like crashing".

It is not fair to shift the blame/responsibility for not replacing Komodo to Houdart. (But luckily it seems that most people understand his refusal.)
Unfortunately some uninformed people see 23% lower nps and think it is a huge disadvantage and that Houdart is "the bad guy" for not allowing a Komodo update.

Each season there are likely to be situations where the rules don't seem to give a 100% clear direction. In those circumstances I think it is a good idea for the tournament organizer (Anton) to consult with the participants, and anyone else, to obtain opinions on how to continue.

After those discussion, he should then take 100% responsibility for the decision and should state something like, "After consulting with the participants in this tournament I have made the decision to...."

This then leaves open the possibility that all participants might agree on a solution that is disadvantageous to their own participation in the event, yet absolves them of ever having to take blame for any other decision.

Modern Times
Posts: 2417
Joined: Thu Jun 07, 2012 9:02 pm

Re: Houdini with a six point lead near the halfway point of

Post by Modern Times » Fri Dec 01, 2017 6:59 am

The 23% is also just a guess from my reading of this.

That was a rough observation by the tournament director. NPS as we know varies hugely during the phase of the game. Of course he would have been smart enough for example to look at it out of the opening for example, but even then there is variation. Mark said 8% on his 22 core machine and then assumed that it was a reasonable extrapolation to go from 8% to 23% when going from 22 to 44 cores. I don't accept that the 23% is real necessarily, only way is to run a proper benchmark outside gameplay. It may be 15% for example or even closer to the 8% Mark measured. And even if it is 23% during the opening for example it may be entirely different during the mid-game and end-game.
.

Opinions expressed here are my own, and not necessarily those of the CCRL Group.

shrapnel
Posts: 1245
Joined: Fri Nov 02, 2012 8:43 am
Location: New Delhi, India

Re: Houdini with a six point lead near the halfway point of

Post by shrapnel » Fri Dec 01, 2017 11:33 am

egiovannotti wrote:For me, the progress of Houdini goes hand in hand with the Stockfish and if the latter does not have significant progress there is very little to improve in Houdini and rather, with so little time, Houdart runs the risk of introducing a bug.
Spoken well and truly with Tongue-in-Cheek :D
Many people didn't get what you were hinting at though...
For the record, I fully agree with you.
i7 5960X @ 4.1 Ghz, 64 GB G.Skill RipJaws RAM, Twin Asus ROG Strix OC 11 GB Geforce 2080 Tis

Jouni
Posts: 2024
Joined: Wed Mar 08, 2006 7:15 pm

Re: Houdini with a six point lead near the halfway point of

Post by Jouni » Fri Dec 01, 2017 4:36 pm

Just for curiosity I played one top engine with 23% time handicap against itself. First 100 games (default engine first):

Code: Select all

+20/=70/-10 55.00%   55.0/100
+10/=70/-20 45.00%   45.0/100
and next 100

Code: Select all

+23/=62/-15 54.00%   54.0/100
+15/=62/-23 46.00%   46.0/100
In these fast games 23% can easily result with 10 wins less :o .
Jouni

Milos
Posts: 3387
Joined: Wed Nov 25, 2009 12:47 am

Re: Houdini with a six point lead near the halfway point of

Post by Milos » Fri Dec 01, 2017 5:06 pm

Jouni wrote:Just for curiosity I played one top engine with 23% time handicap against itself. First 100 games (default engine first):

Code: Select all

+20/=70/-10 55.00%   55.0/100
+10/=70/-20 45.00%   45.0/100
and next 100

Code: Select all

+23/=62/-15 54.00%   54.0/100
+15/=62/-23 46.00%   46.0/100
In these fast games 23% can easily result with 10 wins less :o .
First there is not even a remotely reliable figure regarding what is actual slow down of K 1960-1970 vs. 1959 on TCEC hardware. Talking about 23% when on 22 cores it is 8% is just ridiculous.
It is quite easy to compare Komodo in early opening phase of the games in stage 2 and now in stage 3. In stage 2 it was typically around 47Mnps and now is 40Mnps. So that is reduction of 15% at best.
Second, on TCEC conditions doubling single core nodes gives around 30-40Elo, doubling it in terms of additional cores hardly 20Elo. Question is how much of the so-called K bug is related to single core performance. The indication is quite strong that it is exclusively SMP speed loss. In that case 15% speed loss would translate into log(0.85)/log(0.5)*20 = 4.7 Elo.
That is like 0.5 points more for K in 100 games match against H.
And Robert Houdard with his 9Elo estimation was more than generous.

syzygy
Posts: 4457
Joined: Tue Feb 28, 2012 10:56 pm

Re: Houdini with a six point lead near the halfway point of

Post by syzygy » Fri Dec 01, 2017 6:46 pm

Modern Times wrote:The 23% is also just a guess from my reading of this.
Not really:
It is important to point out that the approximately 8% speed reduction we noted on our best hardware (24 cores) is apparently as high as 23% on TCEC’s 44-core machine based on Komodo’s relative nodes per second vs. Houdini in Stage 2. A difference of this magnitude is what Anton noted in his original email signalling that there might be a problem.
Anton only noted a difference "of this magnitude". The 23% number is based on the nps numbers reported by Komodo in the Superfinal and in Stage 2. It is not difficult to verify that the difference in nps is substantial.

That Komodo 1970 seemed to be about 20% slower had already been noticed by others:
http://talkchess.com/forum/viewtopic.php?t=65795

Obviously there is still a lot of room for discussion on whether the exact number is 22% or 23% or maybe 22.78%.

Post Reply