Leto wrote:
No I don't think it's the AMD system causing the issue, with Houdini 4 on my two Intel Dual Xeon machines I'm still getting barely any performance increase going from 4 cores to 8 or 12 cores. And other engines such as Stockfish and Komodo do benefit going from 4 cores to 8 on my AMD system, so it's definitely not that.
Agreed - AMD won't be the issue. Robert used to do all his development and testing on AMD in fact.
Larry K also believes that AMD actually favours Houdini vs Komodo.
Its ok...I don't claim that AMD is not good for chess
But with adapted hardwares and adapted ratings: there is one small issue, which I have difficulties to understand...
In shortly I mean...
Lets say I am member of CCRL or CEGT
And I run testings with MP engines with 4 cores, 40/4...on my i7 920 @ 4.20GHz
So please tell me,
My played MP engine games (on my i7 920's 4 core games) in which ranking should be:
-Below than AMD 6 cores / AMD 8 cores or higher than AMD ?
Lets say....I run a test on my i7 980X machine
My played MP engine games (on my i7 980X 6 core games) in which ranking should be:
-Below than AMD 6 cores / AMD 8 cores or higher than AMD ?
Leto wrote:
No I don't think it's the AMD system causing the issue, with Houdini 4 on my two Intel Dual Xeon machines I'm still getting barely any performance increase going from 4 cores to 8 or 12 cores. And other engines such as Stockfish and Komodo do benefit going from 4 cores to 8 on my AMD system, so it's definitely not that.
Agreed - AMD won't be the issue. Robert used to do all his development and testing on AMD in fact.
Larry K also believes that AMD actually favours Houdini vs Komodo.
Its ok...I don't claim that AMD is not good for chess
But with adapted hardwares and adapted ratings: there is one small issue, which I have difficulties to understand...
In shortly I mean...
Lets say I am member of CCRL or CEGT
And I run testings with Houdini 4 cores, 40/4...on my i7 920 @ 4.20GHz
So please tell me,
My played Houdini or any other MP engine games (on my i7 920's 4 core games) in which ranking should be:
-Below than AMD 6 cores or 8 cores cores or higher than AMD ?
I can't speak for CCRL but for CEGT we adapt our time control to get as close to the computational power of an Athlon 4200 as possible. We determine our time control using a Crafty benchmark. On modern hardware such as your system for the 40/4 list you'd probably test at 40/3 (CEGT has agreed this is as low as they're willing to go with that list), and for the 40/20 list you'd probably test at 40/11.
Leto wrote:
No I don't think it's the AMD system causing the issue, with Houdini 4 on my two Intel Dual Xeon machines I'm still getting barely any performance increase going from 4 cores to 8 or 12 cores. And other engines such as Stockfish and Komodo do benefit going from 4 cores to 8 on my AMD system, so it's definitely not that.
Agreed - AMD won't be the issue. Robert used to do all his development and testing on AMD in fact.
Larry K also believes that AMD actually favours Houdini vs Komodo.
Its ok...I don't claim that AMD is not good for chess
But with adapted hardwares and adapted ratings: there is one small issue, which I have difficulties to understand...
In shortly I mean...
Lets say I am member of CCRL or CEGT
And I run testings with Houdini 4 cores, 40/4...on my i7 920 @ 4.20GHz
So please tell me,
My played Houdini or any other MP engine games (on my i7 920's 4 core games) in which ranking should be:
-Below than AMD 6 cores or 8 cores cores or higher than AMD ?
I can't speak for CCRL but for CEGT we adapt our time control to get as close to the computational power of an Athlon 4200 as possible. We determine our time control using a Crafty benchmark. On modern hardware such as your system for the 40/4 list you'd probably test at 40/3 (CEGT has agreed this is as low as they're willing to go with that list), and for the 40/20 list you'd probably test at 40/11.
Good...
Both teams (CEGT and CCRL) are great rating list...and I am so thankful for their enormous efforts...
And especially CCRL and CEGT testings with one core per engine are very useful...!
But in case of testing MP engines and using different books or using different hardwares (AMD and Intel),
We can see very strange results...so this is one of main reasons about why I refused the offer about to join into both teams....
In my opinion (about the current published data)
This time CCRL ratings are correct...!
But unfortunately,
I can't say the same thing about CEGT, there is something wrong...!!
Btw,
I have no free time to check the played games, but to be honest and my 6th feeling says that,
We see such strange results...probably due to many of the testers still use not so strong opening lines!!
In my opinion (about the current published data)
This time CCRL ratings are correct...!
But unfortunately,
I can't say the same thing about CEGT, there is something wrong...!!
Btw,
I have no free time to check the played games, but to be honest and my 6th feeling says that,
We see such strange results...probably due to many of the testers still use not so strong opening lines!!
Best,
Sedat
I agree that the likely culprit for these examples is the different opening books used. I'd prefer if all testers used the same book. Critter 0.90 2CPU getting only 4 elo below the 4CPU version is very suspect considering CCRL got 80 elo difference. Something definitely wrong there.
As for Houdini 4 I just ran a quick 100 game match between the A version and the B version and it ended 50.5-49.5, so the problem is not that either. I know now that it's not the numa setting either because I only run a single instance of Houdini. I'm starting to suspect the problem is the default split depth, unfortunately if I were to change the default setting it wouldn't be considered Houdini 4 anymore by CEGT, which I'd agree with. If it's allowed to enter it'd probably have to be renamed to something like Houdini 4 x64 8CPU SD=13 or whatever turns out to be better for my machine.
Stockfish doesn't have a problem with more than 4 cores, I can run with the default settings up to 12 cores with no problems. Why should I have to change settings to make Houdini perform better with more than 4 cores?
Komodo 8 dev version beat Houdini 4 8CPU by a score of +16 =30 -4, or 31-19. That's a 38% performance for Houdini, estimated rating difference of 87 elo. At CCRL 40/40 Komodo 8 4CPU is rated 29 elo higher than Houdini 4 4CPU. Houdini 4 8CPU, if it actually was scaling correctly, should be scoring around 46% against Komodo 8CPU.
Last edited by Leto on Sun Sep 21, 2014 5:26 pm, edited 1 time in total.
We discuss origins, we discuss engines, etc...in Talkchess
but unfortunately no much discussions about opening books
For example, maybe you remember, (a few weeks ago)
The poll winner was: books don't play a big influence...
So...the conclusion is that: still the majority of people don't believe in the opening lines
But in reality is not true...the openings play a big role, any engine can be performed 0-200 Elo or even more...
And it's time for revolution for engine testers too!
Leto wrote:As for Houdini 4 I just ran a quick 100 game match between the A version and the B version and it ended 50.5-49.5, so the problem is not that either. I know now that it's not the numa setting either because I only run a single instance of Houdini. I'm starting to suspect the problem is the default split depth, unfortunately if I were to change the default setting it wouldn't be considered Houdini 4 anymore by CEGT, which I'd agree with. If it's allowed to enter it'd probably have to be renamed to something like Houdini 4 x64 8CPU SD=13 or whatever turns out to be better for my machine.
Stockfish doesn't have a problem with more than 4 cores, I can run with the default settings up to 12 cores with no problems. Why should I have to change settings to make Houdini perform better with more than 4 cores?
Actually if I was in your shoes,
I would contact Robert Houdart...I am sure he will inform you about the right settings...
Normally 2-3 years ago he was quite active...but nowadays I cant see him
I miss Robert Houdart ))
Sedat Canbaz wrote:I am glad that you see the reality !
Normally many people can not see this...!!
We discuss origins, we discuss engines, etc...in Talkchess
but unfortunately no much discussions about opening books
For example, maybe you remember, (a few weeks ago)
The poll winner was: books don't play a big influence...
So...the conclusion is that: still the majority of people don't believe in the opening lines
But in reality is not true...the openings play a big role, any engine can be performed 0-200 Elo or even more...
And it's time for revolution for engine testers too!
Rybka 3 scoring almost 50% against Stockfish 5 thanks to an opening book shocked me at that time. Now I know that engines are still not good enough in the opening phase.
