Intel vs AMD: which does your engine prefer?

[Albert Silver]

We often read that some engines favor AMD or Intel, yet I wonder why no one ever measures this precisely. All I ever see are maximum NPS, and the like, which is utterly pointless to measure the relative speed gained by one engine or another according to the processor it is running on.

I'd therefore like to propose such a test.

Running on a single core, take the start position, and run the engine on it for about one minute. Here are the results on an i5-2500K (Sandy Bridge).

Houdini 15a x64 1-core:

21/44 01:01 151,909,342 2,481,000 +0.17 e2-e4 e7-e5

Houdini 1.5a x64 (512MB hash) achieves a speed of 2,490,317 NPS average (151 Million divided by 61 seconds)

Rybka 4.1 SSE42 x64:

18 01:22 12,214,461 148,739 +0.22 e2-e4 e7-e5

Rybka 4.1SSE42 x64 reaches 148,956 NPS average.

Critter 1.2 64bit SSE4:

21/43 01:04 153,833,934 2,394,712 +0.12 d2-d4 Ng8-f6

Critter 1.2 64bit SSE4 reaches 2,403,655 NPS average.

Stockfish-211-64-ja:

25/33 01:08 106,008,326 1,558,693 +0.28 e2-e4 e7-e6

Stockfish-211-64-ja reaches 1,558,946 average

Komodo 3 x64 SSE4:

22 01:28 133,246,706 1,502,912 +0.28 d2-d4 d7-d5

Komodo 3 x64 SSE4 reaches 1,624,960 average.

Now setting Houdini at a parameter of 1000 based on its score, the other engines receive relative NPS scores.

i5-2500K

Code: Select all

Houdini 15a x64           1000
Critter 1.2 64bit SSE4    965.2
Komodo 3 x64 SSE4         652.5
Stockfish 2.1.1 x64       626.0
Rybka 4.1 SSE42 x64       59.81

If someone with a different processor, would test these engines single core, and post their one minute results (anywhere between 50+ seconds to the first iteration that appears after one minute), the results can be compared. Please be sure to identify the processor used.

Remember, the score of Houdini is merely a measurement of comparison, and will always be 1000. If Critter suddenly gets a score of 921 on a Phenom, that means that Houdini gains a special edge on that platform compared to Critter.

[IWB]

Hi there

... yet I wonder why no one ever measures this precisely...

I do!

Have a look at my web site (http://www.inwoba.de). In "Archive" you find a bench of the 20 best IPON UCI engines. Starting position, 5 min, nomalized to 1GHz and compared in %.

If you study the table for a while you will find that Komodo is running very nice on the AMD Phenoms while Naum is a desaster or vice versa (Naum good on i920 and Komodo bad).
These are just some examples. Sjeng, Spike, Hannibal are very good on AMD as well while Houdini, Naum or Onno are very good on the i7 (but not nessesarily bad on th ePhenoms ...).

This is just a basic test. You could do much more sophisticated things, but therefore you need time and the drive to do so. I would have some ideas, the problem is why ...


BTW: The outcome of this is that compared per cycle over the best 20 UCI Engines the Phenom and the Core 2 are identical while the i7 is about 14% faster. the i5m (Sandy for Notebook, half the cache!) on the other hand is a bit disapointing as it is just 6/7% faster than AMD or Core2 ...

My conclusion is, that single benchmarks, of one particular enigne tend to over- or underestimate CPUs. There is no "typical" engine!

Bye
Ingo

[ivoryknight]

...

Remember, the score of Houdini is merely a measurement of comparison, and will always be 1000. If Critter suddenly gets a score of 921 on a Phenom, that means that Houdini gains a special edge on that platform compared to Critter.

If Houdini is going to be the baseline, you, and everyone else, needs to be sure they've run the Houdini's ''autotune'' immediately after a reboot and extraneous applications have been stopped, so you can get your split depth to its most efficient setting. But you can't just run autotune. You have to run it with the same hash size you will use for the NPS test. Then when you run the test, you have to, again, be sure extra applications' processes have been stopped. (Not only do you have to do this w/ Houdini, but for the other engines, too.) Will the testers do this? Will they do it right? Do they even know how to do it?

[IWB]

Hi,

If Houdini is going to be the baseline, you, and everyone else, needs to be sure they've run the Houdini's ''autotune'' ...

Stopping here as he (and I) is running on 1 core. "autotune" is completly useless under this conditions.

Bye
Ingo

[Albert Silver]

Hi there

... yet I wonder why no one ever measures this precisely...

I do!

Have a look at my web site (http://www.inwoba.de). In "Archive" you find a bench of the 20 best IPON UCI engines. Starting position, 5 min, nomalized to 1GHz and compared in %.

If you study the table for a while you will find that Komodo is running very nice on the AMD Phenoms while Naum is a desaster or vice versa (Naum good on i920 and Komodo bad).
These are just some examples. Sjeng, Spike, Hannibal are very good on AMD as well while Houdini, Naum or Onno are very good on the i7 (but not nessesarily bad on th ePhenoms ...).

This is just a basic test. You could do much more sophisticated things, but therefore you need time and the drive to do so. I would have some ideas, the problem is why ...


BTW: The outcome of this is that compared per cycle over the best 20 UCI Engines the Phenom and the Core 2 are identical while the i7 is about 14% faster. the i5m (Sandy for Notebook, half the cache!) on the other hand is a bit disapointing as it is just 6/7% faster than AMD or Core2 ...

My conclusion is, that single benchmarks, of one particular enigne tend to over- or underestimate CPUs. There is no "typical" engine!

Bye
Ingo

I looked over your numbers, and have to say one thing intrigues me: your result of Komodo on the i7. I normalized my numbers to compare with yours, and my results are completely different on an i5. In fact, presuming your other numbers are correct, I'm surprised you don't mention that on a Phenom compared to the i7, Stockfish improves by over 11% compared to Houdini 2, which is huge. Note that all the engines seem to improve on the Phenom compared to Houdini 2 ST.

Using the same scoring as above, here is what I got, using your numbers as well:

Code: Select all

			                  PhII	   i5-2500k	i7-920
			
Houdini 2.0 STD	       1000.00	   1000.00	1000.00
Deep Rybka 4.1 SSE42	    65.79	     57.67	  61.94
Critter 1.2		         948.83	    930.72	 901.71
Komodo 3 SSE		        638.09	    629.20	 551.01
Stockfish 2.1.1 JA	     666.04	    603.64	 599.81

[IWB]

Hi Albert,

I looked over your numbers, and have to say one thing intrigues me: your result of Komodo on the i7. I normalized my numbers to compare with yours, and my results are completely different on an i5.

Keep in mind that your i5 is MUCH different than my i5 AND I am using an i7-920 which is different than your i5 as well (920 is one generation older than i5 Sandy bridge)! I am using a Notebook i5 CPU. Intel simply sells a much less efficent version of the chip under the same name for notebooks. The CPU cache on my i5 is half as big as on your i5. I noticed a difference as well, but I am not rich enoug to run ALL CPUs ...

In fact, presuming your other numbers are correct, I'm surprised you don't mention that on a Phenom compared to the i7, Stockfish improves by over 11% compared to Houdini 2, which is huge.

Here we have a different understanding of "huge"!

How much ELO is 100% Speed deifference? 50, 60, 70? Lets assume 70; 11% would be less than 8 ELO but ELO is not linear, so lets say 10 ELO. 10 ELO, considereing 2 error bars is well within my error of the IPON with approx. 2500 games each engine. Basicaly I consider 10% as nothing.

Note that all the engines seem to improve on the Phenom compared to Houdini 2 ST.

Note, that I consider the standard benchmarks as not valid for chess. My numbers are repeatable. I would say the Phenom II is an underestimated CPU - at least for chess and seeing the price (saying that I would like to see some numbers of a Buldozer. I do not expect too much, but seeing that standard benchmarks do not work for chess engines who knows ...) !
In other words: Houdini just performes under average on the AMD and still is leading with a huge margin on the IPON ...
And your statement is not right. Houdini has an index of 0.81, while the average is 0.87 (6% below average ... see abouve abouve what that might mean in ELO ...). So it is below average but enignes like Naum and Stelka 2.0 are worse.
(If programmers or compilers would take more care of the AMD CPUs (like Don does) the results for the AMDs would be even better in average!)

Using the same scoring as above, here is what I got, using your numbers as well:

Code: Select all

			                  PhII	   i5-2500k	i7-920
			
Houdini 2.0 STD	       1000.00	   1000.00	1000.00
Deep Rybka 4.1 SSE42	    65.79	     57.67	  61.94
Critter 1.2		         948.83	    930.72	 901.71
Komodo 3 SSE		        638.09	    629.20	 551.01
Stockfish 2.1.1 JA	     666.04	    603.64	 599.81

I dont like your "1000" approach. Mathematicaly you can express that as 100% as I did in my bench sheet. Nonetheless I will have a closer look at it ... tomorrow!

Bye
Ingo

[Albert Silver]

Hi Albert,

Using the same scoring as above, here is what I got, using your numbers as well:

Code: Select all

			                  PhII	   i5-2500k	i7-920
			
Houdini 2.0 STD	       1000.00	   1000.00	1000.00
Deep Rybka 4.1 SSE42	    65.79	     57.67	  61.94
Critter 1.2		         948.83	    930.72	 901.71
Komodo 3 SSE		        638.09	    629.20	 551.01
Stockfish 2.1.1 JA	     666.04	    603.64	 599.81

I dont like your "1000" approach. Mathematicaly you can express that as 100% as I did in my bench sheet. Nonetheless I will have a closer look at it ... tomorrow!

Bye
Ingo

Actually you cannot express it as 100%, because it is misleading. Suppose you say Houdini gets 100%, does that mean Rybka is scoring 6%? And 6% of what? As I explained, the absolute scores are utterly meaningless. What matters are the relative scores compared to other processors.

In other words, on your i7 and Phenom II, using Houdini as a standard parameter, Stockfish improves its relative NPS performance on a phenom II by over 11%. Also, I have trouble believing that Komodo, on a single core, thus no hyperthreading, performs 20% worse on an i7 compared to an i5. Remember, we are not talking absolute performance. We are saying that if the speeds are normalized, it will perform 20% worse on an i7, compared to an i5, which again, is hard to believe.

[ivoryknight]

Hi,

If Houdini is going to be the baseline, you, and everyone else, needs to be sure they've run the Houdini's ''autotune'' ...

Stopping here as he (and I) is running on 1 core. "autotune" is completly useless under this conditions.

Bye
Ingo

You are correct. My brain shorted out there, didn't it!

However, the other things I mentioned still apply. Also, other factors to consider are the operating system used and the speed of the RAM. You cannot compare system to system. You need *many* people to submit their results for each processor type, and then you can see the peaks, lows, and averages.

[IWB]

...
In other words, on your i7 and Phenom II, using Houdini as a standard parameter, Stockfish improves its relative NPS performance on a phenom II by over 11%. Also, I have trouble believing that Komodo, on a single core, thus no hyperthreading, performs 20% worse on an i7 compared to an i5. Remember, we are not talking absolute performance. We are saying that if the speeds are normalized, it will perform 20% worse on an i7, compared to an i5, which again, is hard to believe.

We are not talking about god, there is nothing to believe! Have a look at the Naum - Komodo pairing. The difference between phenom and core 2 is 42% and you are complaining about 20%

It is a fact - regardless what one belive and again - if programmers and compilers would take more care about the AMDs they (the AMds) might perform even better ...

BYe
INgo

[IWB]

However, the other things I mentioned still apply. Also, other factors to consider are the operating system used and the speed of the RAM. You cannot compare system to system. You need *many* people to submit their results for each processor type, and then you can see the peaks, lows, and averages.

In general you are right, a clean system freshly booted will give the best performance. what I doubt is the "many people" aproach, or better the "few people" apreoach. If you have 500 people, I am fine with your statement, if you have 5 I dounbt that this would be of any use compared to one single trustfull person ...

Nonetheless, we agree in basics understanding I think.

Bye
Ingo