Some Notes about Hyper-Threading

[Sedat Canbaz]

Oh,sorry...Friends,i forgot to share the games played by Houdini 2.0c Pro x64 12t:
http://www.multiupload.com/BPH7HCX1GX

BTW, Houdini 2.0c Pro x64 12t's next opponent:Critter 1.2 x64 4c

Best,
Sedat

[bob]

I don't see much that is useful there. Knps is not a useful measure. Time to depth is the key, as always.

[Sedat Canbaz]

I don't see much that is useful there. Knps is not a useful measure. Time to depth is the key, as always.

As far as i know,there is no a such useful chess benchmarking tool-'Time to depth',which is working on all latest modern hardwares ?!

Or maybe i am missing something ??

Actually i need a very good indicator,which will work accurately under all hardwares (i mean e.g up to 12 Cores,16 Cores,32 Cores...)

For example,the current my Houdini 2.0 benchmark list is a quite good indicator about which are the fastest processors for computers chess
http://www.sedatcanbaz.com/chess/houdin ... enchmarks/

But unfortunately,not all people have this commercial top engine
Plus the current benchmarking method seems to be not so easy for some chess friends
Sometimes,even my tutorial does not help too

One thing more,honestly i like your Crafty bench program,but unfortunately it does not work properly on machines with more than 8 cores
As you know, i have created a successful Crafty benchmark list:
http://sedatchess.110mb.com/index.php?p=1_12

BTW,Axon bench was another great bench tool,but what a pity it does not work on the latest systems
http://sedatchess.110mb.com/index.php?p=1_14


Best,
Sedat

[bob]

Dear Robert,

Btw,do you plan to release a new a well-optimized Crafty version which to support many cores

For example,the current available Crafty versions are up to 8 cores,thats why i paused Crafty 22.8 Benchmarks,due to it supports up to 8 cores

And as far as i know,there are some Crafty compilers,which support more than 8 cores,but unfortunately,theirs MP scaling are not very good...
I mean the chess benchmarks by Crafty are not performing quite good at 12 CPUs or higher CPUs...

In other words,its will be great if you release a new Crafty version which to support to many cores and later maybe i can resume my benchmarks with your great engine !

Best,
Sedat

Crafty supports any number of cores. It is up to the person that compiles it as to what limit they want to use. I run on ICC all the time using 12 cores, and have tested up to 64 cores... There's no limit within the software, just the compile-time option "CPUS=n". Probably everyone should use N of at least 16 today, at a minimum...

[bob]

Hello Dear Friends,

Honestly,i am surprised and impressed by the latest results of Houdini 2.0c Pro x64 12 Threads (HT ON)

Of course,for a better conclusion more games are needed...
But however,looking at the current results,still i don't expect the ELO performance to be in favor for HT Enabled
Some Notes:
-Since December 2011,SCCT Auto232 Participants are started to use 512 MB Hashtable size
-Houdini 2.0c Pro x64 1c performed approx.10 ELO better than Houdini 2.0 Pro x64 1c
-Houdini 2.0 Pro x64 4c performed approx.10 ELO better than Houdini 2.0b Pro x64 4c
In other words,Houdini 2.0c Pro x64 6c's expected ELO performance to be min 15 ELO stronger than Houdini 2.0b Pro x64 6c
And that means (as i mentioned before) Houdini 2.0c Pro x64 6c is expected to be approx. 10 ELO stronger than Houdini 2.0c Pro x64 12t


Let’s see, what will the HT ELO performance after more games…

Code: Select all

Rank Name                        Elo    +    - games score oppo. draws 
   1 Houdini 2.0c Pro x64 12t   3424   34   34   254   65%  3325   41% 
   2 Houdini 2.0b Pro x64 6c    3419   17   17  1009   69%  3300   46% 
   3 Houdini 2.0 Pro x64 4c     3362   16   16  1131   61%  3293   49% 
   4 Deep Rybka 4.1 x64 6c      3359   18   18   848   61%  3294   58% 
   5 Houdini 2.0b Pro x64 4c    3351   17   17   991   51%  3343   47% 
   6 Houdini 1.5a x64 4c        3344   16   16  1086   62%  3272   48% 
   7 Deep Rybka 4.1 x64 4c      3293   13   13  1603   47%  3314   56% 
   8 Critter 1.2 x64 4c         3288   16   16  1037   50%  3290   58% 
   9 IvanHoe 47c GH x64 4c      3281   15   15  1109   49%  3288   60% 
  10 Fire 2.2 xTreme x64 4c     3275   15   15  1165   41%  3328   59% 
  11 IvanHoe 0B.09.18 x64 4c    3270   16   16  1008   46%  3294   57% 
  12 DeepSaros 2.3i x64 4c      3269   16   16   982   48%  3279   60% 
  13 IvanHoe B47d x64 4c        3266   17   17   967   43%  3309   56% 
  14 Houdini 2.0c Pro x64 1c    3262   20   20   711   61%  3193   47% 
  15 IvanHoe B47f02 x64 4c      3260   16   16  1004   48%  3274   59% 
  16 Stockfish 2.1.1 JA x64 4c  3255   16   16  1012   46%  3275   55% 
  17 Houdini 2.0 Pro x64 1c     3253   14   13  1575   54%  3227   48% 
  18 Strelka 5.1 x64 1c         3243   17   17   921   56%  3210   53% 
  19 Rybka 4.1 x64 1c           3193   17   17   888   48%  3204   53% 
  20 Ivanhoe B46fa x64 1c       3190   27   27   359   46%  3214   57% 
  21 Komodo 3.0 x64 1c          3188   14   14  1563   40%  3248   45% 
  22 Ivanhoe B50kBf x64 1c      3186   20   20   637   47%  3206   58% 
  23 Ivanhoe B46a x64 1c        3174   17   17   949   43%  3213   57% 
  24 Naum 4.2 x64 4c            3174   18   18   898   36%  3258   46% 
  25 Stockfish 111026 x64 1c    3173   18   18   861   44%  3208   49% 

Individual statistics:

1 Houdini 2.0c Pro x64 12t  : 3424  254 (+114,=104,- 36), 65.4 %

Deep Rybka 4.1 x64 4c         : 114 (+ 61,= 41,- 12), 71.5 %
Houdini 2.0b Pro x64 4c       : 140 (+ 53,= 63,- 24), 60.4 %

For SCCT Auto232 Conditions:
http://www.sedatcanbaz.com/chess/rating ... onditions/



Greetings,
Sedat

You are way inside the "noise window" here. You need tens of thousands of games, not a couple of hundred. This is likely the result of simple SMP non-determinism...

[Sedat Canbaz]

Dear Robert,

Btw,do you plan to release a new a well-optimized Crafty version which to support many cores

For example,the current available Crafty versions are up to 8 cores,thats why i paused Crafty 22.8 Benchmarks,due to it supports up to 8 cores

And as far as i know,there are some Crafty compilers,which support more than 8 cores,but unfortunately,theirs MP scaling are not very good...
I mean the chess benchmarks by Crafty are not performing quite good at 12 CPUs or higher CPUs...

In other words,its will be great if you release a new Crafty version which to support to many cores and later maybe i can resume my benchmarks with your great engine !

Best,
Sedat

Crafty supports any number of cores. It is up to the person that compiles it as to what limit they want to use. I run on ICC all the time using 12 cores, and have tested up to 64 cores... There's no limit within the software, just the compile-time option "CPUS=n". Probably everyone should use N of at least 16 today, at a minimum...

As i mentioned before, i have Crafty compilers,which support more than 8 cores,but i noticed that Crafty's MP benchmarks (as far as i remember with 12 cores)did not not perform quite well...there were misunderstanding benchmark results...

Actually Crafty 22.8 up to 8 cores is a great benchmarking tool...
But with more cores,i think it needs update-optimizing


Best,
Sedat

[Sedat Canbaz]

Hello Dear Friends,

Honestly,i am surprised and impressed by the latest results of Houdini 2.0c Pro x64 12 Threads (HT ON)

Of course,for a better conclusion more games are needed...
But however,looking at the current results,still i don't expect the ELO performance to be in favor for HT Enabled
Some Notes:
-Since December 2011,SCCT Auto232 Participants are started to use 512 MB Hashtable size
-Houdini 2.0c Pro x64 1c performed approx.10 ELO better than Houdini 2.0 Pro x64 1c
-Houdini 2.0 Pro x64 4c performed approx.10 ELO better than Houdini 2.0b Pro x64 4c
In other words,Houdini 2.0c Pro x64 6c's expected ELO performance to be min 15 ELO stronger than Houdini 2.0b Pro x64 6c
And that means (as i mentioned before) Houdini 2.0c Pro x64 6c is expected to be approx. 10 ELO stronger than Houdini 2.0c Pro x64 12t


Let’s see, what will the HT ELO performance after more games…

Code: Select all

Rank Name                        Elo    +    - games score oppo. draws 
   1 Houdini 2.0c Pro x64 12t   3424   34   34   254   65%  3325   41% 
   2 Houdini 2.0b Pro x64 6c    3419   17   17  1009   69%  3300   46% 
   3 Houdini 2.0 Pro x64 4c     3362   16   16  1131   61%  3293   49% 
   4 Deep Rybka 4.1 x64 6c      3359   18   18   848   61%  3294   58% 
   5 Houdini 2.0b Pro x64 4c    3351   17   17   991   51%  3343   47% 
   6 Houdini 1.5a x64 4c        3344   16   16  1086   62%  3272   48% 
   7 Deep Rybka 4.1 x64 4c      3293   13   13  1603   47%  3314   56% 
   8 Critter 1.2 x64 4c         3288   16   16  1037   50%  3290   58% 
   9 IvanHoe 47c GH x64 4c      3281   15   15  1109   49%  3288   60% 
  10 Fire 2.2 xTreme x64 4c     3275   15   15  1165   41%  3328   59% 
  11 IvanHoe 0B.09.18 x64 4c    3270   16   16  1008   46%  3294   57% 
  12 DeepSaros 2.3i x64 4c      3269   16   16   982   48%  3279   60% 
  13 IvanHoe B47d x64 4c        3266   17   17   967   43%  3309   56% 
  14 Houdini 2.0c Pro x64 1c    3262   20   20   711   61%  3193   47% 
  15 IvanHoe B47f02 x64 4c      3260   16   16  1004   48%  3274   59% 
  16 Stockfish 2.1.1 JA x64 4c  3255   16   16  1012   46%  3275   55% 
  17 Houdini 2.0 Pro x64 1c     3253   14   13  1575   54%  3227   48% 
  18 Strelka 5.1 x64 1c         3243   17   17   921   56%  3210   53% 
  19 Rybka 4.1 x64 1c           3193   17   17   888   48%  3204   53% 
  20 Ivanhoe B46fa x64 1c       3190   27   27   359   46%  3214   57% 
  21 Komodo 3.0 x64 1c          3188   14   14  1563   40%  3248   45% 
  22 Ivanhoe B50kBf x64 1c      3186   20   20   637   47%  3206   58% 
  23 Ivanhoe B46a x64 1c        3174   17   17   949   43%  3213   57% 
  24 Naum 4.2 x64 4c            3174   18   18   898   36%  3258   46% 
  25 Stockfish 111026 x64 1c    3173   18   18   861   44%  3208   49% 

Individual statistics:

1 Houdini 2.0c Pro x64 12t  : 3424  254 (+114,=104,- 36), 65.4 %

Deep Rybka 4.1 x64 4c         : 114 (+ 61,= 41,- 12), 71.5 %
Houdini 2.0b Pro x64 4c       : 140 (+ 53,= 63,- 24), 60.4 %

For SCCT Auto232 Conditions:
http://www.sedatcanbaz.com/chess/rating ... onditions/



Greetings,
Sedat

You are way inside the "noise window" here. You need tens of thousands of games, not a couple of hundred. This is likely the result of simple SMP non-determinism...

Sure...more games better !

Note also the Engine's ELO difference depends of what kind of book usage

I mean,there are some opening books,where you need to run min 2000-3000 games (for accurate rating)

From my experience i can say (in case of using Perfect 2012a book) 500 games will be enough,you don't need to run thousands of games

And during my book testings,there was a very small ELO difference (approx. 5 or 10 ELO) between 400-500 games and 2000-3000 games

Best,
Sedat

[Sedat Canbaz]

Here are available some SCCT Auto232 games,where the engines are played 300-500 games:
http://sedat-chess.110mb.com/index.php?p=1_8

Note also that the latest version:Perfect 2012a book is more powerful and stronger than the previous Perfect 2011 book
In other words,it plays more balanced openings for both sides...

And please compare the engines,which are played more than 1000 games:
http://www.sedatcanbaz.com/chess/ratings/scct-auto232/

As we see the ELO difference is approx. 5-10 ELO !!

Regards,
Sedat

[bob]

Hello Dear Friends,

Honestly,i am surprised and impressed by the latest results of Houdini 2.0c Pro x64 12 Threads (HT ON)

Of course,for a better conclusion more games are needed...
But however,looking at the current results,still i don't expect the ELO performance to be in favor for HT Enabled
Some Notes:
-Since December 2011,SCCT Auto232 Participants are started to use 512 MB Hashtable size
-Houdini 2.0c Pro x64 1c performed approx.10 ELO better than Houdini 2.0 Pro x64 1c
-Houdini 2.0 Pro x64 4c performed approx.10 ELO better than Houdini 2.0b Pro x64 4c
In other words,Houdini 2.0c Pro x64 6c's expected ELO performance to be min 15 ELO stronger than Houdini 2.0b Pro x64 6c
And that means (as i mentioned before) Houdini 2.0c Pro x64 6c is expected to be approx. 10 ELO stronger than Houdini 2.0c Pro x64 12t


Let’s see, what will the HT ELO performance after more games…

Code: Select all

Rank Name                        Elo    +    - games score oppo. draws 
   1 Houdini 2.0c Pro x64 12t   3424   34   34   254   65%  3325   41% 
   2 Houdini 2.0b Pro x64 6c    3419   17   17  1009   69%  3300   46% 
   3 Houdini 2.0 Pro x64 4c     3362   16   16  1131   61%  3293   49% 
   4 Deep Rybka 4.1 x64 6c      3359   18   18   848   61%  3294   58% 
   5 Houdini 2.0b Pro x64 4c    3351   17   17   991   51%  3343   47% 
   6 Houdini 1.5a x64 4c        3344   16   16  1086   62%  3272   48% 
   7 Deep Rybka 4.1 x64 4c      3293   13   13  1603   47%  3314   56% 
   8 Critter 1.2 x64 4c         3288   16   16  1037   50%  3290   58% 
   9 IvanHoe 47c GH x64 4c      3281   15   15  1109   49%  3288   60% 
  10 Fire 2.2 xTreme x64 4c     3275   15   15  1165   41%  3328   59% 
  11 IvanHoe 0B.09.18 x64 4c    3270   16   16  1008   46%  3294   57% 
  12 DeepSaros 2.3i x64 4c      3269   16   16   982   48%  3279   60% 
  13 IvanHoe B47d x64 4c        3266   17   17   967   43%  3309   56% 
  14 Houdini 2.0c Pro x64 1c    3262   20   20   711   61%  3193   47% 
  15 IvanHoe B47f02 x64 4c      3260   16   16  1004   48%  3274   59% 
  16 Stockfish 2.1.1 JA x64 4c  3255   16   16  1012   46%  3275   55% 
  17 Houdini 2.0 Pro x64 1c     3253   14   13  1575   54%  3227   48% 
  18 Strelka 5.1 x64 1c         3243   17   17   921   56%  3210   53% 
  19 Rybka 4.1 x64 1c           3193   17   17   888   48%  3204   53% 
  20 Ivanhoe B46fa x64 1c       3190   27   27   359   46%  3214   57% 
  21 Komodo 3.0 x64 1c          3188   14   14  1563   40%  3248   45% 
  22 Ivanhoe B50kBf x64 1c      3186   20   20   637   47%  3206   58% 
  23 Ivanhoe B46a x64 1c        3174   17   17   949   43%  3213   57% 
  24 Naum 4.2 x64 4c            3174   18   18   898   36%  3258   46% 
  25 Stockfish 111026 x64 1c    3173   18   18   861   44%  3208   49% 

Individual statistics:

1 Houdini 2.0c Pro x64 12t  : 3424  254 (+114,=104,- 36), 65.4 %

Deep Rybka 4.1 x64 4c         : 114 (+ 61,= 41,- 12), 71.5 %
Houdini 2.0b Pro x64 4c       : 140 (+ 53,= 63,- 24), 60.4 %

For SCCT Auto232 Conditions:
http://www.sedatcanbaz.com/chess/rating ... onditions/



Greetings,
Sedat

You are way inside the "noise window" here. You need tens of thousands of games, not a couple of hundred. This is likely the result of simple SMP non-determinism...

Sure...more games better !

Note also the Engine's ELO difference depends of what kind of book usage

I mean,there are some opening books,where you need to run min 2000-3000 games (for accurate rating)

From my experience i can say (in case of using Perfect 2012a book) 500 games will be enough,you don't need to run thousands of games

And during my book testings,there was a very small ELO difference (approx. 5 or 10 ELO) between 400-500 games and 2000-3000 games

Best,
Sedat

Here's the problem. When you compare two programs that are VERY close in Elo, 400-500 games will NEVER be enough to find which is stronger. To get down to even the +/-4 Elo range requires 30,000 games. With SMP, the standard deviation is even larger because of the way the search behaves...

[bob]

Here are available some SCCT Auto232 games,where the engines are played 300-500 games:
http://sedat-chess.110mb.com/index.php?p=1_8

Note also that the latest version:Perfect 2012a book is more powerful and stronger than the previous Perfect 2011 book
In other words,it plays more balanced openings for both sides...

And please compare the engines,which are played more than 1000 games:
http://www.sedatcanbaz.com/chess/ratings/scct-auto232/

As we see the ELO difference is approx. 5-10 ELO !!

Regards,
Sedat

How can you say 5-10 Elo when the error bar is larger than that???