ICCR project is planning to be canceled

[Sedat Canbaz]

Dear Chess Friends,

Unfortunately i am sorry to announce:
-I am planning to cancel my new project ICCR

More details about why i am planning to stop working on the new ICCR project:
-I still strongly believe (in case of such list) that we will see high-quality games,played under super fast machines
-I plan to cancel this project is not due to there is no interest over it...
*For example,even in a very short time,ICCR has already 3 Testers,which are waiting instructions from me

The main reason of planning to not continue working over ICCR is that:
-There will be wrong Elo calculations,in case of using adapted time controls under different speed processors
-I mean,in case of creating a such adapted rating- all participant Elo standings will be effected...
-So far...I could not find any right solution for measuring the real Elo strength of the ICCR engines
-The biggest Elo calculation problem is appearing e.g 6c against 4c or 6c against 12c ...

*A simple example of wrong Elo calculation about in case of if we are not combing the mp engines in one version:
How can we calculate the game results played by:

Code: Select all

Intel Core i7 920       @ 4.00 GHz    4 core   12454   50m+10s
and
AMD Phenom II X6 1035T    2.60 GHz    6 core   9283    70m+10s

-The right hardware Elo calculation (6c against 4c):should be done on two separate machines via Auto232 mode
*I mean for hardware speed test (6c against 4c):
-Engines should be tested with adapted time controls,on two separate machines via Auto232 mode
-Even if we combine ICCR played game results by Quads/Six-Cores... in one chess engine version
-Then is appearing another Elo calculation problem:as Clemens Keck stated...some buggy mp engines will be effected from that
In my opinion,combining in one chess engine version is the right way,but only for NON-buggy MP chess engines
-But in case of combining all mp participants,then it will lead to another wrong Elo calculation results

A little note more:
-For the first time in my Computerchess life, i could not complete a project with a success

And after all: I am sorry dear Chess Friends for all that i can not continue to work over this new project

But i have good news:
-I plan to start a new Rating list (15m+10s) based on only i7 Six-core machines
Note:more info coming soon...

BTW,i stopped the current hardware Elo speed test and the results are here:

Games:
http://www.sedatcanbaz.com/chess/games/ ... o_Test.rar


Best Wishes,
Sedat Canbaz

[hgm]

I am not sure what exactly you are trying to explain, but it would indeed be wrong to compensate a different number of cores by thinking time or clock speed. Different engines profit in a different way from extra cores. So there is o fixed relation between the playing strength of the same engine using 4 cores and using 6 cores. They are independent entities, and should be rated separately.

Clock speed can be reasonably well compensated by thinking time, though. If you play through Auto-232 between different machines, such compensation can only work with ponder off, though, and even then it would be risky (e.g. because memory speed might not scale the same way as CPU speed, and some engines could be more sensitive to memory speed than others).

[Sven]

Why not simply have two separate rating lists, 4-core and 6-core?

Sven

[Sedat Canbaz]

Why not simply have two separate rating lists, 4-core and 6-core?

Sven

I will try to explain you in this way,

For example,currently i have 4 processors for chess:
i7 980X 6 core
i7 970 6 core
i7 920 4 core
QX9650 4 core


And in case of creating separate rating lists,4-core and 6-core:
1)i think its pointless-i dont want to spend my cpu time
2)Instead testing same chess engine on my quads,i will prefer to test them on much faster processors: i7 980X/i7 970
3)ICCR will have less games played with slow time controls
4)Plus adapted time controls+different speed machines dont give us accurate Elo rating

Best,
Sedat

[Sedat Canbaz]

I am not sure what exactly you are trying to explain, but it would indeed be wrong to compensate a different number of cores by thinking time or clock speed. Different engines profit in a different way from extra cores. So there is o fixed relation between the playing strength of the same engine using 4 cores and using 6 cores. They are independent entities, and should be rated separately.

Clock speed can be reasonably well compensated by thinking time, though. If you play through Auto-232 between different machines, such compensation can only work with ponder off, though, and even then it would be risky (e.g. because memory speed might not scale the same way as CPU speed, and some engines could be more sensitive to memory speed than others).

Actually i am trying to explain this:
-Rating list,based on adapted time controls+different speed machines dont give us accurate Elo rating

And i strongly believe in that:
-Rating list,based on same hardware speed and same time control is ideal for measuring the real Elo strength of the engines

Regards,
Sedat

[Adam Hair]

I am not sure what exactly you are trying to explain, but it would indeed be wrong to compensate a different number of cores by thinking time or clock speed. Different engines profit in a different way from extra cores. So there is o fixed relation between the playing strength of the same engine using 4 cores and using 6 cores. They are independent entities, and should be rated separately.

Clock speed can be reasonably well compensated by thinking time, though. If you play through Auto-232 between different machines, such compensation can only work with ponder off, though, and even then it would be risky (e.g. because memory speed might not scale the same way as CPU speed, and some engines could be more sensitive to memory speed than others).

Actually i am trying to explain this:
-Rating list,based on adapted time controls+different speed machines dont give us accurate Elo rating

And i strongly believe in that:
-Rating list,based on same hardware speed and same time control is ideal for measuring the real Elo strength of the engines

Regards,
Sedat

Sedat,

I think you are putting a little too much emphasis the necessity on having identical hardware speed and identical time control. It would give a more precise measurement of Elo. However, that measurement would be specific to the reference computer system. Certain engines may be stronger for that particular configuration relative to other engines, or vice-versa. That prevents a truly accurate measurement of strength, which ideally would be in reference to a randomly chosen computer.

How much precision and accuracy is lost by using various computers using different time controls? Compare IPON and CCRL 40/4. Ingo (I believe) uses identical computers. The CCRL 40/4 is generated by various computers with different time controls. When confounding factors are accounted for, the two lists are surprisingly similar. There is variance in the actual computed Elos, no doubt due in part to the varying conditions. However, you and your fellow testers would be hard-pressed to play enough games where these sort of errors would really stand out. Certainly not enough to invalidate the information produced by the quality of the games you suggested playing.

One thing I would suggest, if you change your mind, is to run either 6 core or 4 core, but not both.

Adam

[Sedat Canbaz]

Hello Adam,


See bellow please (postings on CSS forum)

Clemens Keck schrieb:

Hi Sedat

other tester groups also have 2, 4, 6 cpu in their list.
How do they do that? Could you not learn from them a good system?

Regards, Clemens

Dear Clemens,

Probably you mean about CEGT/CCRL ?!

First of all,i'd like to say that i have big respect to their works-CEGT/CCRL

And the both great teams are putting a lot free efforts-for this i am so thankful for that...

But anyway,i think the both rating list (CEGT/CCRL) include misunderstanding results,especially i mean for 6CPU and 4CPU
Or maybe 4CPUs and 2CPU or 1CPU too

For example,as far as i know,they are owner of AMD Phenom II X6

And their testings with 6CPU against 4 CPU on same hardware
In my opinion a such test is wrong-it will not give us right Elo performance

BTW,strange results indeed,e.g CEGT has 15 ELO difference:
http://www.husvankempen.de/nunn/40_40%2 ... liste.html

1 Houdini 1.5a x64 6CPU 3290 23 23 600 73.0% 3117 36.0%
2 Houdini 1.5a x64 4CPU 3275 11 11 2641 73.0% 3102 34.9%

CCRL Rating has 45 ELO difference:
http://computerchess.org.uk/ccrl/404/ra ... t_all.html

Houdini 2.0c 64-bit 6CPU 3404
Houdini 2.0 64-bit 4CPU 3359

And i strongly believe in reality,AMD Phenom II X6 is not stronger than Intel Quads
Or maybe there will be 5-10 Elo difference-no more no less (it depend on the clock speed)

Honestly this is one of the main reason of canceling my new project-ICCR

Once more i'd like to thank you for your useful note

Greetings,
Sedat

[rbarreira]

Sedat,

I think you are putting a little too much emphasis the necessity on having identical hardware speed and identical time control. It would give a more precise measurement of Elo. However, that measurement would be specific to the reference computer system. Certain engines may be stronger for that particular configuration relative to other engines, or vice-versa. That prevents a truly accurate measurement of strength, which ideally would be in reference to a randomly chosen computer.

+1

Precision does not equal relevance or usefulness. An elo valued averaged over different hardware configurations is less precise, but may be more useful anyway.

[Sedat Canbaz]

Sedat,

I think you are putting a little too much emphasis the necessity on having identical hardware speed and identical time control. It would give a more precise measurement of Elo. However, that measurement would be specific to the reference computer system. Certain engines may be stronger for that particular configuration relative to other engines, or vice-versa. That prevents a truly accurate measurement of strength, which ideally would be in reference to a randomly chosen computer.

+1

Precision does not equal relevance or usefulness. An elo valued averaged over different hardware configurations is less precise, but may be more useful anyway.

Just to make it more clear...

It seems there are some people who really missed to read some of my notes:
http://www.talkchess.com/forum/viewtopic.php?t=41797

And i am asking again to all experienced Chess Experts:

How can we calculate the game results played by:

Intel Core i7 920 @ 4.00 GHz 4 core 12454 50m+10s
and
AMD Phenom II X6 1035T 2.60 GHz 6 core 9283 70m+10s

*In case of such test:the expected ELO performance to be approx.same (almost identical)

So,in case of such games...how we will calculate/publish the results ?

I say again:
AMD Phenom II X6 is not 40-50 ELO stronger than Intel Quads
*The both processors have almost same chess speed and the both Elo standings should be same

For example:
i7 980X @4.33GHz is approx.70 Elo stronger than i7 920 @3.0GHz
Note:i7 980X @4.33GHz is approx. two times faster than i7 920 @3.0GHz


SCCT Auto232-ELO difference:71

Code: Select all

Rank Name Elo + - games score oppo. draws
1 Houdini 2.0b Pro x64 6c 3426 17 17 1009 69% 3307 46%
5 Houdini 2.0b Pro x64 4c 3355 17 17 1009 51% 3350 47%

SCCT Auto232-ELO difference:67

Code: Select all

Rank Name Elo + - games score oppo. draws
4 Deep Rybka 4.1 x64 6c 3367 18 18 857 62% 3301 57%
7 Deep Rybka 4.1 x64 4c 3300 13 13 1603 47% 3321 56%

BTW,still i did not check the 6CPUs against 4CPUs games played in CEGT/CCRL
But really i wonder a lot,if they will run AMD Phenom II X6 against Intel i7 Quads (in Auto232 mode)

Can we expect 50 Elo difference (in case of under CEGT/CCRL conditions) ??



Best,
Sedat

[Adam Hair]

Hello Adam,


See bellow please (postings on CSS forum)

Clemens Keck schrieb:

Hi Sedat

other tester groups also have 2, 4, 6 cpu in their list.
How do they do that? Could you not learn from them a good system?

Regards, Clemens

Dear Clemens,

Probably you mean about CEGT/CCRL ?!

First of all,i'd like to say that i have big respect to their works-CEGT/CCRL

And the both great teams are putting a lot free efforts-for this i am so thankful for that...

But anyway,i think the both rating list (CEGT/CCRL) include misunderstanding results,especially i mean for 6CPU and 4CPU
Or maybe 4CPUs and 2CPU or 1CPU too

For example,as far as i know,they are owner of AMD Phenom II X6

And their testings with 6CPU against 4 CPU on same hardware
In my opinion a such test is wrong-it will not give us right Elo performance

BTW,strange results indeed,e.g CEGT has 15 ELO difference:
http://www.husvankempen.de/nunn/40_40%2 ... liste.html

1 Houdini 1.5a x64 6CPU 3290 23 23 600 73.0% 3117 36.0%
2 Houdini 1.5a x64 4CPU 3275 11 11 2641 73.0% 3102 34.9%

CCRL Rating has 45 ELO difference:
http://computerchess.org.uk/ccrl/404/ra ... t_all.html

Houdini 2.0c 64-bit 6CPU 3404
Houdini 2.0 64-bit 4CPU 3359

And i strongly believe in reality,AMD Phenom II X6 is not stronger than Intel Quads
Or maybe there will be 5-10 Elo difference-no more no less (it depend on the clock speed)

Honestly this is one of the main reason of canceling my new project-ICCR

Once more i'd like to thank you for your useful note

Greetings,
Sedat

Hi Sedat,

First of all, I am not going to try to convince you to do something that you do not want to do .

But I would like to clear up a bit of a misunderstanding. In your example above, you are comparing two different versions of Houdini. Furthermore, you have to take into account the error bars when comparing those two differences. The discrepancy you see has more to do with the number of games played than differences in computer systems.

Also, as I noted in my first post, if possible I would chose to test either 4 CPU or 6 CPU, but not both. That does get rid of one source of possible error. However, if that is not possible it still does not mean that highly useful data could not be generated from the time control you are proposing.

Good Luck with whatever you choose to do,

Adam