A question about adapted time controls

[Sedat Canbaz]

Hello Chess Friends,

I am not much experienced with adapted time controls under different speed hardwares

So urgently help is needed by any Chess Expert

I mean,which is the right way of testing,calculating with adapting time controls ?

Thanks in advance,
Sedat


BTW,there were some comments on CSS forum about for the current issue:

Dear Sedat

there is one important thing I recommend you to think well about:
ELO performance on multiple cores is NOT measurable between 4 core and 6 core computers!
The main reason is: the engines are different in their work with 2, 4, 6, 8 etc. cores.
Some benefit more from +cores, some less.
So if you do this test with houdini, you only can say you meassured Houdini 2.0c x64, and even this result will be not really correct.

regards, Clemens

Thank you for your useful comments dear Clemens

A good point...

Actually this issue is too complicated...

Yes,sure...there can be X mp engine which will have worse or stronger ELO performance on 4core or 6core machines
But not for Rybka or Houdini,see the bellow SCCT Auto232 results

About your question...
Hmm...then is appearing another BIG problem-how we can calculate the all results played by 4core and 6core in ICCR rating list

And its completely wrong...if we start testing the engines on same machine-6c against 4c

And will explain you why...

Let's say we used a engine,which all cores work properly,e.g Houdini 2.0c

1)For Example on my i7 980X 3.33 Six core machine:
-Intel Core i7 980X 3.33 GHz 6 core 14744 kns 40m+10s
-Intel Core i7 980X 3.33 GHz 4 core 10490 kns 40m+10s
*Note that Time control:40m+10s (for both engines)

*Elo difference is expected to be around 25 ELO (on i7 980X @4.33GHz between 6c and 4c)

-----------------------

2) For Example,let's say other Tester played many games only by 4core engines on Quad-i7 2600k 3.40GHz:
Intel Core i7 2600K 3.40 GHz 4 10473 61m+10s

So...i am asking you which one is stronger,if there will be mach between them?
-Intel Core i7 980X 3.33 GHz 6 core 14744 kns 40m+10s
or
Intel Core i7 2600K 3.40 GHz 4 core 10473 kns 61m+10s

In my opinion,in case of similar conditions...it will be ended at equal ELO performance for both engines (i mean i7 980X 6c against i7 2600K 4c)

---------------------
Another VERY IMPORTANT question:which one is stronger,if there will be mach between them?
-Intel Core i7 980X 3.33 GHz 4 core 10490 kns 40m+10s
or
-Intel Core i7 2600K 3.40 GHz 4 core 10473 kns 61m+10s

In my opinion,in case of such match...it will be ended approx.25 ELO in favor for i7 2600K 4c

----------------------------------

Conclusion:
In case of testing the engines,e.g 6c against 4c on same PC- there will be big misunderstanding results-wrong Elo calculations
All played standings will be effected....you can not see and compare the real ELO strength of the engines
Actually the right testing (6c against 4c) on same machine will be ok,if:
- all Rating games are played with same time control and on same hardware
The problem is appearing due to adapting the time controls under many different speed processors
And after all...really i am disappointed...it seems no any right solution for the current complicated issue
But anyway, i think the current Auto232 hardware ELO method calculation is better,of course i mean for NON-buggy MP engines

*************************************

More details about Houdini and Rybka with 4 cores or 6 cores,which i believe that both engines are playing at full performance

SCCT Auto232-ELO difference:71
Rank Name Elo + - games score oppo. draws
1 Houdini 2.0b Pro x64 6c 3426 17 17 1009 69% 3307 46%
5 Houdini 2.0b Pro x64 4c 3355 17 17 1009 51% 3350 47%


SCCT Auto232-ELO difference:67
Rank Name Elo + - games score oppo. draws
4 Deep Rybka 4.1 x64 6c 3367 18 18 857 62% 3301 57%
7 Deep Rybka 4.1 x64 4c 3300 13 13 1603 47% 3321 56%

That means, i7 980X @4.33GHz is approx.70 Elo stronger than against i7 920 @3.0GHz


Best,
Sedat

[bob]

Here's my take.

This "adapting stuff" is unique for each engine. One can't say that processor X vs processor Y should be played where the program on processor Y gets 1.2x more time than the program on processor X, because processor X is faster. You need to take each program, run a reasonable number of chess positions to a fixed depth, and save the total time. Then run it on the other machine and do the same. Divide these total times to compute the speedup/slowdown factor for THAT program. Repeat for other programs. If you are going to use multiple cores for one engine, I would run that time-to-depth run several times and average the results, perhaps throwing out the high and low ones.

[Sedat Canbaz]

Thank you for your useful info dear Robert

In other words,it seems there is no way creating a accurate rating (which will show us the real performance of the engines) with adapted time controls under different speed hardwares ?

And in my opinion:
-Rating list which is based on same hardware speed and same time control is ideal for measuring the real Elo strength of the engines

Best,
Sedat

[bob]

Thank you for your useful info dear Robert

In other words,it seems there is no way creating a accurate rating (which will show us the real performance of the engines) with adapted time controls under different speed hardwares ?

And in my opinion:
-Rating list which is based on same hardware speed and same time control is ideal for measuring the real Elo strength of the engines

Best,
Sedat

Yes, you can get good results. But you need to compute the "adjustment" factor for each program separately.

For example, take 3 programs, A, B and C.

And two machines X and Y, where X is the faster.

Run the test for A on X and Y. Note the "adjustment" computed by time(y) / time(x). Let's say you get 1.2x here.

Run the test for B on X and Y. Say you get 1.3x here.

And run the test for C on X and Y and say you get 1.0 here (programs run at same speed on either box for some reason).

When you play A on X and B on Y, you use the correction factor for B on Y and give B 1.3x longer than you give A.

Then you run A on X and C on Y, and you give C on Y the same time as X since there was no adjustment.

When you play A on Y and B on X, you give A 1.2x the time you would give it on X, since that is how much slower it will run on Y and that adjusts it to be equal to X. Note that ponder off has to be used here or the results will be invalid completely...

[Terry McCracken]

Thank you for your useful info dear Robert

In other words,it seems there is no way creating a accurate rating (which will show us the real performance of the engines) with adapted time controls under different speed hardwares ?

And in my opinion:
-Rating list which is based on same hardware speed and same time control is ideal for measuring the real Elo strength of the engines

Best,
Sedat

Yes, you can get good results. But you need to compute the "adjustment" factor for each program separately.

For example, take 3 programs, A, B and C.

And two machines X and Y, where X is the faster.

Run the test for A on X and Y. Note the "adjustment" computed by time(y) / time(x). Let's say you get 1.2x here.

Run the test for B on X and Y. Say you get 1.3x here.

And run the test for C on X and Y and say you get 1.0 here (programs run at same speed on either box for some reason).

When you play A on X and B on Y, you use the correction factor for B on Y and give B 1.3x longer than you give A.

Then you run A on X and C on Y, and you give C on Y the same time as X since there was no adjustment.

When you play A on Y and B on X, you give A 1.2x the time you would give it on X, since that is how much slower it will run on Y and that adjusts it to be equal to X. Note that ponder off has to be used here or the results will be invalid completely...

You know, you're quite possibly the most logical man at this site. I don't know how you handle their abuse?

I can't.