Elo difference for 4-8-16-24-32 cores

[Modern Times]

Houdini has the built-in autotune facility.

For the CEGT lists, for 8CPU and 12CPU, personally I would advocate changing the split depth to appropriate values as recommended by the author. Sure it is a departure from pure default settings, but if you really want to show how the engines scales, it would be a good idea.

[Sedat Canbaz]

I am glad that you see the reality !

Normally many people can not see this...!!

We discuss origins, we discuss engines, etc...in Talkchess
but unfortunately no much discussions about opening books

For example, maybe you remember, (a few weeks ago)
The poll winner was: books don't play a big influence...
So...the conclusion is that: still the majority of people don't believe in the opening lines

But in reality is not true...the openings play a big role, any engine can be performed 0-200 Elo or even more...

And it's time for revolution for engine testers too!

I've been a believer in the importance of opening books since I saw this:
http://talkchess.com/forum/viewtopic.ph ... w=&start=0

Rybka 3 scoring almost 50% against Stockfish 5 thanks to an opening book shocked me at that time. Now I know that engines are still not good enough in the opening phase.

Nice....

I worked only approx. 3 hours over the used book...
But If I start to work more seriously...we can expect at least 60% the results to be in favor for Rybka !

And here is the poll results:
.


BTW,
Many people are interested in beauty, but it's time to concentrate on performance and strength!!

That means,
Many people (including some GMs too) have no clue about the latest engine opening theory...!!
Many testers prefer weak openings...and that leads to misunderstanding and strange results !

Who still does not believe...
I suggest them to participate with their opening books in SCCT Book Tournaments....only then they will understand what I am meaning...!!

[lkaufman]

I am glad that you see the reality !

Normally many people can not see this...!!

We discuss origins, we discuss engines, etc...in Talkchess
but unfortunately no much discussions about opening books

For example, maybe you remember, (a few weeks ago)
The poll winner was: books don't play a big influence...
So...the conclusion is that: still the majority of people don't believe in the opening lines

But in reality is not true...the openings play a big role, any engine can be performed 0-200 Elo or even more...

And it's time for revolution for engine testers too!

I agree that opening books can make a significant difference in the sense that the better book will help against the weaker book. But in testing, both sides use the same book, and colors are reversed, so it's not clear why the book should matter much. A book with boring openings (exchange slav, exchange French, etc.) will produce more draws and the ratings will be closer together than if the book is full of double-edged lines, but if the book is just a selection of lines popular at elite GM level that should be fair. So my question is whether you think that your own books should produce more accurate ratings than a book that just picks from Elite GM games, and if so why. What should be the characteristics of the ideal book for reversal testing?

[Sedat Canbaz]

I agree that opening books can make a significant difference in the sense that the better book will help against the weaker book. But in testing, both sides use the same book, and colors are reversed, so it's not clear why the book should matter much. A book with boring openings (exchange slav, exchange French, etc.) will produce more draws and the ratings will be closer together than if the book is full of double-edged lines, but if the book is just a selection of lines popular at elite GM level that should be fair. So my question is whether you think that your own books should produce more accurate ratings than a book that just picks from Elite GM games, and if so why. What should be the characteristics of the ideal book for reversal testing?

Dear Larry,

I never use own or larger opening books for my testings, tournaments, ratings...
Exceptions:
-I used a special book for Rybka vs Stockfish + no book
http://talkchess.com/forum/viewtopic.ph ... w=&start=0
-Plus I used own opening books for my recent 3 type Swiss Tournaments
http://www.sedatcanbaz.com/chess/?page_id=133
-Also I use own/large books, in case of organizing SCCT Book Tournaments...
http://www.sedatcanbaz.com/chess/?page_id=602
Where the book authors participate with larger, optimized strong opening books

In other words,
Usually I test the engines with neutral short opening book (Perfect 2014t up to 8 moves)
http://www.sedatcanbaz.com/chess/?page_id=822

More details,
I think creating ratings with own books is not very good idea, due to many own books have a big holes
And in case of using own old-dated books... we can see different standings (I mean not accurate...)

Note that a very good top book maker can create a superior book for Komodo style,
where this book probably would be at least 50 Elo over Stockfish!
Or just opposite.... would be at least 50 Elo over Komodo!

Own books are good idea, e.g in case of participating in online tournaments, but not good for ratings...


About reverse colors testings,
Yes...it sounds good, but however it depends of the used opening lines
For example....there are a lot weak openings, where many chess engines suffer too...
And it does not mean if the testing is with reversed color is alright...
For example, (in case of using not so strong lines) weaker engines can be rated 50 or 100 Elo or even more

My recommendation is that (for ideal testing)
Engine testers should be careful in case of using the opening books or lines
And as far as possible, the most modern, decent strongest openings to be used...
The strong chess engines should not be tested with openings (based on many games per same opening line):
-where the winning percentage will be less than 50 % for Whites and less than 40 % for Blacks
More details:
http://www.sedatcanbaz.com/chess/?page_id=127

Note also that (under SCCT conditions)
My current Perfcect 2014t books's overall lines are almost 55 % for Whites and 45 % for Blacks
Thats why the Elo error bar margins are disappearing after 500 games per player )!
It appears but around 5-10 Elo...no more no less...

And depending on used conditions: players, tc, ponder on/off, mp/1 core etc...
Even Perfect 2014t book suffers too...even after several years of hard working over the used lines
It's true that mostly of the lines work almost perfectly, but sometimes the winning percentage of some lines are not so good
I mean less than 50% for Whites or less than 40% for Blacks
And here is an example...
Note: I already disabled those lines which perform less than 50% (Whites) and 40% (Blacks)

1)A book based on Perfect 2014t games (during my SCCT Open Source Rating)



.
2)A book based on Perfect 2014t games (during my decent SCCT MP Rating)





Best,
Sedat

[Adam Hair]

My recommendation is that (for ideal testing)
Engine testers should be careful in case of using the opening books or lines
And as far as possible, the most modern, decent strongest openings to be used...
The strong chess engines should not be tested with openings (based on many games per same opening line):
-where the winning percentage will be less than 50 % for Whites and less than 40 % for Blacks
More details:
http://www.sedatcanbaz.com/chess/?page_id=127

Note also that (under SCCT conditions)
My current Perfcect 2014t books's overall lines are almost 55 % for Whites and 45 % for Blacks
Thats why the Elo error bar margins are disappearing after 500 games per player )!
It appears but around 5-10 Elo...no more no less...

Hi Sedat,

Do you also remove lines that tend to be overly drawish?

Adam

[Sedat Canbaz]

Hi Sedat,

Do you also remove lines that tend to be overly drawish?

Adam

Hello dear Adam,

A good question...

As far as possible,
I use the strongest openings where the top engines (at near strength) have equal chances for win!
Or at least Blacks have more chances to finish the game as draw! better draw than to loose !!

In other words...
SCCT's each played game is very important for me...it's something like FIFA World Cup Finals!)

And as I mentioned before, I strongly believe in that:
In case of using Perfect 2014t book, it is not needed running thousands of games per player
500-1000 games per player is quite enough data for determining about the Elo strength of X Engine

Note also that, in case of allowing many risky, critical, weak openings...
Then (for accurate ratings and to balance...)
We will need min. 2000-3000 games per player...but using various other openings too

Note also that,
Currently I am quite satisfied with Perfect 2014t's performance...the best neutral book of my series, which I've created so far!!


Btw, for the new future,
I expect almost all computer chess games will be ended as DRAW !
Not sure how many years will take exactly...but we will not wait much...maybe in 50 years !
Of course...in case of using the strongest chess engines and strongest opening lines!!


SCCT Rating MP - Ponder ON's overall general statistics
http://www.sedatcanbaz.com/chess/?page_id=822

*Notes:
- Perfect 2014t (up to 8 moves) is used for all participants
- The current statistics are based on Top 10 chess engines

Games : 3150 (finished)

White Wins : 1032 (32.8 %)
Black Wins : 690 (21.9 %)
Draws : 1428 (45.3 %)
Unfinished : 0

White Perf. : 55.4 %
Black Perf. : 44.6 %

ECO A = 265 Games ( 8.4 %)
ECO B = 1099 Games (34.9 %)
ECO C = 719 Games (22.8 %)
ECO D = 853 Games (27.1 %)
ECO E = 214 Games ( 6.8 %)

-----------------------------------------------------------


SCCT Book Championship I/II's overall general statistics
http://www.sedatcanbaz.com/chess/?page_id=602
http://www.sedatcanbaz.com/chess/?page_id=129

*Notes:
- Stockfish 110514 is used for all book participants
- Houdini 4's played games are not included...


Games : 3574 (finished)

White Wins : 552 (15.4 %)
Black Wins : 231 ( 6.5 %)
Draws : 2791 (78.1 %)
Unfinished : 0

White Perf. : 54.5 %
Black Perf. : 45.5 %

ECO A = 197 Games ( 5.5 %)
ECO B = 1703 Games (47.6 %)
ECO C = 1044 Games (29.2 %)
ECO D = 576 Games (16.1 %)
ECO E = 54 Games ( 1.5 %)




Best,
Sedat

[x3]

Results so far
K8 16 core vs k8 8 core + 52 elo 1049 games xeon 3.4
K8 24 core vs K8 16 core +19 elo 985 games opteron 3.1

Book 8 moves repeating
3m +1s
Ponder off

x3

[Sedat Canbaz]

Results so far
K8 16 core vs k8 8 core + 52 elo 1049 games xeon 3.4
K8 24 core vs K8 16 core +19 elo 985 games opteron 3.1

Book 8 moves repeating
3m +1s
Ponder off

x3

Interesting...

Komodo is doing better than Stockfish on your 16 core and 24 core machines!

Btw, are you planing to publish the games ?


Best,
Sedat

[Sedat Canbaz]

One thing more,
In 10 years, we noticed the top engines become approx. 1000 Elo stronger!
Note: Hardware+Engine+Book... improvements!
And I can't imagine after 10 years what will be their improvement...?!
But definitely 'chess' will be solved as draw by the strongest engines and by strongest books!

Just 'd like to add this too (for the new future),
SCCT Book Tournaments confirm my point of view that almost all games are expecting to be ended as draw!
And then (in case of any win) we will discuss why this game is not ended as draw

Note also,
Perfect 2014t book has highest draw percentage comparing with other neutral testings...!
Do you know why ?!


To find the answer...
You need to check the draw statistics of SCCT Book Tournaments too
Note: SCCT Book Participants are one of the World's Strongest Book Makers !
That's why I can say: it's a very hard league...no much chance to take a win...


Hopes helps...