In case you missed it: SSDF, shiny new hardware...

[attakinski]

In my opinion the SSDF-list is the best there is. Ponder on, learning on, opening book on and 40 in 120 is exactly as it should be.

The disadvantage is also very clear. a game 40 in 120 last at least 5 hours.
I test with 3 minutes a game and even then a tournament of 120 games lasts about 10 hours....

[Milos]

In my opinion the SSDF-list is the best there is. Ponder on, learning on, opening book on and 40 in 120 is exactly as it should be.

The disadvantage is also very clear. a game 40 in 120 last at least 5 hours.
I test with 3 minutes a game and even then a tournament of 120 games lasts about 10 hours....

40 moves in 120 on strong hardware is just plain rediculous, without special openings, there will be over 90% draw rate between top engines and additionally it is extremely stupid TC once you reach endgame.
Moreover, regular books are just too weak already after move 10 for such a TC and strong machine. So you take Brainfish book, maybe tune it a bit, limit it to 10 moves and with the latest SF you can't loose a single game.

[gerold]

Ed and Ray : I agree!

I agree also. One of the best test site for years.

They try not to test clones.

Best,
Gerold.

[Milos]

They try not to test clones.

They actually try to test only commercial engines. It's clear that some "sponsoring" money is involved.
It's just pathetic.

[matejst]

I haven't posted for years, although I am still reading regularly new threads, and remain very present on the forum. But I felt I had not much to say on many topics -- I regret though not to have thanked the many authors of engines and GUIs who offered us something different and original, useful for training and enjoying chess (Ed, Frank, Steven, HGM, Lucas, Brendan, Norman and many others).

After all these years, I still feel that the SSDF does an excellent job, and that their approach -- testing products, not engines -- herited from years when chess programs were complete software packages, not only engines, with a GUI, an opening book, is the right one. I like also the fact that they test at LTC -- it's probably the most important element for me.

I also do understand why they don't test Houdini, nor every version of engines. That said, in the comments of the chairman, I read that they soon will test the latest version of SF, so it's not that they avoid testing SF. Testing at LTC does severely limit the number of programs/engines they can test, so a choice has to be made. Some things could be done differently, but it's always easy to criticize.

For me, this list has still a lot to offer, especially since it's the only one testing dedicated chess computers, and conserving a testing paradigm I find is usefully complementing the work of other testing groups and individuals.

[Milos]

Code: Select all

 1 Komodo 11.01 MP x64 16GB 1800X 3,6 GHz, 3406                                 
Komo9.1 Q6600   25,5-18,5  Stockf6 Q6600   24,5-15,5  DShre13 Q6600     30-10
Komo7.0 Q6600   29,5-10,5  Stockf3 Q6600   34,5-15,5  DRybka4 Q6600   30,5-9,5
DHiar14 Q6600   30,5-9,5   Spike14 Q6600     34-6

Here is an example to illustrate all the ridiculousness of SSDF "testing".
Komodo played against 2 versions of itself, 2 prehistoric versions of SF one version of Shredder, Hiaracs, prehistoric Rybka and pathetic Spike. Most of engines it played couldn't even use 6-men. In total 320 games against those "determine" the rating.
In reality the result skewed beyond belief.

[mjlef]

"opening books distort a fair competition.". Yes, which is why humans are not allowed to use memory of opening preparation when they play in tournaments. Oh wait. They can use their memory of opening preparation.

It is true that if you knew the opening book used by your opponent you could spend a lot of time and make a counter book to help your winning chances. Hard, but with enough resources it could be done. Komodo has just used a general opening book tuned to lines it understands well, and not against a specific opponent. It is not very practical to have a unique book for each opponent. We do change some lines to prevent repeating games, and Erdo uses his bets judgement to select the first few moves.

I agree it would be unfair to allow one program to use an opening book and another one not. But just as adding an evaluation or search modification to a program can make that program stronger, a well tuned book can help a program play stronger. But I cannot call me adding a new evaluation component to Komodo "unfair" anymore than someone writing a better book.

As long as both parties are allowed to have a book they choose, how is it unfair?

I personally like testing without opening books, but only because I am lazy and am not skilled at making such books. But if you show up to an event like WCCC without a book, your chances will drop a lot.

SSDF has a different set of conditions they use in testing. Variety is good. And I am glad they are back. And I am sure they will test Stockfish 8 or whatever is ready when they get to it. I am happy they are back in action and with some better hardware.

[Rebel]

The SSDF is now using AMD Ryzen 7 1800X, which is an 8-core processor at 3.6 GHz. They have opted for 16 GB RAM as standard on this hardware.

what is SSDF?

I understand in the past, but now?

We still honor them for what they did in the past, as first one offer a platform for competition long before we heard of Lyudmil Tsvetkov

We still honor them now for willing to play LTC games + permanent brain + opening books + allowing learning software, meaning a rating list where programs are allowed to compete on their strongest settings.

permanent brain = bad;
opening books = bad;
learning features = bad;

Are table bases also bad?

are you really that basic-level?

Nope, just pointing out your (expected) inconsistency.

tablebases provide equal conditions for all,

They are not, some have them, others don't.

opening books distort a fair competition.

Not at all, an opening book is an essential part of a chess program, in there the programmer decides what suits his program best.

CCRL | CEGT etc. are all fine, so is the SSDF concept.

[Rebel]

why not test Houdini?

The SSDF folks don't buy, As a commercial you have to give them a couple of free versions.

[Leo]

The SSDF is now using AMD Ryzen 7 1800X, which is an 8-core processor at 3.6 GHz. They have opted for 16 GB RAM as standard on this hardware.

Good hardware, bad software. Stockfish 6??