Beta for Stockfish distributed testing

[mcostalba]

The current queue of tests for Stockfish is up at http://54.235.120.254:6543/tests. Please note, operations like start run, delete run, etc. are restricted.

Gary, this is really another big step in the evolution of chess engine develpment.

What you've done will have far reaching consequences for engine development. I'm not sure many of the people here fully understands the deep implications.

I'm not only talking of the amazing technical achievement of this very sophisticated distributed testing framework, but especially to the fact that testing and devloping is now open, for the first time, to everybody.

With Galurung we started to have open source and GPL official releases

Then, with Stockfish we open the development branch through github, from when it is possible to track the build of a new relese change by change.

And now the last step: the opening of the testing process and validation. People can now see (and contribute) to what we test, to see what it works and what it doesn't.

These are IMHO the big 3 milestones we reached with Stockfish development, and it is a world first and I'm proud of it.

If you think that in this world majority of authors are still very secretive and jealous of their work and most engines are distributed only in binary format, you can apreciate even more the revolution that Stockfish development has been and still is.

Thanks Gary !

[jd1]

The current queue of tests for Stockfish is up at http://54.235.120.254:6543/tests. Please note, operations like start run, delete run, etc. are restricted.

Gary, this is really another big step in the evolution of chess engine develpment.

What you've done will have far reaching consequences for engine development. I'm not sure many of the people here fully understands the deep implications.

I'm not only talking of the amazing technical achievement of this very sophisticated distributed testing framework, but especially to the fact that testing and devloping is now open, for the first time, to everybody.

With Galurung we started to have open source and GPL official releases

Then, with Stockfish we open the development branch through github, from when it is possible to track the build of a new relese change by change.

And now the last step: the opening of the testing process and validation. People can now see (and contribute) to what we test, to see what it works and what it doesn't.

These are IMHO the big 3 milestones we reached with Stockfish development, and it is a world first and I'm proud of it.

If you think that in this world majority of authors are still very secretive and jealous of their work and most engines are distributed only in binary format, you can apreciate even more the revolution that Stockfish development has been and still is.

Thanks Gary !

Congratulations, wish you all the best

[gladius]

And now the last step: the opening of the testing process and validation. People can now see (and contribute) to what we test, to see what it works and what it doesn't.

I agree, I think this will be very interesting to see how it grows!

And thank you for doing such a great job with Stockfish! (And all the work on the distributed tester .

[Steve Maughan]

(...)
Gary, this is really another big step in the evolution of chess engine development.

What you've done will have far reaching consequences for engine development. I'm not sure many of the people here fully understands the deep implications.
(...)

I agree.

This is massive.

I predict, as a result of this initiative, Stockfish will gain 200 ELO in the next twelve to eighteen months!!

Steve

[mcostalba]

I predict, as a result of this initiative, Stockfish will gain 200 ELO in the next twelve to eighteen months!!

Actually I was not referring to ELO, I'd think we will gain some of it but much less than 200, and anyhow even in the totally unrealistic case of 200 ELO gain I'd still say that this is not the most important point.

The realy novelity is the fact that engine is tested out in the wild, fully open and free for everybody to see. This is, admittely, more attarctive for engine developers than for end users that just care to engine's ELO improvment, but for me is by far more important becuase it will foster a new wave of engines development.

IOW the importance of this novelity is not for the advantages of Stockfish itself but for the impact that it will have on the other engines and authors (like have been and still is the opening of the sources). An advantage only for one engine IMO cannot be called a milestone for the comunity.

[M ANSARI]

Vas showed that you can quickly improve ELO quickly by massive testing and fine tuning. I think a lot of Houdini's success comes from using a similar method and improving on the efficiency ... and if this works with Stockfish this takes everything to an even higher level. Should be a very interesting experiment. I always thought that a similar idea can be worked for creating an incredibly strong book as well. Chessbase has started with something, but it can be done much better.

[pohl4711]

Fantastic project. I wish all the best.
Unfortunately I need my hardware for my LS-ratinglist, so I cannot support this milestone of engine-development....

Stefan

[Kyodai]

Yes - this is a very interesting project, that surely has a great future!

The progress can be followed at the testsite mentioned in this thread and also here (? - if I got it right)

http://www.abrok.eu/stockfish/

[ThomasJMiller]

Machine: glinscott - 23 cores

is that just one computer or the sum of several?

[Houdini]

Vas showed that you can quickly improve ELO quickly by massive testing and fine tuning. I think a lot of Houdini's success comes from using a similar method and improving on the efficiency ...

Actually not. The real challenge is designing and implementing ideas that make a program stronger.
The massive testing is a necessary evil, a tool, but nowadays it's something anyone can do at low cost. Buy a couple of cheap AMD-8350 servers and you can run 100,000 games a day. But just "fine tuning" Stockfish will not yield a lot of Elo, somebody will have to come up with bright ideas about how to improve evaluation and search.

Robert