Beta for Stockfish distributed testing

[Adam Hair]

Vas showed that you can quickly improve ELO quickly by massive testing and fine tuning. I think a lot of Houdini's success comes from using a similar method and improving on the efficiency ...

Actually not. The real challenge is designing and implementing ideas that make a program stronger.
The massive testing is a necessary evil, a tool, but nowadays it's something anyone can do at low cost. Buy a couple of cheap AMD-8350 servers and you can run 100,000 games a day. But just "fine tuning" Stockfish will not yield a lot of Elo, somebody will have to come up with bright ideas about how to improve evaluation and search.

Robert

Exactly! There can be some gain in strength for an engine if the parameters need to be tuned. But after that point, it takes an influx of coded ideas to increase strength. Large scale testing can filter the effective ideas from the ineffective ideas and tune new parameters. But there is no replacement for the art of detecting a missing idea and embodying that idea into code in such a manner that the engine becomes stronger.

[Steve Maughan]

(...)
Actually not. The real challenge is designing and implementing ideas that make a program stronger.
(...)

Oh I agree wholeheartedly! All innovation occurs first in the mind of the inventor. But don't you think a public stream of data showing the previous tests and whether or not they were successful will spark some new ideas?

Steve

[gladius]

Actually not. The real challenge is designing and implementing ideas that make a program stronger.
The massive testing is a necessary evil, a tool, but nowadays it's something anyone can do at low cost. Buy a couple of cheap AMD-8350 servers and you can run 100,000 games a day. But just "fine tuning" Stockfish will not yield a lot of Elo, somebody will have to come up with bright ideas about how to improve evaluation and search.

Robert

Yes, of course, ideas are key. However, running test games across multiple servers is a labor intensive task! That's exactly what this project is meant to help with. Not having to worry about all logistics behind running games across many computers, and just having a result show up after a few days.

In the case of Stockfish, there is another benefit, as people can see the ideas that are being tried out, and which worked and didn't work.

[mcostalba]

Fishtest distributed testing framework is now Windows compatible.

So if anyone with a Windows machine is interested can now join us.

[Joerg Oster]

The current queue of tests for Stockfish is up at http://54.235.120.254:6543/tests. Please note, operations like start run, delete run, etc. are restricted.

Gary, this is really another big step in the evolution of chess engine develpment.

What you've done will have far reaching consequences for engine development. I'm not sure many of the people here fully understands the deep implications.

I'm not only talking of the amazing technical achievement of this very sophisticated distributed testing framework, but especially to the fact that testing and devloping is now open, for the first time, to everybody.

With Galurung we started to have open source and GPL official releases

Then, with Stockfish we open the development branch through github, from when it is possible to track the build of a new relese change by change.

And now the last step: the opening of the testing process and validation. People can now see (and contribute) to what we test, to see what it works and what it doesn't.

These are IMHO the big 3 milestones we reached with Stockfish development, and it is a world first and I'm proud of it.

If you think that in this world majority of authors are still very secretive and jealous of their work and most engines are distributed only in binary format, you can apreciate even more the revolution that Stockfish development has been and still is.

Thanks Gary !

Congrats to both of you!!!
This is really breathtaking and a big step in future development of SF.

Best, Jörg.

[gladius]

Fishtest distributed testing framework is now Windows compatible.

So if anyone with a Windows machine is interested can now join us.

Huge thanks to Marco and Ilari for getting Windows up and running.

Since the beta announcement, we have had up to six computers testing at once, and have played over 400,000 games!

[lucasart]

The current queue of tests for Stockfish is up at http://54.235.120.254:6543/tests. Please note, operations like start run, delete run, etc. are restricted.

Gary, this is really another big step in the evolution of chess engine develpment.

What you've done will have far reaching consequences for engine development. I'm not sure many of the people here fully understands the deep implications.

I'm not only talking of the amazing technical achievement of this very sophisticated distributed testing framework, but especially to the fact that testing and devloping is now open, for the first time, to everybody.

With Galurung we started to have open source and GPL official releases

Then, with Stockfish we open the development branch through github, from when it is possible to track the build of a new relese change by change.

And now the last step: the opening of the testing process and validation. People can now see (and contribute) to what we test, to see what it works and what it doesn't.

These are IMHO the big 3 milestones we reached with Stockfish development, and it is a world first and I'm proud of it.

If you think that in this world majority of authors are still very secretive and jealous of their work and most engines are distributed only in binary format, you can apreciate even more the revolution that Stockfish development has been and still is.

Thanks Gary !

Congrats to both of you!!!
This is really breathtaking and a big step in future development of SF.

Best, Jörg.

Joerg,

I see you're only using "--concurrency 1" (http://54.235.120.254:6543/tests)

But your signature suggests that you have much more unutilized horsepower.

[ZirconiumX]

Me and Gary are just squashing out a few bugs with the tester.

The main one is that stockfish was compiled with ARCH=x86-64-modern which is brilliant for the Nehalem/Sandy Bridge/Ivy Bridge based Core i7s, but useless for Core 2 based machines

The next one is that kernel version seems to be used as an identifier - which creates a problem if they have to switch kernels midway through testing (computer - an 8-core machine - was lagging, so I switched to Linux 3.8.3-ck1, confusing the tester)

Matthew:out

[mcostalba]

Me and Gary are just squashing out a few bugs with the tester.

The main one is that stockfish was compiled with ARCH=x86-64-modern which is brilliant for the Nehalem/Sandy Bridge/Ivy Bridge based Core i7s, but useless for Core 2 based machines

This is easily fixed using a custom_make.txt file with your make command as described in the README at the section "Override default make command"

The next one is that kernel version seems to be used as an identifier - which creates a problem if they have to switch kernels midway through testing (computer - an 8-core machine - was lagging, so I switched to Linux 3.8.3-ck1, confusing the tester)

This one I have not understood what you mean. Who is "they" ? Who is switching the kernel ?

[ZirconiumX]

Me and Gary are just squashing out a few bugs with the tester.

The main one is that stockfish was compiled with ARCH=x86-64-modern which is brilliant for the Nehalem/Sandy Bridge/Ivy Bridge based Core i7s, but useless for Core 2 based machines

This is easily fixed using a custom_make.txt file with your make command as described in the README at the section "Override default make command"

The next one is that kernel version seems to be used as an identifier - which creates a problem if they have to switch kernels midway through testing (computer - an 8-core machine - was lagging, so I switched to Linux 3.8.3-ck1, confusing the tester)

This one I have not understood what you mean. Who is "they" ? Who is switching the kernel ?

They means the computer operator.

Matthew:out