Fishtest Distributed Testing Framework

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
mcostalba
Posts: 2679
Joined: Sat Jun 14, 2008 7:17 pm

Fishtest Distributed Testing Framework

Post by mcostalba » Wed May 01, 2013 9:07 am

The big news of Stockfish 3 release is the extensive use of the new distributed testing framework by Gary Linscott called Fishtest.

The idea is simple: to distribute games across different machines to reduce the test latency and increment throughput.

On the main forum I have already expressed my opinion on the general importance of such an open framework and I have called it a milestone. Here I will describe it from a more technical point.

Fishtest is a web application written mainly in Python under Pyramid Application Development Framework. Sources are open and public:

https://github.com/glinscott/fishtest

The supported features are:

- Full Web UI interface: tests are created, run, and view all from within a browser. Current instance is on

http://54.235.120.254/tests

- Deep integration with Github: all the sources of teh engines to test are on github repos (cloned from official Stockfish), each developer has his repo and fishtest is able to pick the sources on each of them, according to who is the user creating the test.

- Tests are run on each machine by cutechess-cli tournament manager

- Support for unlimited number of developers and testing machines

- Support for many types of test:
a. Self play with fixed number of games
b. Self play with sequential probability ratio test (SPRT) stop logic
c. Self play with CLOP tuning
d. Regression test against many engines

- At test creation time, patch author can choose:
a. Time control
b. Number of threads to run the test
c. Add specific custom options that cutechess will pass to the engine
d. Opening book in any supported format: polyglot, epd, pgn
e. Book depth


- The machines doing the test (workers) can run Linux or Windows and can have any number of cores (although now we restrict access to at least QUADS)

- Sources of engines under test can be compiled locally on each machine or precompiled for each platform on a remote builder (that can run on a different machine from web server, in a fully distributed fashion) and then downloaded to worker upon request.

- Speed of each worker is benchmarked at the beginning of the test and time control adjusted accordingly to allow uniform results across different machines.

- All test files are downloaded automatically: books, cutechess, sources, engines binaries. So setup of fishtest on a worker machine is very easy and fast.

- All games results are sent to host in real-time and stored and backed up in a central MongoDB.

- Security policy is implemented and login credentials are required for each developer. So although open, it is well protected: only allowed users can create and manage tests.


There is more than this, but this is enough for a short introduction :-)

jdart
Posts: 3572
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Fishtest Distributed Testing Framework

Post by jdart » Wed May 01, 2013 2:29 pm

Nice. My "testing framework" if you can call it that is a few bash and perl scripts. It does let me run tests from one machine and have them distributed. There is a "scatter" command that pushes an executable to test machines, and a "gather" command that pulls the results back and gives a summary. It is all on Linux. I see you have Windows machines in the cluster you are using so it seems your python stuff is cross-platform.

--Jon

Post Reply