The idea is simple: to distribute games across different machines to reduce the test latency and increment throughput.
On the main forum I have already expressed my opinion on the general importance of such an open framework and I have called it a milestone. Here I will describe it from a more technical point.
Fishtest is a web application written mainly in Python under Pyramid Application Development Framework. Sources are open and public:
https://github.com/glinscott/fishtest
The supported features are:
- Full Web UI interface: tests are created, run, and view all from within a browser. Current instance is on
http://54.235.120.254/tests
- Deep integration with Github: all the sources of teh engines to test are on github repos (cloned from official Stockfish), each developer has his repo and fishtest is able to pick the sources on each of them, according to who is the user creating the test.
- Tests are run on each machine by cutechess-cli tournament manager
- Support for unlimited number of developers and testing machines
- Support for many types of test:
a. Self play with fixed number of games
b. Self play with sequential probability ratio test (SPRT) stop logic
c. Self play with CLOP tuning
d. Regression test against many engines
- At test creation time, patch author can choose:
a. Time control
b. Number of threads to run the test
c. Add specific custom options that cutechess will pass to the engine
d. Opening book in any supported format: polyglot, epd, pgn
e. Book depth
- The machines doing the test (workers) can run Linux or Windows and can have any number of cores (although now we restrict access to at least QUADS)
- Sources of engines under test can be compiled locally on each machine or precompiled for each platform on a remote builder (that can run on a different machine from web server, in a fully distributed fashion) and then downloaded to worker upon request.
- Speed of each worker is benchmarked at the beginning of the test and time control adjusted accordingly to allow uniform results across different machines.
- All test files are downloaded automatically: books, cutechess, sources, engines binaries. So setup of fishtest on a worker machine is very easy and fast.
- All games results are sent to host in real-time and stored and backed up in a central MongoDB.
- Security policy is implemented and login credentials are required for each developer. So although open, it is well protected: only allowed users can create and manage tests.
There is more than this, but this is enough for a short introduction
