If you are running on a single machine then cutechess-cli is really all you need.
Several engine developers are using OpenBench https://github.com/AndyGrant/OpenBench, developed by Andy Grant, author of Ethereal. It is similar in functionality to Fishtest (https://github.com/glinscott/fishtest, which is what the Stockfish team is using. These support tests run on multiple machines, possibly in mulitiple locations (WAN). OpenBench makes some assumptions, for example that an engine supports the "bench" command.
I use a combination of Python and shell scripts, together with cutechess-cli, to run matches on multiple machines, monitor the results, and do the SPRT test to determine when to stop the test. This assumes all machines are on a LAN. I haven't released all the pieces of this setup, but some parts of it are in the Arasan tool directory (https://github.com/jdart1/arasan-chess/ ... ster/tools) on Github: in particular, match.py, match_status.py and monitor.py.
--Jon
Looking for automatic Engine Testing Software
Moderators: hgm, Rebel, chrisw
-
- Posts: 4368
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
-
- Posts: 3557
- Joined: Thu Jun 07, 2012 11:02 pm
Re: Looking for automatic Engine Testing Software
Yes cutechess for me, although I use the GUI as I do like to see the games in progress from time to time.
-
- Posts: 725
- Joined: Tue Dec 18, 2007 9:38 pm
- Location: Munich, Germany
- Full name: Dr. Oliver Brausch
Re: Looking for automatic Engine Testing Software
Thank you for your tips!
As far as I can tell, there are two possibilities: Cutechess or OpenBench. Both seem to provide:
- Cross platform
- Concurrent Games on multiple CPU Cores
- Xboard Protocol
What are the main differences between those two?
As far as I can tell, there are two possibilities: Cutechess or OpenBench. Both seem to provide:
- Cross platform
- Concurrent Games on multiple CPU Cores
- Xboard Protocol
What are the main differences between those two?
Interesting. Where can I find more about those assumptions?
-
- Posts: 4368
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: Looking for automatic Engine Testing Software
Cutechess-cli provides the functionality to run matches, with many options.
OpenBench uses cutechess-cli for that layer of functionality but adds a UI on top and coordination across multiple machines, among other features. There is some basic documentation for it on the Github page. See "Engine Compliance with Openbench" section for what requirements are placed on engines.
--Jon
OpenBench uses cutechess-cli for that layer of functionality but adds a UI on top and coordination across multiple machines, among other features. There is some basic documentation for it on the Github page. See "Engine Compliance with Openbench" section for what requirements are placed on engines.
--Jon
-
- Posts: 4346
- Joined: Tue Apr 03, 2012 4:28 pm
Re: Looking for automatic Engine Testing Software
I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to dojdart wrote: ↑Sun Jul 19, 2020 6:20 pm Cutechess-cli provides the functionality to run matches, with many options.
OpenBench uses cutechess-cli for that layer of functionality but adds a UI on top and coordination across multiple machines, among other features. There is some basic documentation for it on the Github page. See "Engine Compliance with Openbench" section for what requirements are placed on engines.
--Jon
-
- Posts: 4368
- Joined: Fri Mar 10, 2006 5:23 am
- Location: http://www.arasanchess.org
Re: Looking for automatic Engine Testing Software
Not so. You can run your own private server. And in any case, you are not linking with OpenBench, or distributing your engine along with it, so its license provisions would not apply to your engine, IMO.I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do
--Jon
-
- Posts: 4346
- Joined: Tue Apr 03, 2012 4:28 pm
Re: Looking for automatic Engine Testing Software
I thought the idea was that engines were spread out over participating users PC and ran on those. A puts his engine on the system for testing and (unknown?) users XYZ have the engine on their system at home. Seems a pretty good way to get your engine “distributed”, willing or not. Or am I missing something.jdart wrote: ↑Sun Jul 19, 2020 7:41 pmNot so. You can run your own private server. And in any case, you are not linking with OpenBench, or distributing your engine along with it, so its license provisions would not apply to your engine, IMO.I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do
--Jon
-
- Posts: 725
- Joined: Tue Dec 18, 2007 9:38 pm
- Location: Munich, Germany
- Full name: Dr. Oliver Brausch
Re: Looking for automatic Engine Testing Software
For Mac Users:
then new Terminal Window:
let them play:
Many thanks and great respect for the creators of cutechess. Open source rules!
Code: Select all
brew install qt
echo 'export PATH="/usr/local/opt/qt/bin:$PATH"' >> ~/.bash_profile
Code: Select all
git clone https://github.com/cutechess/cutechess.git
cd cutechess
qmake -spec macx-g++ -config static
make
cp projects/cli/cutechess-cli /usr/local/bin/
Code: Select all
cutechess-cli -engine cmd=olithink557a -engine cmd=../olithink557 -each proto=xboard tc=40/60 -rounds 10 -concurrency 2
-
- Posts: 27869
- Joined: Fri Mar 10, 2006 10:06 am
- Location: Amsterdam
- Full name: H G Muller
Re: Looking for automatic Engine Testing Software
If you want to be able to watch the games, you could also use XBoard. (Available for Linux and Mac.)
-
- Posts: 1871
- Joined: Sat Nov 25, 2017 2:28 pm
- Location: France
Re: Looking for automatic Engine Testing Software
You can host your own openbench server and client on your own set of machines. Nobody else will see your dev or fill your hardware, but you also won't be able to use others resources ...chrisw wrote: ↑Sun Jul 19, 2020 8:21 pmI thought the idea was that engines were spread out over participating users PC and ran on those. A puts his engine on the system for testing and (unknown?) users XYZ have the engine on their system at home. Seems a pretty good way to get your engine “distributed”, willing or not. Or am I missing something.jdart wrote: ↑Sun Jul 19, 2020 7:41 pmNot so. You can run your own private server. And in any case, you are not linking with OpenBench, or distributing your engine along with it, so its license provisions would not apply to your engine, IMO.I don’t think you can have a private engine on that system. Which means basically you are forced into some kind if GPL licence which some people may not want to do
--Jon
For Minic I've build a docker image for this : https://github.com/tryingsomestuff/Mini ... ter/Docker