Scripting matches in Windows environment

[Greg Strong]

I'm looking to build a distributed, poor-man's cluster for engine testing. I don't have a real, dedicated cluster. (Not many people do.) But what I do have are some powerful machines at work that are sometimes inactive. I've also got a couple powerful computers at home and friends with powerful computers at home that have spare cycles that they'd happily donate.

So the plan is roughly this ... Each machine has a command line tournament director (like Winboard or Cutechess or whatever), a variety of good, non-commercial "reference" chess engines, and whatever other engines are to be tested. And a new program that I'll write that the user can run when the machine isn't doing anything else and has cycles to spare. This program will make an HTTP connection to a server which will tell it what match to play - what engines, time controls, settings, starting position, etc. It will run the match, send the results back, and request the next match.

In this way, I can decide what tests I want to run by lineing up matches in the central database, and they'll get played out as resources are available. I can come back and look at the results and schedule new matches accordingly. This will help to tweek paramters that require a large number of games to reduce the error bar enough to determine optimum values.

Given that lofty goal, I've got some questions before I embark on a specific path ...

1. What should I use to actually run the matches? Winboard? Cutechess? Something else? Sorry, first I must point out that I know most of you are Linux people, and, being an open-source freak myself, you'd think I would be too. But sadly, I'm in Windows land at home, and even if I wanted to change, the machines at work that I'd be using run Windows, and my friend's computers are running Windows, etc. So it has to work with Windows. Cutechess-cli claims to be cross-platform but there are no builds for any environment available that I've located and before I put time into trying to recompile it, I'd be interested to hear if it's been done and how easy it is. Winboard obviously offers Windows executables, but the number of command line options is so long I looked at it for over an hour without figuring out exactly what the heck I need and don't need. That'll require a lot of testing, so I'd be interested to hear if anyone knows exactly what works there (and many good engines are UCI.) Any other ideas here?

2. I'm concerned that test results might be inaccurate if I run thousands of games on different machines, even if they have the same time controls, are limited to only one thread, and the tables are small enough to fit in ram on all machines, simply because CPUs will be of different speeds (and thus will reach different search depths on different machines.) On the other hand, maybe it'll balence out. And I can make sure all machines are reasonably powerful. Maybe I could run a chess benchmark and send that with the results. Any thoughts about this?

3. Anything I'm overlooking?

Thanks in advance for any feedback. You can be sure that whatever I develop I'll make freely available for all as I have with ChessV

[ilari]

Cutechess-cli claims to be cross-platform but there are no builds for any environment available that I've located and before I put time into trying to recompile it, I'd be interested to hear if it's been done and how easy it is.

There are binaries for cutechess-cli at http://koti.mbnet.fi/~ilaripih/bin/
Is this what you're looking for?: http://koti.mbnet.fi/~ilaripih/bin/cute ... -win32.zip
Compiling isn't difficult either, but that requires installation of Nokia's Qt libraries.

[hgm]

You could consider applying a real ICS to this task. You would have to set it up a little differently, though: you could assign each machine one or more fixed engines (as many as they could handle simultaneously, given their nr of cores), which they would make logon to the ICS as soon as the machine has CPU time to spare. An adapted version of mamer logged on as a bot to the ICS would then assign them games, and collect the results.

[mcostalba]

2. I'm concerned that test results might be inaccurate if I run thousands of games on different machines, even if they have the same time controls, are limited to only one thread, and the tables are small enough to fit in ram on all machines, simply because CPUs will be of different speeds (and thus will reach different search depths on different machines.) On the other hand, maybe it'll balence out. And I can make sure all machines are reasonably powerful. Maybe I could run a chess benchmark and send that with the results. Any thoughts about this?

If you are worried about this you could validate the testing environment before to proceed: pick some engines with well known strenght and repeat the same tournament across all the machines and at the end verify results are within error bar.

[hgm]

In the ICS setup you would avoid this by considering each combination engine+machine a separate entity.

[garybelton]

I'm doing something similar on windows ... I have a script which I manually give some information to 1) machine name 2) engine (eg. Rybka, Stockfish etc) 3) book name (I accept Aquarium (therefore CTG) or polyglot books. When run the script creates a bunch of files - batch files for the matches to play on my ICS and batch files for the matches to play without an ICS (using cutechess). They are exclusive, I either run the games on ICS or via cutechess. I don't use an http server though, I use a windows machine to drive the matches, the windows machine invokes any remote engines using ssh, the ICS when it runs runs in a Linux virtualbox. The advantages of using cutechess are in adjudication and more flexible time controls, the advantage of using the ICS are that the games are adjourned if there is a network disconnection between the driving windows machine and the remote engine. I would be interested in how you get on with your http solution.

[nthom]

I've written a tournament manager program for windows for my own testing. It can be configured to use multiple threads for parallel games and works with very fast time controls. Not sure if its suitable for your use but feel free to try it out and suggest changes.

http://www.kimiensoftware.com/littletho ... litzer.php