Ultrafast UCI engines tournaments : a freeware tool

[Marc Lacrosse]

Vasik Rajlich and Larry Kaufman recently revealed on the Rybka forum that they test slight modifications of Rybka through the running of a very large number of ultrafast games : 80,000 three-seconds-total-duration (!) games played on two quads each night. This allows to detect improvements/worsenings of their engine in the one elo point range ...

For those who would like to go on this way toward ultrafast testing, no software I know of (free or commercial) is able to manage these kinds of tournaments and games.

In fact Rajlich and Kaufman designed a specific tool for this but they do not intend to make it publicly available.

So I tried to put together something of my own. This is not completely trivial under windows because as you go faster you have to deal with aborted or uncorrectly finished threads and processes after each game.

The resulting little tool is a kind of preprocessor : you define parameters in a small text file, the utility takes it and produces an enormous batch file. You launch the batch and the tournament is run. The process makes use of WinboardX (Tim Mann / Alessandro Scotti), Polyglot 1.4 (Fabien Letouzey) and of several freeware utilities dealing with threads and processes. I only tested it under XP64 but I have been told it also works under 32bits XP.

I have put together all required utilities in a package available at http://chessbazaar.mlweb.info.

The fastest I could go with it is a little more than four complete games per minute. At this rate more than 97% of games are played successfully with most top range engines (Zappa Mexico 1 being an exception with a 25% failure rate).

If you stay more reasonably at a duration of 30 seconds per game the success rate is close to 100% with all engines.

This is still far of the stunning testing scheme that Larry Kaufman uses but not too bad yet IMHO.

Last word : this is an amateur non-polished package for those who are not afraid to deal with winboard commands and parameters. It comes with a kind of manual and example config files but I will not be able to give more detailed explanations.

Marc Lacrosse

[Uri Blass]

Vasik Rajlich and Larry Kaufman recently revealed on the Rybka forum that they test slight modifications of Rybka through the running of a very large number of ultrafast games : 80,000 three-seconds-total-duration (!) games played on two quads each night. This allows to detect improvements/worsenings of their engine in the one elo point range ...

For those who would like to go on this way toward ultrafast testing, no software I know of (free or commercial) is able to manage these kinds of tournaments and games.

This is simply not correct because winboard allow fast time control like game in 1 second.

I can add that I think that for testing small changes it is better not to use time control and simply use fixed number of nodes assuming that you know the number of nodes that you need.

Arena allow game fixed number of nodes but unfortunately the interface is too slow and when you use small number of nodes the games are not finished fast because of the slow interface and not because of the engines.

Uri

[Marc Lacrosse]

This is simply not correct because winboard allow fast time control like game in 1 second.

Uri

My package uses winboard.

But the point is :

1. No tournament manager so far makes any use of this possibility

2. You probably never tried to have winboard playing hundreds or thousands of games at these time controls, chaining them with the shortest possible delay. You soon end up with windows slowing down and then crashing because of lack of memory cleaning between games.

The point is not to launch winboard for one game to be completed in one or a few seconds. The point is to have it running successfully for 250 games in one hour.

So once more your favorite definitive five-word comment ("This is simply not correct") is erroneous.

Marc

[hgm]

It is easy enough to add a limit-number-of-nodes command to WinBoard, but the problem is that not many native WB engines do support it.

Speed of the interface and scheduler will always be a severe bottleneck. I could easily provide a severely blindfolded mode to WinBoard, where it would not do update any graphics at all. But I think the main timing loss occurs because the OS might not immediately start up a thread that is listening to a pipe if another thread writes something in the other end of the pipe. You would need a clever OS, and clever setting of the various thread priorities to make sure there are no such scheduling delays. Using intermediaries like Polyglot or InBetween multiply the problem.

On the sunny side: if 75% of the wall-clock time is wasted because of communication delays, this 75% does not tax the CPU cores. So you can play 4 games at once on a single core, without getting a slowdown. The engines should just base their timing decisions on CPU time they used, not on wall-clock times, and should ignore what the GUI says. In WinBoard this is quite possible (suppress the time and otim commands, and play without /autocallFlag).

[F. Bluemers]

Thanks Marc,very interesting and it works over here.
At these fast controls the winboard graphics are slowing it down(ymmv).
making these changes in winboard.ini fixes that:
/animateMoving=true -> false
/animateDragging=true -> false
/moveHistoryUp=true -> false
/evalGraphUp=true -> false
/engineOutputUp=true -> false

Best
Fonzy

[hgm]

You can also use /blindfold, to suppress the drawing of pieces completely.

Tournament managers typically start up a new WinBoard (which in turn starts up new engine processes), which causes a lot of overhead. The alternative is to use WinBoard in match mode (/matchGames=N) while re-using the engines, so that many games can be played once the processes are started. WinBoard pauses between the games, though, because engines can often not be interrupted once they are thinking (e.g. if the opponent resigns or claims draw after his move). To limit this waiting tim, you can use /matchPause=<msec> in WinBoard_F. But if you set it too short, often a move from the previous game of an engine that could not be interrupted ends up in the next game, and is flagged as an illegal move. Only engines that support 'ping' offer a true soultion to that. When both engines support 'ping', you can make the /matchPause as short as you want; WinBoard always waits for the corresponding 'pong', and then immediately starts the new game.

[Andres Valverde]

Hi Marc, will have a look at your tool

I usually use Pradu's TM and Winboard X with every graphic feature disabled and fixed depth. At 4 ply depth it plays 7-8 games/min and 5-6 games/min at 5 ply. It can play 3 seconds/games though

Thanks and will inform asap

[F. Bluemers]

You can also use /blindfold, to suppress the drawing of pieces completely.

Thanks,works great
And for another minor tweak
/showThinking=false
to surpress the pv printing.
I guess thats about it for simple tweaks.
Best
Fonzy

[bob]

Vasik Rajlich and Larry Kaufman recently revealed on the Rybka forum that they test slight modifications of Rybka through the running of a very large number of ultrafast games : 80,000 three-seconds-total-duration (!) games played on two quads each night. This allows to detect improvements/worsenings of their engine in the one elo point range ...

For those who would like to go on this way toward ultrafast testing, no software I know of (free or commercial) is able to manage these kinds of tournaments and games.

In fact Rajlich and Kaufman designed a specific tool for this but they do not intend to make it publicly available.

So I tried to put together something of my own. This is not completely trivial under windows because as you go faster you have to deal with aborted or uncorrectly finished threads and processes after each game.

The resulting little tool is a kind of preprocessor : you define parameters in a small text file, the utility takes it and produces an enormous batch file. You launch the batch and the tournament is run. The process makes use of WinboardX (Tim Mann / Alessandro Scotti), Polyglot 1.4 (Fabien Letouzey) and of several freeware utilities dealing with threads and processes. I only tested it under XP64 but I have been told it also works under 32bits XP.

I have put together all required utilities in a package available at http://chessbazaar.mlweb.info.

The fastest I could go with it is a little more than four complete games per minute. At this rate more than 97% of games are played successfully with most top range engines (Zappa Mexico 1 being an exception with a 25% failure rate).

If you stay more reasonably at a duration of 30 seconds per game the success rate is close to 100% with all engines.

This is still far of the stunning testing scheme that Larry Kaufman uses but not too bad yet IMHO.

Last word : this is an amateur non-polished package for those who are not afraid to deal with winboard commands and parameters. It comes with a kind of manual and example config files but I will not be able to give more detailed explanations.

Marc Lacrosse

I am not sure how accurate that is. I can run 80,000 such games in a few minutes, but the variance is so high when the operating system timer resolution is such a large percentage of the target time for a particular move. If your target time is 10ms for a move, and the operating system timer runs at a resolution of 18ms (the PC) then the variability from move to move is incredibly high. A large number of games is certainly a step in the right direction based on the testing I have been doing for a couple of years now. But very short (and non-increment) games are not very trustworthy, because you are really testing with a high randomness factor built in...

I have made changes that looked good at _very_ fast time controls, but which seriously hurt Crafty at longer time controls... That is a scarey thought...

BTW xboard has been able to play game/1sec type games since I have been using it. I have reported lots of game/1sec matches over the years, although games at that speed have little to do with how a program will perform at long time controls...

[CRoberson]

I've seen the same. James and I discussed this on our way to
UAB last October.

A partial list of parameters that can easily look good at blitz but bad
at long time control:

• King Safety tunings
  Passed Pawn tunings
  Search parameters

Basically, anything that makes it more aggressive tends to perform
better at bullet and fast blitz than at long TC's.

That is why I run both blitz and long TC testing on new versions.

I perceive extremely fast TC's as good for crash testing.