engine-engine testing isues

[JBNielsen]

Regarding the delay betweengames, ive never seen this!

Now I have turned off 'restart engines before each game'.
The gap between the games is now only 10 seconds

- - -

Thanks for all your input here.

Noone has mentioned the option to play the same openings (in the same order) in every match...
Maybe this can be solved by using a pgn/epd file as input - combined with the option to play every line/position with reversed colours?!

[Adam Hair]

Regarding the delay betweengames, ive never seen this!

Now I have turned off 'restart engines before each game'.
The gap between the games is now only 10 seconds

- - -

Thanks for all your input here.

Noone has mentioned the option to play the same openings (in the same order) in every match...
Maybe this can be solved by using a pgn/epd file as input - combined with the option to play every line/position with reversed colours?!

Yes, you can do that with Arena 2.01 and later versions. I assume that you can do it with Arena 1.1

[hgm]

Noone has mentioned the option to play the same openings (in the same order) in every match...
Maybe this can be solved by using a pgn/epd file as input - combined with the option to play every line/position with reversed colours?!

Yes, to repeat the same opening suite every gauntlet, it is best to play from a PGN file with opening lines.

You can extract such a file from a book by making an engine that resigns as soon as it gets a 'go' command, and then play a big match between two copies of the engine where you set the book depth to the desired number of moves. This is what I do for my mini-Shogi engine, where nothing exists, and I have to generate everything myself. So I first created a book by doing tens of thousands of fast self-play games in WinBoard's Monte-Carlo book creation mode (which plays the moves with the best score most often), and then extracted a file with opening lines from that book.

[Sven]

Regarding the delay betweengames, ive never seen this!

Now I have turned off 'restart engines before each game'.
The gap between the games is now only 10 seconds

If your engine uses the WinBoard protocol then there are ways to drastically reduce the delay between two games, especially if your engine does not need much time for its initialization:

- Start WinBoard with the "/matchPause <msec>" option. When playing with fast time control I use a very small value, e.g. 100 msec. The default is 10 seconds (10000 msec).

- Let your engine send "feature done=0" at early startup, and later "feature done=1" when replying to the "protover" command. This helps to adapt the startup time of your engine to almost exactly the time it needs for its initialization, instead of the default which is 2 seconds IIRC.

The combination of both should be better than using "/reuse" if you want to play many short games.

Sven

[hgm]

WinBoard has the option -first/secondProtocolVersion, which can be used to inform the GUI that the engine uses protocol v1, so that it won't wait for the engine to send any features. (I think in WinBoard the feature timeout is even 10 sec.) In WinBoard's Load Engine dialog there is a checkbox you can tick to install this option with an engine.

This way you can conveniently run engines that do not send features, without the need to alter the engine.

I thought that Arena had something similar.

[JBNielsen]

I run engine-engine tests with arena 1.1 and mainbook.

It is very stable, but these things could be better:

1) many opening lines are much too long. Up to ca. 30 moves. 8-10 moves would be fine.
2) Some lines has a big or even winning position for white or black. Almost even positions should be played.
3) The system uses 30 seconds from a game ends until computing of a move in the next game starts.

As I only have limited computer capacity, I rarely play much more than 100 games (yes I know it is much too few).
But to give a better comparison of the 2 engines and comparisons with earlier matches, I would like:
4) The same openings (and in the same order) were played in every match.
5) Every opening were played twice with reversed colours.

Any suggestions that can give a better testing?

Arena can do everything of the above.

1) You can limit book size to any depth you wish (see Michael's post).
2) There are much better books for download than the package mainbook. E.g. Perfect 2012 by Sedat Canbaz which I use regularly.
3) I don't really understand that. Anyway, you can choose the time between games at the tournament settings.
4) Use a pgn of your choice for openings and tell the program to use the lines sequentially.
5) There is a setting for that, too.

Thanks.

Are you referring to version 3.0 of Arena?

PS
in an earlier post I wrote this to 3) :
"Now I have turned off 'restart engines before each game'.
The gap between the games is now only 10 seconds"

[JBNielsen]

You can extract such a file from a book by making an engine that resigns as soon as it gets a 'go' command, and then play a big match between two copies of the engine where you set the book depth to the desired number of moves. This is what I do for my mini-Shogi engine, where nothing exists, and I have to generate everything myself. So I first created a book by doing tens of thousands of fast self-play games in WinBoard's Monte-Carlo book creation mode (which plays the moves with the best score most often), and then extracted a file with opening lines from that book.

Thanks. A funny and tricky way to do it.

I will first try to find a file that is ready to use.

[JBNielsen]

I run engine-engine tests with arena 1.1 and mainbook.

It is very stable, but these things could be better:

1) many opening lines are much too long. Up to ca. 30 moves. 8-10 moves would be fine.
2) Some lines has a big or even winning position for white or black. Almost even positions should be played.
3) The system uses 30 seconds from a game ends until computing of a move in the next game starts.

As I only have limited computer capacity, I rarely play much more than 100 games (yes I know it is much too few).
But to give a better comparison of the 2 engines and comparisons with earlier matches, I would like:
4) The same openings (and in the same order) were played in every match.
5) Every opening were played twice with reversed colours.

Any suggestions that can give a better testing?

Arena can do everything of the above.

1) You can limit book size to any depth you wish (see Michael's post).
2) There are much better books for download than the package mainbook. E.g. Perfect 2012 by Sedat Canbaz which I use regularly.
3) I don't really understand that. Anyway, you can choose the time between games at the tournament settings.
4) Use a pgn of your choice for openings and tell the program to use the lines sequentially.
5) There is a setting for that, too.

You are completely right about Arena 3.0.

Even the gap between the games is zero now; Arena 1.1 played the opening moves a little slowly although I had turned sliding pieces off.