Crafty problem with cutechess-cli

[bob]

Which is my point. During development, I try to use my time as efficiently as possible, because it is a limited resource. Remembering to restore every value to its starting point requires effort that can be used elsewhere. And I have elected to do so. I'm not aware of any issues in Crafty with the "new" command. I simply pointed out that _I_ prefer to do all testing of _all_ engines without depending on this so that I don't introduce a little unexpected bias to the results.

Perhaps you should re-read the first post in this thread then.

Last I heard, Crafty was able to play as many games as you want, not mysteriously getting weaker as the games are played. So it does what I wanted it to do. The log-file issue is beyond tiny, although I have fixed it in the current code.

Idea occurs in no programming book I own, and shouldn't. There are reasons for run-time initialization vs compile-time initialization. And they don't have anything to do with whether the value is modified during execution or not.

Sure there are. Say your board representation is based on an array int board[64]; (ignore the fact that this is bad design in itself). Do you want to instantiate it at compile time, or have a function to set up the initial position? If you do the former, then you're going to need to duplicate the data to do the latter. Much better is to simply call said function to initialize it in the first place.

Depends. If this application runs one time and dies, I will initialize the data in the most resource-friendly way. If I initialize at run-time, it takes execution time to deal with this. If I initialize at compile time, it takes way more disk space and thus may take more time to load on a slow disk machine. I once had a version of Crafty that didn't need to initialize the EGTB decompression indices, which is what takes so very long. I just hard-coded those. But ran into trouble when not everyone had _every_ file for reasons of their own choosing, and found it easier to continue doing this at run-time since the egtb file set can be changed at any point in time.


For chess, if I were going to play one game, I would initialize the board at compile time. If I were going to play more than one game, I would initialize at run time. I initialized read-only data in Crafty at compile-time for some things, at run-time for others.

So the decision has nothing to do with whether or not the data is modified, it is simply which way of initialization is the easiest or most efficient way of doing it.

Likewise, if you have a variable, say "use_log", that should persist across games, you probably don't want to initialize it in the same function you use to start a game. Set a default either at compile time or at startup and don't touch it until the user says so. Doing otherwise is simply a bug.

[krazyken]

The log-file issue is beyond tiny, although I have fixed it in the current code.

Thank you.

[ernest]

The log-file issue ..., although I have fixed it in the current code..

Thanks!
Which version will that be?

[bob]

The log-file issue ..., although I have fixed it in the current code..

Thanks!
Which version will that be?

23.1 and all versions beyond...

My advice is to still not use log=off, but instead add a clean-up script to the testing harness so that all the log/game files get removed after the match is completed. You would not believe how many reports of problems I get, and when I ask for a log.nnn file there is none to look at, making any sort of debugging impossible. Most of the time we track these problems back to the opponent engine, but without a log there is nothing I can do until the problem is replicated with the logs enabled...

I generally test with 'em on, and remove 'em regularly.

[bob]

What a silly statement. I add a new feature for testing. I now need to add code to make sure it gets re-set on a new command. I change the code. I have to change the new code. If you do something "trivial" enough times, the total time suddenly becomes non-trivial. Most people don't complain when their compiler or other large application won't just sit around waiting on the next program to compile, but instead has to be re-loaded into memory for execution.

Most people would complain if their compiler didn't allow this:

Code: Select all

cc *.c

but instead required:

Code: Select all

cc a.c
cc b.c
cc c.c
...
cc *.o

I wouldn't complain. I wouldn't have to change anything in any makefile I've ever written (assuming the -c option which you left out is used the same as it is now).
Also, from my understanding of GNU's gcc, the program you run is not the actual compiler, but invokes the appropriate compiler (and/or preprocessor, assembler, linker, etc) exactly once for each output file, which would be akin to invoking the chess engine exactly once for each game.

I have worked on several assembler/compiler projects over the years, and I _never_ wrote one that could just restart itself and compile the next file. There may well be compilers that do that, but nothing I have worked on the internals of did that. Too much stuff gets changed over the course of a single module compile to try to restore everything to initial state for the next module.

But there are other cases of things you are not even considering. take a cluster run, where I want to vary some value over a range, so I add some code in option.c to handle a "scale" command (this code is actually in there right now, but does nothing for released versions. If I enter scale 50, that means to take the default value (or values) and reduce them to 50% of their original value, and then play a game. A "new" is problematic here, I need to restore the values to their original state. Or I can just restart the engine.

Sorry, but this sounds to me like a bad design. If you run "scale 50", it divides every evaluation parameter by 2? A much better way would be to simply divide by 2 at the end of the evaluation. But even then, I have a function that takes a set of evaluation parameters and makes a copy, so that you can restore it later. This was created for tuning parameters. Very trivial code.

In general, if you have data, you either initialize it explicitly in the instantiation, or you calculate its value in a function. Values that are modified during a game should obviously be initialized in a function. Values that are constant throughout a game can be initialized either way.

Don't put something down until you have considered all angles. Things are not always as cut-and-dried as you might assume.

I have. I honestly can't think of one good reason why a well-designed chess program would need to restart every game (unless you are HG and need to minimize characters).

[Zach Wegner]

Last I heard, Crafty was able to play as many games as you want, not mysteriously getting weaker as the games are played. So it does what I wanted it to do. The log-file issue is beyond tiny, although I have fixed it in the current code.

Then I suppose you might say that it is easy to fix? And that the code to start a new game is trivial?

Depends. If this application runs one time and dies, I will initialize the data in the most resource-friendly way. If I initialize at run-time, it takes execution time to deal with this. If I initialize at compile time, it takes way more disk space and thus may take more time to load on a slow disk machine. I once had a version of Crafty that didn't need to initialize the EGTB decompression indices, which is what takes so very long. I just hard-coded those. But ran into trouble when not everyone had _every_ file for reasons of their own choosing, and found it easier to continue doing this at run-time since the egtb file set can be changed at any point in time.


For chess, if I were going to play one game, I would initialize the board at compile time.

That seems to be what you are advocating. You are basically saying Crafty doesn't officially support playing more than one game. You might as well ignore the "new" command, and initialize everything at compile time. Why give the impression that you support playing multiple games (BTW, I looked at the latest Crafty source and it sends reuse=1 to xboard), when playing more than one game might (and in this case, did) give buggy behavior? I could understand better if you were to say that Crafty is only designed to play one game in order to keep the code clean, initialize everything at compile time, and send reuse=0. But in Crafty there's simply a bunch of option variables reinitialized in the NewGame() function, which IMO should definitely not be there.

I also looked at the code to handle log files. There isn't a variable to keep track of the option at all, but the FILE pointer gets set to 0 when it is turned off, making it indistinguishable from the initial state. Again, doesn't seem like a good design to me.

[bob]

Last I heard, Crafty was able to play as many games as you want, not mysteriously getting weaker as the games are played. So it does what I wanted it to do. The log-file issue is beyond tiny, although I have fixed it in the current code.

Then I suppose you might say that it is easy to fix? And that the code to start a new game is trivial?

You keep trying to read between the lines for some sub-plot or something. I clearly pointed out why I do not like the reuse approach. It is too easy to make changes and forget to do the necessary initialization / re-initialization for values that would need this. I generally write code the fastest way I can to get to the test point as soon as possible. For ideas that succeed, I go back and clean them up later, and it is not that hard to forget the "new" case.

So the -xreuse option solves _all_ problems. Not using it can introduce one or more problems. Hence my position on the reuse discussion. Avoid the problems.

Depends. If this application runs one time and dies, I will initialize the data in the most resource-friendly way. If I initialize at run-time, it takes execution time to deal with this. If I initialize at compile time, it takes way more disk space and thus may take more time to load on a slow disk machine. I once had a version of Crafty that didn't need to initialize the EGTB decompression indices, which is what takes so very long. I just hard-coded those. But ran into trouble when not everyone had _every_ file for reasons of their own choosing, and found it easier to continue doing this at run-time since the egtb file set can be changed at any point in time.


For chess, if I were going to play one game, I would initialize the board at compile time.

That seems to be what you are advocating. You are basically saying Crafty doesn't officially support playing more than one game.

I've not said any such thing. I've said that way of running any program is preferable. I try to make "new" work. But I do not test that way and so I do not claim that it works perfectly in every version. I do claim that by using -xreuse, you will _never_ have a problem with Crafty in any game. It won't crash. It won't create log.nnn files if you tell it not to. It just runs solidly each and every game.

You might as well ignore the "new" command, and initialize everything at compile time. Why give the impression that you support playing multiple games (BTW, I looked at the latest Crafty source and it sends reuse=1 to xboard), when playing more than one game might (and in this case, did) give buggy behavior? I could understand better if you were to say that Crafty is only designed to play one game in order to keep the code clean, initialize everything at compile time, and send reuse=0. But in Crafty there's simply a bunch of option variables reinitialized in the NewGame() function, which IMO should definitely not be there.

I also looked at the code to handle log files. There isn't a variable to keep track of the option at all, but the FILE pointer gets set to 0 when it is turned off, making it indistinguishable from the initial state. Again, doesn't seem like a good design to me.

[sje]

Symbolic's logfile design and operation:

The C++ portion of Symbolic has a LogFile class that handles all logging operation. The class is single instanced, so there's always a single logfile and the file name default is "Symbolic.log". If the user doesn't want a logfile, there's the "-l" command line option that renames the logfile and its operand can be set to "/dev/null". The logfile object is created very early, immediately after command line option parsing. A fault during logfile creation or operation causes a diagnostic to appear on the standard error output and also on a forensics file named "CauseOfDeath".

The logfile is opened in create mode, not append mode. This is the reverse of the earlier version and removes the need for manual truncation.

Because Symbolic is multithreaded and each thread is able to access the logfile object, the logfile object uses a mutex to keep things in order. Every line in the logfile is prefixed with a UTC timestamp with millisecond resolution. All text input and (almost) all text output is echoed to the logfile with an appropriate directional indicator "<" or ">". Each line output to the logfile is immediately followed by a flush as a guard against an unexpected termination.

At the conclusion of the program, the logfile object's destructor cleans up and closes the file.

Symbolic takes a similar approach to recording PGN scores. There is a single instanced class CommProc that includes a PGN text output stream (default: "Symbolic.pgn") , and unlike the logfile, the PGN file is opened in append mode. The CommProc class has a GameOver() method that writes the complete PGN followed by a flush.

So, only a maximum of two output files no matter how many games are played.

[ilari]

Finally back from vacation..
The reason cutechess-cli doesn't support engine restarts (reuse=0) is not philosophical. We just haven't implemented this feature yet. We're definitely going to implement it later, and there will also be an option to override/ignore the engine's preference.

[hgm]

I do claim that by using -xreuse, you will _never_ have a problem with Crafty in any game. It won't crash. It won't create log.nnn files if you tell it not to. It just runs solidly each and every game.

You will _always_ have the problem that is slows you down by having to re-load all the time...

Of course when you don't run Crafty at all, you will really _never_ have any problems with it. But is that really a virtue?