To GUI developers and Linux engine packagers

[hgm]

I think I coined ideas like this before, but my thoughts on this have crystalized somewhat more, so I want to rehash it.

For a long time it has annoyed me that GUIs require 'installing' of engines before you can use them effectively. (The proper terminology would really be 'configuring the GUI to use the engine', as 'installing' really refers to copying the engine files to your computer.) This goes against the philosophy of Linux packaging, where you want to just "sudo apt-get install" (or your distro's equivalent) and be completely done.

I want to improve that situation. But this requires the introduction of some new standards, and I cannot do that on my own. So I hope that you all want to cooperate to achieve this, by observing the following conventions for installing engines, and equipping GUIs with code to make use of it:

In the following I will use GAMESDATADIR as a symbol to indicate the place where architecture-independent data for games would be stored on the applicable Linux distro. For Ubuntu / Debian this would be

/usr/share/games/
or
/usr/local/share/games/

depending on whether you installed from the repository or from source. My first proposal would be for everyone to pack a (100x50 or 130x65) logo png image with each engine, and install it as

GAMESDATADIR/plugins/logos/<enginecommand>.png

That is, outside the normal data sub-tree for the engine (where it would store books and ini files), which would be at GAMESDATADIR/<packagename>/ . Note the use of <enginecommand> rather than the <packagename>; engine packages can contain binaries for multiple engines (e.g. hoichess and hoixiangqi are in the same package), so that a GUI cannot always know the package name for a given engine. The <enginecommand> does not include any arguments that might have to go on the command line. (In C terms, it is argv[0] only.) So for Fruit the file would be fruit.png, for Crafty it would be crafty.png, etc.

GUIs that want to display logos are then encouraged to look for them in the system plugin logos directory. I will configure the next XBoard release to use this system-wide logo folder rather than one in its own sub-tree.

Then I want to introduce a second convention as well, to make it possible for Linux GUIs to automatically 'install' engines in their engine list. This involves new standard directories

GAMESDATADIR/plugins/<protocol>/

where <protocol> can be "xboard" or "uci" (for starters; any protocol could be added). Engine packages are expected to install a file <enginecommand>.eng here. Any GUI could then search the directories for the protocols it supports (e.g. XBoard would scan the "xboard", "uci" and "usi" directories), at startup or on user command, to see if engines have been installed that it was not yet aware of, and add those (automatically or after user confirmation) to the engine list of that particular user).

For now the contents of the *.eng files would consist of only two lines:

Code: Select all

plugin spec 0.0
<commandline>

where the first line acts like a 'magic phrase' to help Linux recognize the MIME type of the file, and supply a number that will help the GUI interpret the info in the rest of the file, which is subject to future extension. The <command-line> is the command needed to start the engine for this protocol. (E.g. "pos --io-mode xboard".) If an engine can run under more than one protocol, it would simply place multiple .eng files in the applicable directories, possibly specifying different startup commands.

In the future this could be extended to contain other information about the engine that would be generally useful to GUIs. Such as games/variants it supports (so the user could for instance limit automatic install to variants he is interested in), how it uses leeway granted to it by the protocol (i.e. whether it prints its PVs as SAN or long algebraic, reports score as own or white POV), non-compliancies with the protocol. (Why require people to admit non-compliancy rather than to fix it? Well, GUIs often have work-around options that could cure them, that now have to be applied by the user, but through this mechanism could be applied by the package maintainers without requiring involvement of the engine programmers, who might no longer be active.)

Obviously standards will have to be developed on how to convey such information. This would be a good place to discuss that. For now this minimalistic version 0.0 of the 'plugin-spec standard' allows us to already start using this mechanism.

I really hope that all developers of Linux GUIs will add support for automatic installing of engines through these plugin/logo and plugin/<protocol> standards.

[JoshPettus]

Please, PLEASE use PNG for engine logos. Not JPEG, GIF, TGA or whatever else an author may feel inclined to use. Sometimes I come across some really strange formats and I can't help but feel the author is having a laugh. Oh boy, time to open gimp...

[sje]

There are more games than just chess, so an accommodation has to be made for them. Somewhere in the directory path should be the word "chess" and this could be substituted with "draughts", "go", etc. as needed.

I suppose that games which are sufficiently close to orthodox chess like Chess960 (FRC) could have their files reside in the chess path without too much confusion.

Also living somewhere in the chess path should be any repositories like tablebases, opening books, PGN databases, test suites, etc. These items should also be considered when attempting to unify chess engine placements.

[hgm]

There are more games than just chess, so an accommodation has to be made for them. Somewhere in the directory path should be the word "chess" and this could be substituted with "draughts", "go", etc. as needed.

This is exactly what I would want to avoid at all cost. The proposal is based on protocols, and whether uci is used for chess or xiangqi or xboard protocol is used for chess or draughts is really of minor importance. GUIs in general do support more than one game. The XBoard Alien Edition, for instance, supports Checkers, Reversi, Go, Amazons next to Chess. Some engines (like NebiyuAlien) support Chess, Xiangqi, Shogi and a lot more. The number of games and variants is vastly larger than the number of protocols used by GUIs, and it would become very inefficient, and a nightmare for GUI developers if they would have to scan directories for each possible game or variant they can support. XBoard now has the concept of engine-defined variants, so it can support variants that it has never heard about.

So the only role for "chess" in my proposal would be in a future extension of the plugin-specs file format, where the third line could contain a list of the supported games (after agreement is reached on some standard naming). Like

chess,chess960

(for many Chess engines) or (for HaChu)

nocastle,shoshogi,chushogi,makruk

or (for Pulsar)

chess,chess960,suicide,losers,atomic,threecheck,twokings,shatranj

or (for Shokidoki)

shogi,minishogi,judkinsshogi,torishogi

or (for Fairy-Max)

chess,capablanca,spartan,makruk,knightmate,cambodian,ai-wok,courier,greatshatranj

What I imagine for XBoard (but that is just a matter of GUI design, apart from any standards) is to have a string option -autoInstall, which a user could set to (say)

-autoInstall "chess,crazyhouse"

if he is only interested in engines that can play those games, so that these will automatically be added to his engine list, first time he runs XBoard after the engine package is installed. To make his life easier that option would also understand some names for variant groups, like ALL, CHESS, SHOGI, ASIAN, WESTERN, LARGE, SMALL, NONSTANDARD.

GUIs that only support Chess obviously would not need such an option, and will discard all engines that do not contain "chess" in the list.

[stegemma]

You have a standard protocol to let programs talk together (xboard/UCI) and you want to standardize the way one of the programs (xboard) find the other (the engines), to start talking. This sounds very similar to Windows OLE or the old DDE. Forgetting about DDE... what is the counterpart of OLE in Linux environment? There are other standard protocols to let programs talk togheter?

Maybe the correct answer is not to invent another "protocol" or folders convention or something similar but is to find a way to encapsulate xboard/UCI protocol to a standard inter-program communication protocol?

[hgm]

I am not completely sure what you mean. Could you give a simple but specific example that implements what you have in mind?

The main problem I want to solve is not how programs should talk to each other, but how they can know who to talk to. Before a GUI can use UCI, it must start up a UCI engine, and just starting up every executable it can find and sending it "uci" to see how it responds seems a very inefficient way to to it (with most likely disastrous side effects). So engines most somehow announce themselves to GUIs, but the engine has no idea which of the installed programs are GUIs that could understand UCI. Should they install themselves in the settings file of XBoard, eboard, SCID, Omaha? It seems an unfair burdon on engine packagers to require their install script supports every existing GUI. And what if a GUI is installed after the engine?

BTW, since the vast majority of GUIs and engines using XBoard protocol or UCI only support chess, and perhaps chess960, it might be a good ide to already require a third line in the 0.0 plugin-spec files, which should contain "chess", "chess960", "chess,chess960" (all without the quotes) or be empty when they support neither of those. Then we can defer the problem of standardizing other game or variant names to later versions of the standard, but engines that only play chess or chess960 would then already be fully compliant with this extension. So with the fairymax package I will ship a file

/usr/share/games/plugins/xboard/fairymax.eng

containing the 3 lines

Code: Select all

plugin spec 0.0
fairymax
chess

as it doesn't play Chess960, and we don't care about all the other stuff it plays.

[stegemma]

As you know, in Windows you can register an object and then call its method or access its properties. It's a pain but this give you the opportunity to run an engine and ask him only knowing its name. I've used OLE to call AutoCAD, for sample. This is the cleanest standard way to register a software that can act as a plug-in, in Windows. It is not the simplest but it works. The same is for the "old" DDE interface.

In Linux, there are similar standard but i'm not an expert. DCOM or CORBA and similar, could give a way to "register" an engine so that it could be called from any other process, maybe only to know if it is an engine.

I think that one should use this standard, to ask something to an uncknow program, when you only know its name. In Windows you can enumerate registered OLE/DDE providers, in Linux i don't know.

All of this could be only academical, because is a lot of work, for a chess engine, and a lot of complexity to handle. Maybe there are some simpler standard, in Linux, that help registering a software and knowing its capabilities.

[hgm]

Thanks, I will study your proposal. Which will take some time, as I don't even know what OLE is.

As for a simpler standard, I think this is basically what I am proposing: a sort of registry exclusively for engines using xboard protocol, and another one for uci engines. And there is no complexity at all: the only thing the engine package would have to do is to copy a very small file to a certain system directory. This does not require any programming or modification to the engine, so that packagers/maintainers can do it on their own even with closed-source engines.

In trying to implement an -autoInstall option in XBoard, however, I ran into the problem of engine directory. In my proposal above the .eng files did not include a line to indicate in which directory the engine would have to be started. I did not think of that, because compliant Linux engines would run in any directory, and know where to find their files by themselves (in /usr/sahare/games/<packagename>). But we should of course not assume all engines are compliant with Linux standards; some could be imported Windows engines (even running under wine), which expect files in the current directory.

One way to allow engines to specify they would only work when launched in a certain directory would be to prefix their engine command with a cd command:

Code: Select all

plugin spec 0.0
cd <directory>;<engine command line>
chess,ches960

An alternative would be to reserve an extra line in the plugin-spec format for the directory. But I like that less, because it would force compliant engines to mention a directory, or leave the line empty so that it is not obvious what its purpose is, and invite non-compliancy in general, because once the line is reserved people will feel compiled to actually use it, etc. So the optional 'cd' command as prefix, only needed on non-compliant engines seems a better solution.

The risk, of course, is that the latter notation might suggest that people could just put any semicolon-separated sequence of shell commands in stead of a single engine command line, while this is a feature that most GUIs would not support.

[kbhearn]

The equivalent to OLE in linux is D-Bus, or rather usually services layered on top of d-bus. The applications interested in some form of interoperability use it with a standard name to pass messages with each other or with a service. I do not believe this suits your purpose, but it's not something i've worked with myself so i could be wrong (i suspect to function properly you might need some sort of separate service program that handles the matchmaking). Even in windows i don't believe you can use OLE (or whatever the current standard is) to ask the question : are there any unknown applications that happen to do X installed? You can ask for a copy of excel to open and interact with it through its exposed interfaces. But you can't discover that 'excel isn't installed but joe's spreadsheet program is and supports the same functionality'

With regards to your original proposal, a couple questions:
1) what other interfaces for linux are actively developed that you're hoping would agree to your proposal?

2) is a file per engine preferable over a single file modified by script by the package maintainer?

[hgm]

1) I was hoping for SCID (vs PC). And Yann Dirson is working on a new interface, 'Omaha' (which is a multi-game interface like Tagua). I think there are several Linux Chess interfaces that are open source, and even if they are not actively developed now, it does not mean they never will be in the future. Also, OS X is related to Linux, and the HIARCS Chess Explorer could benifit from a standard too. I don't know if there are other commercial interfaces available as specific Linux versions, but if there are, these could benefit from it too. Especially if there is public-domain code available for a generic auto-install procedure.

2) A script to edit a file seems more cumbersome than just installing a file, which could make use of existing standard installation procedures and OS file-system handling. I don't think it would differ very much in efficiency, as directories are basically also just files of entries.

I discussed another system before with a Debian maintainer, where engine packages would edit the /etc/xboard.conf file to 'broadcast' their presence. (This system is actually implemented and operational in the XBoard development version.) To make the script trivial, XBoard was equiped with command-line arguments to do the required editing, in a very generic way. (The command xboard -addMasterOption {any text} -autoClose would append a timestamped line with the text to its master file, so the script would only have to issue that command. And the command would be a new option -installEngine "xxx" which would append a line "xxx" to the -firstChessProgramNames list if the time-stamp suggested it was the first time it saw it.) The bad feature of this is that it is very dependent on the presence of XBoard, and the formats of its data files. A more general format would require a more specific script to edit it, however, as it could not rely on any specific GUI to know the details of how to edit, to which it could delegate the task. So every engine package would then have to include a non-trivial script, which might discourage the engine packagers to take the trouble, and thus slow down acceptance of the standard.