Distributing engines for Mac/OSX

[JoshPettus]

I know I'm fairly late to much of the conversation. But as I tried to point out in the last post. It easy to come up with a container folder for the engine bundle (call it something unique like .cengine), and associate it with xboard.app. Then clicking on the container we could open it with xboard, and have xboard register the engine correctly. No third/second party installer required.

Aside from that I kind of agree with everyone else. Let people do what they want. It really isn't hard to select the engine executable in xboard. It's what people have done for years in windows, and has anyone ever felt the need to change it there? In simplifying that process, we very well may be making the whole thing more complicated. What engine developer is going to make these special xboard engine packages? It's hard enough to get just a binary out of them, and most of them will never be aware of the intricacies of whatever system we create for xboard. People that do download such special bundles are going to look at them and think what the heck is this? I'm sorry Harm, but in a system where software isn't managed by a package manager, this whole plug-in thing really isn't as important.

I just suggest that we add the engine folder as a possible place to look for a manual, we already look for the logos there, and it's a logical place to look if someone wanted to not mess with the install scripts.

[hgm]

You are never going to get buy-in for such a packaging standard. Too small a niche. Only Mac power users would ever use the Unixy things you mention, like man pages, and they would be nimble enough to find alternate documentation and configuration.

So I fail to see the problem. It is easy to add UCI engines to Hiarcs Explorer, my preferred Mac client. As to organization, I collect engines one per directory in an "Engines" folder I put in the Applications folder.

There is a weakness in Hiarcs currently: you can't specify a custom engine command line. I would like to use polyglot within Hiarcs for two purposes: 1) a shim for the few engines that don't yet support UCI, and 2) a shim to provide an opening book for the increasing number of UCI engines that don't support OwnBook. However, you either need to recompile polyglot to load a particular custom config file and engine, or specify such file on the command line. HIARCS can't do the latter, though I've submitted a feature request.

The only ones that would have to "buy in" to the packaging standard are people hosting Mac compiles of engines, and GUI programmers. There don't seem to be too many of either, so that is a good thing.

What is good enough for HIARCS Explorer is not necessarily good enough for XBoard. There often is a large gap between what others think good and what I consider acceptable. You might think it 'easy' to add a UCI engine to HIARCS Explorer, but I consider it an abomination that you would have to do it at all. Standard behavior is that it would just be there, without you doing anything at all.

BTW, it is perfectly possible to use Polyglot without arguments, and in the old days people the required method was actually very popular: just copy the polyglot.exe to the engine directory, and put a polyglot.ini there too. The only downside is that you will have many copies of polyglot.exe around. But by modern standards Polyglot is quite small, and on a Unix-like system I suppose you couldlink to it, rather than copy, so that you don't even have that problem.

[JoshPettus]

Polyglot may be a separate project with separate goals then xboard, but I nor most anyone here would ever install xboard without it. It is a key component of what really is the xboard suit of programs. It would be like TCP/IP and not having UDP/IP, they may have separate goals but they compliment each other. Improved integration between the two benefit the whole. The divide between engine and GUI is a lot more pronounced has been part of Xboard's original mission goal. So I don't equate that to be the same.

From a practical standpoint I just don't see it as simple a standard to implement in windows and osx as it is in linux. Heh, in a perfect world we would all be using linux XD. Package distributors are one thing, but getting the GUI authors on board something else. If Hiarcs and Chessbase don't do it this way, how can we expect users to get on board?

[hgm]

I know I'm fairly late to much of the conversation. But as I tried to point out in the last post. It easy to come up with a container folder for the engine bundle (call it something unique like .cengine), and associate it with xboard.app. Then clicking on the container we could open it with xboard, and have xboard register the engine correctly. No third/second party installer required.

What I consider less than ideal with this method is that it treats engines like they are only associated with XBoard. I think it would be better to have a method that is universally valid, can also be used by people that do not have XBoard at all, and requires as little support as possible from the GUI in use. Other GUIs will be much quicker do exploit a given fact that all engines reside in /Library/ChesssEngines, than they would be willing to implement a signal handler that would process an engine bundle.

Aside from that I kind of agree with everyone else. Let people do what they want. It really isn't hard to select the engine executable in xboard. It's what people have done for years in windows, and has anyone ever felt the need to change it there? In simplifying that process, we very well may be making the whole thing more complicated. What engine developer is going to make these special xboard engine packages?

What engine developer is going to make OSX compiles of engines at all? Not too many I know. To convince people to embrace the standard it is obviously important that they can see immediate benefits, and that it is not much work. Most authors would probably not be willing to make a man page for their engine. That is fine, that is his decision. The goal of a standard is to prevent that an existing manual would go to waste because the way people access manuals doesn't work because of non-standard naming.

I don't think that guidlines like "if you do include a man page, name in 'man6' and put it in the root folder of the bundle" would have to overcome broad resistance. Including a .eng file of three lines (possibly already given to the person making the OSX compile in the source tar ballof the engine), rather than discarding it, also doesn't sound like a real deal breaker.

The installer program/script would be the same for every engine, and we could make it available for download. If ordinary bash scripts would work in this context, it seems to me that it could be a 2-line script:

mkdir /Library/ChessEngines
mv . /Library/ChessEngines

It's hard enough to get just a binary out of them, and most of them will never be aware of the intricacies of whatever system we create for xboard. People that do download such special bundles are going to look at them and think what the heck is this? I'm sorry Harm, but in a system where software isn't managed by a package manager, this whole plug-in thing really isn't as important.

I was under the impression that double-clicking a bundle would be somehow the standard thing people do with App bundles. And that would be all that is needed.

I just suggest that we add the engine folder as a possible place to look for a manual, we already look for the logos there, and it's a logical place to look if someone wanted to not mess with the install scripts.

Looking for a manual in the engine directory would be a pointless exercise if it would not have a standard name.

[JoshPettus]

What I consider less than ideal with this method is that it treats engines like they are only associated with XBoard. I think it would be better to have a method that is universally valid, can also be used by people that do not have XBoard at all, and requires as little support as possible from the GUI in use. Other GUIs will be much quicker do exploit a given fact that all engines reside in /Library/ChesssEngines, than they would be willing to implement a signal handler that would process an engine bundle.

That's a very fair point and one I was alluding to with Chessbase and Hiarcs.
As such.

I was under the impression that double-clicking a bundle would be somehow the standard thing people do with App bundles. And that would be all that is needed.

The only way this is implemented is if we put in handling code in XBoard. As someone said before, there isn't a standard for bundled resources and is done on an app by app basis.

Looking for a manual in the engine directory would be a pointless exercise if it would not have a standard name.

I thought it was EngineName.6.gz? If that's not the case, then you are very much right.

What engine developer is going to make OSX compiles of engines at all? Not too many I know. To convince people to embrace the standard it is obviously important that they can see immediate benefits, and that it is not much work.

My point exactly, chess players that use a GUI use Fritz (ChessBase) and Hiarcs primarily or even SCID vs Mac... If these don't do such a standard, I don't see all this work catching on. I'm sad to say XBoard just isn't that popular on OSX .

The installer program/script would be the same for every engine, and we could make it available for download. If ordinary bash scripts would work in this context, it seems to me that it could be a 2-line script:

mkdir /Library/ChessEngines
mv . /Library/ChessEngines

If you do this then you should know
/Library is a system folder, don't touch it.
~/Library is great for Applications to save their own generated content, but users do not have direct access by default.
It should be ~/Documents/ChessEngines. I don't really like it either, but it really is the only option for User resources that require user input.

[hgm]

The only way this is implemented is if we put in handling code in XBoard. As someone said before, there isn't a standard for bundled resources and is done on an app by app basis.

I don't get that. That these bundles are associated with XBoard is entirely coincidental. Most bundles would not be associated with anything, but be independent Apps. The XBoard App bundle we supply doesn't have to be associated with anything else, for people to use it, right? I was under the impression that people only have to click on it to launch it.

I thought it was EngineName.6.gz? If that's not the case, then you are very much right.

It would only be that if we say it is that. But it seems to be problematic. For instance, the Fairy-MAx bundel would contain three executables: fairymax, shamax and maxqi. Normally they share the man page. So how should we call it? On Linux we use "man -w maxqi", and it finds /usr/local/share/man/man6/fairymax.6.gz . But if the ma file is not in a place that the man command searches, we cannot rely on it.

So it seems better to always call it man6.gz. Just like a logo in the engine directory should always be logo.png to be recognized with -autoLogo, and only fairymax.png if you install it in /usr/local/share/games/plugins/logos/ .

My point exactly, chess players that use a GUI use Fritz (ChessBase) and Hiarcs primarily or even SCID vs Mac... If these don't do such a standard, I don't see all this work catching on. I'm sad to say XBoard just isn't that popular on OSX .

The point is that also for users of these GUIs life will become easier if they always find all engine directories in the same ChessEngines directory. I cannot imagine that there would be anyone that would insist that engine executables should be scattered all over the file system, to make life difficult for people to find them. (Not counting the Stockfish developers, of course.)

If you do this then you should know
/Library is a system folder, don't touch it.
~/Library is great for Applications to save their own generated content, but users do not have direct access by default.
It should be ~/Documents/ChessEngines. I don't really like it either, but it really is the only option for User resources that require user input.

It is hard for me to imagine that OSX would force every user that would want to use an App install his private copy of it. Is there really no way to share Apps, by putting them in a shared place that all users could access? I only mention /Library because it was succhested here. As far as I am concerned it could be anything, as long as it does not start with ~. If /Applications is the normal place where people drag the App bundles, couldn't the engines then be stored in /Applications/ChessEngines/ ? Heaping engines with other Apps would make it so much harder for the GUI to find the engines. It would not be as bad as in /usr/local/bin/ on Linux, however, where a binary can be anything (even a disk formatter) without you knowing it. Engine bundles could be recognized by a .eng file in their root directory, so a GUI could scan all bundles for such files (or for a sub-directory .../xboard or .../uci with such files in them). It could take an order of magnitude longer than necessary, however.

[JoshPettus]

Hmm, I think the best way to achieve what you want is .pkg installers. Like I did with xboard all those years ago, and we can use your existing plugin code. Personally I don't like them, there is no way to know what and where it throws on your system and there is no way to know you deleted everything it threw on if you wanted too.

But what you want is more Linux culture and philosophy then apple or windows.

And then there is the issue that all those GUI's I mentioned don't check the unix folders for engines. That's on them, but I don't see it changing anytime soon. Maybe SCID vs Mac if you ask them, but ChessBase and Hiarcs only care about their own products...

[hgm]

I am perfectly happy with the Windows model too, where all files needed by the engine go into the same folder. The point is that I don't want to have these folders scattered all over the system, because that makes it practically impossible for the GUI to find them. From what I understood bundles seemed ideal for this: folders that behave as executables. That would make them functionally equivalent to installers like we use for WinBoard, but far easier to make.

[JoshPettus]

Well bundles that behave like executables is what Xboard.app is. But apple treats these different then console executables. Specifically, it doesn't open it up with a console. So that's probably not what you want.

GPS shogi was released as an apple app that doesn't actually work when you double click on it. I just point XBoard to the app bundle's MacOS folder where the real executable is so it would.

When I was mentioning Engine.bundle, I was picturing self-contained files that XBoard can access and handle. Not really apps in of themselves.

[hgm]

Well bundles that behave like executables is what Xboard.app is. But apple treats these different then console executables. Specifically, it doesn't open it up with a console. So that's probably not what you want.

I don't want a console. I just want it to move itself to the place where we think it belongs (as opposed to where the user put it).

GPS shogi was released as an apple app that doesn't actually work when you double click on it. I just point XBoard to the app bundle's MacOS folder where the real executable is so it would.

I am not sure what exactly you mean by "point XBoard to the app bundle's folder". You add it to XBoard's engine list? Or you move a launch script for the GPS App to a place where XBoard was already configured to expect it?

This does sound pretty much like what I want. Except that I would not want the pointer to it within the XBoard App, but in a generally accessible place outside it. So that all GUIs are pointed to it.

When I was mentioning Engine.bundle, I was picturing self-contained files that XBoard can access and handle. Not really apps in of themselves.

What is still missing in my understanding is how such a bundle looks from the command-line interface (or file-system API). If the bundle root folder would be /Applications/XXX, can other programs then simply access files within it through a pathname like /Applications/XXX/bin/xxx.exe? Or is everything inside the bundel shielded when you look at the outside, like it would be if XXX was a zip file?