Broadcasting games

[hgm]

Design of a broadcaster

The broadcaster receives the GUI-engine communication through a pipe on its standard input. It has separate input and sender threads, to prevent that stalling in the sender due to slow network could stall the GUI. The input pipe must never be allowed to fill up, and anything the GUI sends must be read by the broadcaster without delay, no matter how slow the network is.

The input thread and sender thread communicate through an internal pipe, which acts as a buffer to collect data that is waiting to be up-loaded. The input thread must prevent this pipe to fill up as well, as that would make the input thread stall on writing to the pipe, so it can no longer read immediately from the input pipe. To help preventing such stalls, the input thread is highly selective in what it forwards to the sender thread. For one, it discards immediately those input lines not interesting for the broadcast. (A debug file contains many such lines.)

Showing PVs is considered an optional, real-time aspect of the broadcast. It thus makes no sense to create a large back-log of PVs when the network is slow, because it will only slow down showing the rest of the game. We are only interested in showing the PV info belonging to moves that are actually played, or from thinking about the next move (i.e. from the position the viewers can already see). We are also not interested in sending PVs so fast that the user has no time to see them, assuming the viewer will only show the latest PV of each engine.

Given this, it is pointless to forward PVs from input thread to sender thread when the sender lags behind in uploading moves. So both input thread and sender thread will keep a move counter, the latter counting the moves whose upload was confirmed. If the input thread receives a PV, it will check if the move counts are the same, and only if they are it would forward the PV through the internal pipe to the sender thread for uploading, and only those received some seconds after the engine started thinking. When the count is different (i.e. there is a move buffered in the internal pipe) PVs that are not discarded because of their low depth will be written into a buffer, (one for each engine) where they overwrite the previous PV. So this buffer will always contain the latest PV from the engine that has not (yet) been forwarded, and will be cleared as soon as we forward a PV from that engine. When an engine moves, any buffered PV would be forwarded to the sender thread, after which the move is forwarded. This guarantees every move will be accompanied with a PV. But in times of backlog there would be only a single PV just before each move. These PVs could be shortened to, say, 10 moves plus score/depth info, for inclusion in the PGN file that will be instructed at the server from the uploaded moves.

The sender thread would upload any PV it gets to the 'latest PV' location on the server, so viewers could display such info while the engine is thinking. Any move would be added to the growing game, with the depth/score and (shortened) PV as a comment to it. When such a move is uploaded, the 'latestPV' for the engine playing that move will be cleared (as it no longer pertains to the current game position).

Viewing

A viewer will request game info from the server, preferably showing in the request how much of the game it already has (i.e. a move number). If the current game length is larger than the viewer already has (which will for instance be the case if a newly arriving viewer has nothing yet), the new additions to the game will be sent by the server in response. If the game on the server is not longer, the server can stall the response to the time when the game does grow, or when a PV for that game is updated. So the viewer won't know in advance what it gets in reply to its polling, but will have to check whether it is a new PV, or one (or more) moves of the game.

Depending on this it will either update the fields for the PVs it is displaying, or add the move(s) to the game it holds, and display the new position on its board. It will then issue a new poll for more info, unless the last 'move' it received was actually a game-end indicator.

Chat

It would be best if chat messages were integrated in the same data stream as the game, so that the viewer would not have to poll separately for game data and chat messages. In that case reception of a chat message by the server could also be used to satisfy any data request from a viewer that is pending (i.e. waiting for a move to be played or a PV update). For the viewer this just means there are three possible outcomes of a poll request (move(s), PV or chat). For the server this is a bit tricky, as viewers will not be waiting for their poll requests to be served all the time. They could be served with a move, after which it will take them some time to get in their poll request for the next move. If a chat message arrives on the server during the time interval between the polls, it would not be sent to them. So chat messages must be collected and numbered similar to game moves, and the viewer's polls must not only mention the number of game moves they have already seen, as well as the number of the latest chat message they have seen. The server could then give them any later moves or messages, possibly after waiting for such to arrive.

Server

The server should log requests it cannot immediately serve. On an HTTP server the processes servicing such requests could write their process ID in a file dedicated to this purpose (one per game), and then put themselves to sleep. An event in this game (be it a PV, move or chat-message arrival) could then go through this file, and send a signal to each waiting process. This would interrupt their sleep; have them check again for the requested data (in the order move, chat, PVs), remove their process ID from the file, and satisfy the request.

[Henk]

I doubt if there is anything interesting to broadcast. Only a few people will watch.

[bnemias]

I think it makes sense to have a standard client/server approach to online games. Often all I want to do is fire up a local engine and crank on the current position I'm looking at in a browser. That can be a royal pain because of lack of standards. Sometimes I have to play through the whole game to arrive at the current position. Yuck.

I don't see any reason for chat to be part of the stream. It's a separate issue from the game itself.

[hgm]

Logically the chat is an independent data stream. But if a client has to poll independently for new chat messages and new moves, it would double the number of pending polls.

As for analyzing an on-line game; this should just be a matter of copy-pasting the games into a GUI+engine. So the client definitely should have a button for putting the game on the clipboard. But that is more an issue of the UI design of the client, rather than something the broadcast protocol must take account of.

I have been experimenting abit this morning with some generic server code (formaking a TCP/IP connection) I found on the web, and I was surprised how easy it is to make this into a rudimentary HTTP server. So I am inclined now to completely write the server from scratch, rather than embedding it in an 'official' HTTP server as CGI or PHP script. Then people would not need any third-party software to broadcast; they could simply run the server program on the same computer as the games are running, as you can do with TLCS. Nevertheless I can still use the normal HTTP GET method for requesting and depositing info.

So I am thinking now of a rudimentary HTTP file server that only handles GET requests, and intercepts requests of the form

/directory/play.cgi?game=N&...

where 'directory' would allow you to make a sub-division (e.g. per variant you broadcast, or per tournament). The game=N would specify which game you wanted to access if N>10, (where they would be stored in files ./directory/gameN.txt on the server), while N<10 would order different operations, like requesting an overview of the games (N=0) or create a new game. Depending on what you want to do, additional parameters would be needed. E.g.

game=3&pw=PASSWORD&w=NAME1&b=NAME2
game=N&pw=PASSWORD&ply=M&move=MOVETEXT
game=N&pw=PASSWORD&chat=X&msg=TEXT
game=N&ply=M&chat=X

The first would create a new file gameN.txt, with NAME1-NAME2 in it as first (and only) line. And then return a message like "Game N created", so the broadcaster would know which N to use in the 2nd and 3rd formats for adding a line "MOVETEXT" to that file, or add a line "X: TEXT" to the file chatN.txt, respectively. In all these cases the PASSWORD would have to match that given as argument to the server, when it was started.

The 4th format would be the probing request used by the clients, and return either all moves of the mentioned game beyond M, all chat messages for that game beyond X. Or, when neither is available, be remembered (while the requesting connection is kept open), and return a move, chat message or PV which gets in first.