PGN standard, its improvement and standardization

[phhnguyen]

...
Xboard is better for gameplay and learning because you can have stateful game progress, whereas UCI is stateless
...

Your thought is interesting! However, it is not clear to me. How come it (Xboard) is better than UCI on gameplay/learning? Can you explain more?
Lc0 is a UCI engine and it can train itself a lot. Do you think Lc0 can get some more benefits if it uses XBoard protocol?

[Dann Corbit]

...
Xboard is better for gameplay and learning because you can have stateful game progress, whereas UCI is stateless
...

Your thought is interesting! However, it is not clear to me. How come it (Xboard) is better than UCI on gameplay/learning? Can you explain more?
Lc0 is a UCI engine and it can train itself a lot. Do you think Lc0 can get some more benefits if it uses XBoard protocol?

LC0 does not train during game play. It trains during training sessions.

An Xboard engine just gets the move to play. And so it maintains a state. A UCI engine gets the entire board position sent each time.
I think it is possible to make a UCI engine know the state, but it takes more work because it is not part of the design.

[Robert Pope]

Also, I believe that UCI doesn't send a game result/end game flag to the engine, so if you are doing any post-game processing, you have to infer when this occurs, rather than being told explicitly.

[phhnguyen]

LC0 does not train during game play. It trains during training sessions.

An Xboard engine just gets the move to play. And so it maintains a state. A UCI engine gets the entire board position sent each time.
I think it is possible to make a UCI engine know the state, but it takes more work because it is not part of the design.

That is only the problem of how to sync a UCI engine when receiving a list of moves. I think for anyone who needs to keep all internal states he may simply match that list with current move history, if all matched, all internal states can be kept. In normal cases, there is not different between UCI-XB engines in keeping states. However, only UCI engines have the chance to sync again with chess GUIs in broken/complicated cases.

IMO, implementation UCI protocol is a bit harder (than XB one) in the old day when multi-threads is not popular. But nowadays with multi-thread programming, it makes UCI become easier than WB. On the other hand, like a chess GUI developer, supporting XB takes me at least twice time/labor as UCI one.

[phhnguyen]

Also, I believe that UCI doesn't send a game result/end game flag to the engine, so if you are doing any post-game processing, you have to infer when this occurs, rather than being told explicitly.

Very good point!

It is a missing feature of the UCI protocol. However, we can simply add that as we have added a few (such as chess variants) already.

[Dann Corbit]

For very high speed games, sending only the move is a lot better than sending the whole game state including the board.

The best protocol would be a merger of the two, with redundant things thrown out.

[phhnguyen]

For very high speed games, sending only the move is a lot better than sending the whole game state including the board.

The best protocol would be a merger of the two, with redundant things thrown out.

I agreed partly. Also sending long lists of moves may make a problem of being exceeded some default input buffers.

However, that redundant is not that bad since it is still tiny compared with huge information a system of engines-chess GUI has to process. Modern engines (such as Stockfish, Lc0) are very "talkative" to print out a lot of info/stats, compared with old-style ones (such as Crafty).

Even processing huge data, it is not a big deal with modern computers of multi-cores/threads too. You can see that many programmers have been testing their UCI engines mostly in very fast games and they usually don't have problems working with long move strings. I have been working on my own chess GUI and observe that it can run concurrently several games, process/parse well all data, save them (in fly) into multi-files (for engines, PGN logs), display them on the screen, update boards, animations, clocks, query opening books, syzygy tablebase (for adjudication), etc...

[hgm]

Note that we now have completely drifted away from the original topic; engine communication protocols have nothing to do with game storage formats.

IMO statelessness w.r.t. the game state (including clocks) in UCI was a very bad idea. It is not only that it makes the communication unnecessarily verbose, but w.r.t. clocks there is a real problem: in classical TC the timing info accompanying the 'go' command does not specify how much time will be added after the 'movestogo' have been played. With movestogo=1 and wtime/btime=59000 you could be in a 40moves/hour game, at the brink of receiving another hour for the next 40 moves, in which case it would be wise to completely spend the remaining 59 sec on the upcoming move, as this is already below average. But you could also be in a 40moves/min game, where you got out of book after 39 moves, and receive only 1 new minute for the next 40. Wasting the 59 sec on a single move now effectively reduces your time for the second session by a factor 2, which would be very sub-optimal. The time management in this case should act like you have 1:59 for 41 moves (but be aware of a 'cold-turkey deadline' for the upcoming move). There is no way a UCI engine could know this.

Note that the so-called statelessness of UCI is in fact a red herring, unless you would artificially redefine position-moves plus go as a single command: after position-moves there is a defined game state on which the 'go' then relies. So the only 'merit' of UCI is that it leaves the game state undefined after 'go'. I would have preferred the engine's game state to be defined always, e.g. by having it unaffected by the 'go' command. (So that the GUI would have to feed the bestmove back to the engine if it indeed wanted to play it.)

Statelessness w.r.t. the side the engine is playing for might be a good idea. Note that in the case of learning an engine would always have to check whether it really played all moves for the same side. In CECP it is also possible to have the engine swap sides just before the final moves, to reverse its appreciation of the result.

Of course it is always a very bad idea (almost a crime against humanity) to make a new protocol.

[JohnWoe]

Stateless in UCI is indeed problematic. As you need to play a long list of moves over and over again.
And you get this position startpos/fen + moves at the same time. That's extra hassle to parse. In XBoard protocol after going a long list of ?/random/... you assume a move. I guess back in the old days people used to play games directly using XBoard protocol. But still let's say : a7a9 wouldn't too much to type.

In Sapeli I have lots of parsing code. With simple protocol I could cut it down to 20%. Ruby on the other is very powerful parsing all the nonsense. So that's not a problem w/ RubyShogi+Shuriken.

[hgm]

The CECP command that I find most cumbersome to implement in an engine is 'undo'. I usually do that by internally using the UCI method: the engine records the entire game, so that it can set up any earlier position from scratch by loading the initial FEN and replaying the moves from scratch, UCI-style. Engine life would be easier if the GUI would be responsible for remembering the game history and reloading the necessary part back into the engine, instead of sending 'undo' to make the engine do that.