UCI Win/Draw/Loss reporting

[Gian-Carlo Pascutto]

Someone pointed me a while ago (I forgot where, might've been TCEC chat) to this: https://github.com/DanielUranga/lc0/com ... 0df2f06db2
Which is allegedly how Fat Fritz does WDL reporting?

I have no idea if this is accurate, but in general I feel it would be useful to standardize some format if we want to report this information to interfaces. Based on the above patch, I suggest:

option name UCI_ShowWDL type check default false

If this is toggled to enabled, the engine will include wdl info in the info updates. Making it optional prevents interfaces from breaking, even though UCI does say the interface should ignore stuff it doesn't understand. But depending on the option seems safer as UCI is widely deployed and currmove etc use the same technique. GUIs which don't know about the option (and can't show wdl anyway) won't show it, if they're correctly implemented (UCI spec says UCI_ options shouldn't be displayed to the user).

For the wdl reporting itself I will use the format from the above patch, i.e.:

info depth 16 seldepth 62 time 11780 nodes 65546 tbhits 0 score cp 25 wdl 210 608 181 nps 5564 hashfull 201 multipv 1 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8b4 c1g5 h7h6 g5f6 d8f6 d1a4 b8c6 e2e3 e8g8 f1e2 c8d7 a4b3 d5c4 b3c4 f6e7 e1g1 b4d6 c3e4 e6e5 d4d5 c6d8

So

wdl win_probability_in_permille draw_probability_in_permille loss_probability_in_permille

In the above example, win probability = 21.0%, draw probability = 60.8% and loss probability = 18.1%. Note that due to rounding they might not sum to 100% exactly.

Engines that don't have an explicitly draw probability calculation can still obtain a good estimate from looking at the absolute score (i.e. say 65% at eval = 0 and 1% at eval = +300 with interpolation, to give a random example) so using this format shouldn't block them. (And really, draw probabilities are very useful to users I think...)

Unless there's a very compelling reason to not use this format, I'll go ahead and use this in Stoofvlees, and would encourage others to do the same.

[phhnguyen]

Someone pointed me a while ago (I forgot where, might've been TCEC chat) to this: https://github.com/DanielUranga/lc0/com ... 0df2f06db2
Which is allegedly how Fat Fritz does WDL reporting?

I have no idea if this is accurate, but in general I feel it would be useful to standardize some format if we want to report this information to interfaces. Based on the above patch, I suggest:

option name UCI_ShowWDL type check default false

I am curious why you use prefix UCI_ ? Does it mean standard?

Personally speaking, I prefer to no prefix for being shorter and looking "natural".

BTW, as a view of a chess GUI developer, the name of that option is not important for chess GUIs since they won't interfere: users must know its meaning and turn on/off themself but not chess GUIs (it differs from special ones: hash size, cores, syzygy path - chess GUIs may set up them).

If this is toggled to enabled, the engine will include wdl info in the info updates. Making it optional prevents interfaces from breaking, even though UCI does say the interface should ignore stuff it doesn't understand. But depending on the option seems safer as UCI is widely deployed and currmove etc use the same technique. GUIs which don't know about the option (and can't show wdl anyway) won't show it, if they're correctly implemented (UCI spec says UCI_ options shouldn't be displayed to the user).

For the wdl reporting itself I will use the format from the above patch, i.e.:

info depth 16 seldepth 62 time 11780 nodes 65546 tbhits 0 score cp 25 wdl 210 608 181 nps 5564 hashfull 201 multipv 1 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8b4 c1g5 h7h6 g5f6 d8f6 d1a4 b8c6 e2e3 e8g8 f1e2 c8d7 a4b3 d5c4 b3c4 f6e7 e1g1 b4d6 c3e4 e6e5 d4d5 c6d8

So

wdl win_probability_in_permille draw_probability_in_permille loss_probability_in_permille

In the above example, win probability = 21.0%, draw probability = 60.8% and loss probability = 18.1%. Note that due to rounding they might not sum to 100% exactly.

Engines that don't have an explicitly draw probability calculation can still obtain a good estimate from looking at the absolute score (i.e. say 65% at eval = 0 and 1% at eval = +300 with interpolation, to give a random example) so using this format shouldn't block them. (And really, draw probabilities are very useful to users I think...)

Unless there's a very compelling reason to not use this format, I'll go ahead and use this in Stoofvlees, and would encourage others to do the same.

I am going to release my own chess GUI in the next month. I have been trying to support new techniques thus I may support that (wdl) too. The problem for me is that the evaluation table usually congests. So far I have cut down some information such as tbhits, hashfull to keep it looks simple. Any suggestions to display?

[Gian-Carlo Pascutto]

I am curious why you use prefix UCI_ ? Does it mean standard?

From the UCI spec:

If the GUI get an unknown Option with the prefix "UCI_", it should just ignore it and not display it in the engine's options dialog.

If the GUI doesn't know about the option, it won't be able to display the information in a meaningful way anyway, so it makes no sense to display the option to the user. UCI_ShowCurrLine and UCI_ShowRefutations also work like that: if the GUI doesn't know about them, then allowing the user to enable them won't do any good as the GUI can't display the output. This prevents the user from seeing a useless toggle/option.

BTW, as a view of a chess GUI developer, the name of that option is not important for chess GUIs since they won't interfere: users must know its meaning and turn on/off themself but not chess GUIs (it differs from special ones: hash size, cores, syzygy path - chess GUIs may set up them).

It's exactly the opposite: the option is meant only for the GUI and should be set up by it.

[crem]

Instead of adding a UCI option for every individual feature, I would prefer to have some "extension" of UCI added and switched on with one option.

For example, it can be UCI_jsonExtension, and then uci info command could have something like
uci info ... json {wdl:[0.2,0.3,0.5],other_field:"spaceless\x20one-line\x20json"}

There are lots of other data that we wanted to expose from time to time (e.g. N of the best move in addition to the total N, annotate every ply of PV with probability to reach that point, number of requests to NN, even dump of a tree for some shallow depth, raw P and Q for debugging, etc).

All those data is not that important to introduce separate UCI option for them and hope that GUIs will support them, but if it's just one thing, it's something to be implemented once:

• GUI can show data that it knows "beautifully" (e.g. WDL)
• Unknown data it can hide, or show as raw values or raw json tree, but at least engines will know that GUIs know what it is.

[Gian-Carlo Pascutto]

Defining an extension won't help with the core problem: the goal of the protocol is to have engines and GUIs interoperate. So ideally, the things that are defined are useful across multiple engines, and the GUIs have some good way of representing them to the user. If something fits neither, it makes no sense to define it in the protocol - if the GUI knows you intimately you can send whatever you want and it doesn't need to be standardized, and if the GUI doesn't understand it, there is no point in sending anything.

So, the GUI has to support presenting the data. Changing the data encoding from something similar to what UCI already uses to JSON doesn't solve that. Unless you believe presenting a JSON tree to the user is useful. I don't think so. It's probably about as useful as showing the raw info string output (where you could already encode wdl data without any extension).

About the only thing that happens here is changing the encoding from something that is consistent with the rest of UCI to JSON. Parsing JSON is one of those things that looks easy, but is really hard to do correctly. That means that everything that needs to communicate with engines now needs to depend on a JSON library. (At least the engines themselves can likely avoid the dependency...)

The only advantage you're left with is that you only need 1 toggle to indicate the GUI can accept freeformat JSON data, and you can pile additional info in there. But now you're condemned to sending the pile of info at each possible opportunity, as there's no way for the interface to toggle parts on and off. So maybe that's not an advantage either...

I'm not sure this proposal at the core really fixes the "send arbitrary data" problem sufficiently. At least not to introduce a dependency on JSON?

The "give me the variation tree" example is good to understand this: you definitely don't want to send/receive that on every stats update. So the GUI still needs a way to tell the engine "I want that info now".

Maybe we'd need to say that in response to the engine reporting

option name UCI_JSONExtension type check default false

the GUI can reply:

setoption name UCI_JSONExtension value wdl other_field

So it sends multiple tags instead of a bool - that would be backwards compatible as an older GUI only sees the bool, and the engine will only receive that setoption if the GUI knows about it.

I'm not sure how GUI authors will feel about the overhead here. All the other data points you mention - for it to be of any use, someone has to go ahead and write a spec about how it's formatted.

[Gian-Carlo Pascutto]

Let me put it differently: with some more work (i.e. thinking about enable/disable info issue) it might be possible to specify a way to send more info inside UCI that could contain WDL and also serve as a well-designed vehicle for transmitting more information (which still has to be specified, because JSON by itself doesn't solve that).

But introducing JSON seems like a large barrier for just getting WDL info. So I'm not sure this is a good carrot to get GUI people to implement something.

[hgm]

I don't think it is a good idea to violate UCI protocol by sending non-booleans as value for a check option. I also don't think it is needed. If a GUI needs to let an engine know it supports something non-standard, it can just set an engine-defined option to tell it that it does.

I would argue for a string option UCI_infos, with the empty string as default value. A GUI could set it to a a space-separated list of keywords that it will recognized in an 'info' command besides the standard keywords 'score', 'pv' etc. This tells the engine what extra infos it can put in an 'info' command that the GUI knows how to display. If the GUI says it understands 'wdl', the engine can put wdl info with its PV, otherwise it will refrain from doing so. (Or it will find some backward-compatible way to display the info, like 'info string'). It is up to the community to negociate what infos they want GUIs to display, and what keyword to define for the ability to do so.

For info that should be sent on request a 'button' option would be the natural choice. UCI doesn't allow you to send 'setoption' commands to a searching engine, though. But if we want to make requests during search, we have no choice other than to violate that, and could define that button options with a name starting with 'UCI_' can be sent during search.

[phhnguyen]

Instead of adding a UCI option for every individual feature, I would prefer to have some "extension" of UCI added and switched on with one option.

For example, it can be UCI_jsonExtension, and then uci info command could have something like
uci info ... json {wdl:[0.2,0.3,0.5],other_field:"spaceless\x20one-line\x20json"}

There are lots of other data that we wanted to expose from time to time (e.g. N of the best move in addition to the total N, annotate every ply of PV with probability to reach that point, number of requests to NN, even dump of a tree for some shallow depth, raw P and Q for debugging, etc).

All those data is not that important to introduce separate UCI option for them and hope that GUIs will support them, but if it's just one thing, it's something to be implemented once:

• GUI can show data that it knows "beautifully" (e.g. WDL)
• Unknown data it can hide, or show as raw values or raw json tree, but at least engines will know that GUIs know what it is.

Good idea!
Just suggest to use at bit shorter form since "info" may be included inside too. Note that JSON format requires some quotes.

uci json {"wdl":[0, 1, 2]}

If you or someone installs it for Lc0, I will definitely support. At the moment my chess GUI has to parse Lc0's strings for stats which is not the right way to get info, IMO.

[phhnguyen]

Defining an extension won't help with the core problem: the goal of the protocol is to have engines and GUIs interoperate. So ideally, the things that are defined are useful across multiple engines, and the GUIs have some good way of representing them to the user. If something fits neither, it makes no sense to define it in the protocol - if the GUI knows you intimately you can send whatever you want and it doesn't need to be standardized, and if the GUI doesn't understand it, there is no point in sending anything.

So, the GUI has to support presenting the data. Changing the data encoding from something similar to what UCI already uses to JSON doesn't solve that. Unless you believe presenting a JSON tree to the user is useful. I don't think so. It's probably about as useful as showing the raw info string output (where you could already encode wdl data without any extension).

About the only thing that happens here is changing the encoding from something that is consistent with the rest of UCI to JSON. Parsing JSON is one of those things that looks easy, but is really hard to do correctly. That means that everything that needs to communicate with engines now needs to depend on a JSON library. (At least the engines themselves can likely avoid the dependency...)

We are talking about another channel for morden engines to talk to chess GUIs. Thus we will limit one-way JSON from engines to chess GUIs only.

Lucky, create JSON is so easy, engines don't need any extra library, just normal print command is more than enough. For chess GUIs, parsing JSON is not a big problem. I believe almost all chess GUIs can do.

The only advantage you're left with is that you only need 1 toggle to indicate the GUI can accept freeformat JSON data, and you can pile additional info in there. But now you're condemned to sending the pile of info at each possible opportunity, as there's no way for the interface to toggle parts on and off. So maybe that's not an advantage either...

I'm not sure this proposal at the core really fixes the "send arbitrary data" problem sufficiently. At least not to introduce a dependency on JSON?

The "give me the variation tree" example is good to understand this: you definitely don't want to send/receive that on every stats update. So the GUI still needs a way to tell the engine "I want that info now".

Maybe we'd need to say that in response to the engine reporting

option name UCI_JSONExtension type check default false

So far so good. I agreed!

the GUI can reply:

setoption name UCI_JSONExtension value wdl other_field
So it sends multiple tags instead of a bool - that would be backwards compatible as an older GUI only sees the bool, and the engine will only receive that setoption if the GUI knows about it.

Hm, perhaps not! IMO, more options (especially when the using way is not standard) are much more complicated unnecessarily.

I suggest a simple way: engines just send (JSON info) what they want. A chess GUI will parse and display in specific ways what it can understand, other info (which it does not know) will display as tables, trees or lists of strings. IMO, JSON data has structure thus chess GUIs have more info to display, compared with simple strings.

I'm not sure how GUI authors will feel about the overhead here. All the other data points you mention - for it to be of any use, someone has to go ahead and write a spec about how it's formatted.

[Gian-Carlo Pascutto]

I would argue for a string option UCI_infos, with the empty string as default value.

Ah right. Good idea! I was thinking of a way to set multiple values and that'd work perfectly, and yes, it's better than abusing bool.

So:

option name UCI_JSONInfo type string <empty>

GUI:

setoption name UCI_JSONInfo value wdl

Engine reports back:

info json {"wdl":[0.2, 0.5, 0.3]}