Why PVs aren't displayed -- Robert Hourdart in TCEC chat

[TommyTC]

Editted From TCEC Chat: /videos/205467003:

https://www.twitch.tv/videos/205467003? ... video_chat


marklefler:Robert thanks. Are all Houdini shortened PV due to fail high/low unresolved searches? I ask because the number of one move PVs is very high. For example, just one full PV this game so far.

RobertHoudart:@marklefler just interrupted searches, the PV is simply not yet available at this iteration

marklefler:So mostly because of not even having finished searching the first move?

RobertHoudart:@marklefler correct - with all the threads running independently, there's no point waiting for thread 0 to finish doing something

RobertHoudart:in Houdini the "main" thread does nothing special, all the 43 threads are basically equivalent

marklefler:RobertHoudart: If the first move has not even been full searched, then there is no PV to report. So why not report the PV from the previous iteration?

RobertHoudart:@marklefler that PV is no longer available, it's been sent to the GUI in the previous iteration

fischerandom:@RobertHoudart Does it work like this: One thread starts the analysis, and after the first line have been generated with an eval, then work is distributed to a queue where other threads fetch work and then the first initial thread is doing the same work as any other thread? @marklefler Same question about Komodo : )

marklefler:fischerandom: no.

RobertHoudart:@fischerandom it's even simpler - all the threads start and run independently; they share their results via the hash table

arunsathya1309:@RobertHoudart so @Knight_Moves has to fix it or you can fix the PV issue, it really wud be great

Knight_Moves:Knight Moves isn't fixing anything

RobertHoudart:it's not just for H that the PV is often hidden - it's a generic problem that needs to be addressed in the cutechess-cli program used by TCEC

Knight_Moves:@arunsathya1309 I don't need to fix anything. cutechess reports what it gets

marklefler:or someone spending about 30 seconds could save the previous PV and print that out instead when they do not have a full one. Unfortunately, we do not have 30 seconds free.

RobertHoudart:again, many engines in this tournament have hidden their PVs - I can "solve" this in Houdini but that would not solve it for all the other engines

richard_tcec:Previous PV might not be valid anymore if a different move has been played

Knight_Moves:It's not the GUIs job to figure out what PV an engine wants

marklefler:fischerandom: Komodo always reports what it knows. there is a length limit on the PVs, but it is pretty long. SInce the final score comes from the end of the position in the PV (or a hash of that position), I think it is important to show. In Analysis mode (or go infinite), Komodo always eventually shows a full PV, since without it, what is the point of analysis.

[ilari]

Hopefully at some point we'll get the feature set that TCEC needs in mainline cutechess. Currently they're using this fork (which has some pretty impressive features) which is 534 commits behind mainline and it's almost pointless to try to debug it: https://github.com/jeremybernstein/cutechess

If the PV issue is in cutechess it's possible that it's fixed in the mainline version.

[elcabesa]

I understood that it's houdini that send an empty pv. I don't think that latest cutechess can solve the problem

[Houdini]

Hopefully at some point we'll get the feature set that TCEC needs in mainline cutechess. Currently they're using this fork (which has some pretty impressive features) which is 534 commits behind mainline and it's almost pointless to try to debug it: https://github.com/jeremybernstein/cutechess

If the PV issue is in cutechess it's possible that it's fixed in the mainline version.

It's not a fix of a bug, it's a new feature.
During match play an engine can stop its search iteration without producing a full PV; the final PV only shows 1 to 3 moves. This short PV ends up in the PGN file, and is shown in the TCEC interface. As everyone will have seen during the TCEC stages, this is the case with a lot of the engines - not just Houdini.

For the PGN it would be better to keep the last full PV in memory.
Schematically it could be implemented like this:

Code: Select all

if (pv.length() > 3 || pv.length() >= fullpv.length() || pv.moves(0) != fullpv.moves(0)) fullpv.assign(pv);

The "full PV" is updated when the new PV is long enough, or if there's a different new first move.

Cheers,
Robert

[elcabesa]

Another solution is that the engine will not print the pv of a not finished iteration.i think it's the solution adopted by most engines.
This way the last pv string sent to cutechess is the full pv of the last finished iteration.

[syzygy]

Another solution is that the engine will not print the pv of a not finished iteration.i think it's the solution adopted by most engines.

If I interrupt Stockfish's search, the last line typically looks like this:

Code: Select all

stop
info depth 28 seldepth 38 multipv 1 score cp 29 nodes 107750942 nps 1873180 hashfull 999 tbhits 0 time 57523 pv d2d4
bestmove d2d4 ponder d7d5

A PV with a single move.

The only reason SF usually outputs a full PV as last line is that it is willing to spend an awful lot of time (about 6 times its normal search time) to finish the last iteration. (Until someone submits a patch to improve this.)

[Milos]

The only reason SF usually outputs a full PV as last line is that it is willing to spend an awful lot of time (about 6 times its normal search time) to finish the last iteration. (Until someone submits a patch to improve this.)

If SF needs 6 times its normal search time to finish the last iteration it only means there was fail high/low in that last iteration and it makes perfect sense to spend more time researching it.
It is pretty simple and effective concept that you I'm afraid didn't yet grasp.

[syzygy]

The only reason SF usually outputs a full PV as last line is that it is willing to spend an awful lot of time (about 6 times its normal search time) to finish the last iteration. (Until someone submits a patch to improve this.)

If SF needs 6 times its normal search time to finish the last iteration it only means there was fail high/low in that last iteration and it makes perfect sense to spend more time researching it.
It is pretty simple and effective concept that you I'm afraid didn't yet grasp.

You seem somewhat preoccupied with me.

[Dan Cooper]

Hopefully at some point we'll get the feature set that TCEC needs in mainline cutechess. Currently they're using this fork (which has some pretty impressive features) which is 534 commits behind mainline and it's almost pointless to try to debug it: https://github.com/jeremybernstein/cutechess

If the PV issue is in cutechess it's possible that it's fixed in the mainline version.

It's not a fix of a bug, it's a new feature.
During match play an engine can stop its search iteration without producing a full PV; the final PV only shows 1 to 3 moves. This short PV ends up in the PGN file, and is shown in the TCEC interface. As everyone will have seen during the TCEC stages, this is the case with a lot of the engines - not just Houdini.

For the PGN it would be better to keep the last full PV in memory.
Schematically it could be implemented like this:

Code: Select all

if (pv.length() > 3 || pv.length() >= fullpv.length() || pv.moves(0) != fullpv.moves(0)) fullpv.assign(pv);

The "full PV" is updated when the new PV is long enough, or if there's a different new first move.

Cheers,
Robert

On the occasions that an engine only returns a 1 ply PV, cutechess could/should include the ponder move as ply 2, but not the previous iteration.

[petero2]

Editted From TCEC Chat: /videos/205467003:

https://www.twitch.tv/videos/205467003? ... video_chat


marklefler:Robert thanks. Are all Houdini shortened PV due to fail high/low unresolved searches? I ask because the number of one move PVs is very high. For example, just one full PV this game so far.

RobertHoudart:@marklefler just interrupted searches, the PV is simply not yet available at this iteration

That is not the only case where a short PV is sent to the GUI though. From the S10_Stage_1_debug.txt file:

Code: Select all

36228525   <Houdini 6.02&#40;11&#41;&#58; info multipv 1 depth 27 seldepth 46 score cp -23 time 67342 nodes 2346491893 nps 34844000 tbhits 0 hashfull 715 pv b7b6 a2a3 c7c6 c3c4 b8d7 f1e2 c8b7 e1g1 f8g7 c4d5 c6d5 b1d2 f6e4 a1c1 e8g8 d1c2 a8c8 c2a4 e4d2 f3d2 c8a8 a4c2 a8c8 c2a4 
36228525   <Houdini 6.02&#40;11&#41;&#58; info multipv 1 depth 27 seldepth 46 score cp -19 time 67343 nodes 2346492776 nps 34843000 tbhits 0 hashfull 715 pv b7b6 a2a3 
36228525   <Houdini 6.02&#40;11&#41;&#58; bestmove b7b6 ponder a2a3 

Houdini first produces a long PV for depth 27 but then immediately replaces it with a short PV before the bestmove is announced.