Stockfish hangs

[hgm]

Well, searching also takes a noticeable amount of time. We don't stop the clock for that either. When I was still playing Chess, and arrived at my club for the match at 8 p.m., I could also not say to the referee "Sorry, but I have prepared this great opening repertoire, but I don't know it by heart yet. I want to study my notes on it first, so please start my clock only 15 min from now!".

[Ras]

When I was still playing Chess, and arrived at my club for the match at 8 p.m., I could also not say to the referee "Sorry, but I have prepared this great opening repertoire, but I don't know it by heart yet. I want to study my notes on it first, so please start my clock only 15 min from now!".

Come on, that doesn't make sense. It's more like starting the clock for a multi game blitz match when the pieces aren't even set up in the starting position. It makes even less sense to use "reuse=0" here because UCI engines are meant to play several games. The only engines I've ever seen who can't even re-initialise themselves properly and thus need to be restarted are CECP engines.

All it achieves is making Winboard useless for matches in particular at fast time controls.

[hgm]

Playing multiple games with the same instance has no advantage at all when engines need large amounts of time to reinitialize after every ucinewgame.

[Ras]

Playing multiple games with the same instance has no advantage at all when engines need large amounts of time to reinitialize after every ucinewgame.

I don't see why e.g. EGTBs would be reloaded for a new game - unless the GUI goes through the whole setoption yadda yadda process again, which would be a GUI misdesign.

[Dann Corbit]

Perhaps I should simply refrain from sending 'ucinewgame'? Since there was no earlier game, this seems to comply even with the newer specs.

I am not sure if they are compliant or not, but there are some engines that must have either the position setup or ucinewgame or they will not have the starting position initialized.

[hgm]

I don't see why e.g. EGTBs would be reloaded for a new game - unless the GUI goes through the whole setoption yadda yadda process again, which would be a GUI misdesign.

So why would they need significant time after ucinewgame, then? They can load the EGT before the readyok after the setoptions.

[Ras]

So why would they need significant time after ucinewgame, then?

I'm too lazy to look up SF's source. In my engine, I'm clearing history and hashtables. The latter can take a significant amount of time with large hashtables in relation to short time controls like one second per game without increment, which is the fastest my engine can do. That's why the subsequent "isready" will only be answered after that is done.

Though it's not yet even sure that the missing "isready" is actually the culprit. That would require a counter-test with "isready" and waiting for "readyok" to see whether the problem is gone.

It's also possible that the ponder toggle is a problem - maybe just leaving that out would make more sense to try first because it looks easier. Maybe toggling it that fast triggers some kind of race conditon in Stockfish.

[Joerg Oster]

So why would they need significant time after ucinewgame, then?

I'm too lazy to look up SF's source. In my engine, I'm clearing history and hashtables. The latter can take a significant amount of time with large hashtables in relation to short time controls like one second per game without increment, which is the fastest my engine can do. That's why the subsequent "isready" will only be answered after that is done.

Though it's not yet even sure that the missing "isready" is actually the culprit. That would require a counter-test with "isready" and waiting for "readyok" to see whether the problem is gone.

It's also possible that the ponder toggle is a problem - maybe just leaving that out would make more sense to try first because it looks easier. Maybe toggling it that fast triggers some kind of race conditon in Stockfish.

No, the missing isready is no problem, and also not the ponder toggle.

In Stockfish, after receiving a ucinewgame, not only the hash table gets cleared but also various thread-related stuff, mainly history tables.
Probably very similar to your engine.

Additionally, TBs get re-initialized to free up memory.

Code: Select all

  Tablebases::init(Options["SyzygyPath"]); // Free mapped files

I only vaguely remember reports by users having trouble because of low memory after many games.
You have to find and lookup the relevant issue/commit on GitHub.

[hgm]

In my engine, I'm clearing history and hashtables. The latter can take a significant amount of time with large hashtables in relation to short time controls like one second per game without increment, which is the fastest my engine can do. That's why the subsequent "isready" will only be answered after that is done.

Clearing history should not take measurable time. Why clear the hash table? Usually there is a button Clear Hash for that. Doing unsollicited just wastes time, which will hurt in ultra-fast games. Would you also do it when you knew it came out of your own thinking time? If not, then this shows how counter-productive it is to not penalize such time wastage.

Btw, if don't want to reuse any of the hash entries from the previous game, you could just XOR your incremental key at the start of the game with some new pseudo-random. That would force a miss on everything that was there. You could of course still get collisions, but you can always get those, and they would not be more frequent that otherwise. And it takes virtually zero time.

[Ras]

Clearing history should not take measurable time.

Agree.

Why clear the hash table?

Because that makes sure that the engine is in the same state upon each new game. I like things to be reproducible because it makes troubleshooting easier or even possible.

Would you also do it when you knew it came out of your own thinking time?

I'm using the UCI protocol as specified for that purpose. Re-initialising the engine doesn't count against one's time - and even less so if you re-start it anyway. I don't care about broken GUIs do as long as they don't manage to crash my engine.

Btw, if don't want to reuse any of the hash entries from the previous game, you could just XOR your incremental key at the start of the game with some new pseudo-random.

I could, but why would I add more complexity to think about and test (i.e. whether it will really work), and sacrifice reproducibility for no reason?