My "official" request to top engine programmers

[Guenther]

I solved all the little pv and multipv bugs and I made a new version:

www.andscacs.com/downloads/andscacs091150.zip

This zip file also contains a no popcnt "n" version.

I'm a profane in C++ world, but I'm a bit curious. If main hash is used as PH, the hashfull info should always be 100% after a while. Is really there a way to get them partially empty without deleting valuable entries?

The change proposed by Oliver Roese I think is to hide the "hash_full", nothing more.

Thanks a lot!

[Rodolfo Leoni]

The change proposed by Oliver Roese I think is to hide the "hash_full", nothing more.

Is it really so? As I said, I understand nithing of C++, but I can't relate these changes with hash_full info. 99,999% i'm wrong of course...

deleted line in search.cpp

TT.new_search();

replaced with

if (!Limits.infinite)
TT.new_search();
else
TT.infinite_search();

line added in TT.h

void infinite_search() { generation8 = 4; }

[hgm]

I'm a profane in C++ world, but I'm a bit curious. If main hash is used as PH, the hashfull info should always be 100% after a while. Is really there a way to get them partially empty without deleting valuable entries?

No. If every entry remains valuable, because you can return there in the course of analysis, and memory is finite, and new positions keep being produced, you must lose some of them. Either some older ones get replaced, or the new ones cannot be stored. This is why Chess engines use 'lossy' hash tables.

That everything is valuable doesn't mean that it is equally valuable, though. This makes hash replacement an art in itself: you must figure out what is least important, and replace that.

[Rodolfo Leoni]

I'm a profane in C++ world, but I'm a bit curious. If main hash is used as PH, the hashfull info should always be 100% after a while. Is really there a way to get them partially empty without deleting valuable entries?

No. If every entry remains valuable, because you can return there in the course of analysis, and memory is finite, and new positions keep being produced, you must lose some of them. Either some older ones get replaced, or the new ones cannot be stored. This is why Chess engines use 'lossy' hash tables.

That everything is valuable doesn't mean that it is equally valuable, though. This makes hash replacement an art in itself: you must figure out what is least important, and replace that.

And what an engine could consider more valuable? If I think at this question, the only answer I can find is Depth.

[hgm]

Apart from depth you can consider bound type: entries with an exact score represent more search effort than entries that merely give an upper or lower bound of the same depth. Also, entries that suggest a move can be more useful than entries that don't (e.g. because the null move, which would be tried first anyway, was sufficient to obtain the score bound, or none of the moves was any good, so that you expect having to try them all, and their order does not matter).

Also, recently obtained entries are more likely to be revisited soon than older entries. This because of the way a depth-first search walks through the tree. Transpositions that change a move closer to the root will only occur much later than transpositions involving only moves close to the leaves. It can be better to use a TT slot to keep many shallow entries for a short time (but long enough to be revisited), than to keep a single very deep entry in there for a very long time.

[Rodolfo Leoni]

Apart from depth you can consider bound type: entries with an exact score represent more search effort than entries that merely give an upper or lower bound of the same depth. Also, entries that suggest a move can be more useful than entries that don't (e.g. because the null move, which would be tried first anyway, was sufficient to obtain the score bound, or none of the moves was any good, so that you expect having to try them all, and their order does not matter).

Also, recently obtained entries are more likely to be revisited soon than older entries. This because of the way a depth-first search walks through the tree. Transpositions that change a move closer to the root will only occur much later than transpositions involving only moves close to the leaves. It can be better to use a TT slot to keep many shallow entries for a short time (but long enough to be revisited), than to keep a single very deep entry in there for a very long time.

Thanks! Very detailed description from you, as usual.

[BeyondCritics]

The change proposed by Oliver Roese I think is to hide the "hash_full", nothing more.

I have worked as professional C++ programmer and i think i know a little bit, how to analyze source code.
Your patch screws up the hash replacement logic of TranspositionTable::probe, see https://github.com/official-stockfish/S ... tt.cpp#L95. This method is now called in a way, which has never occured before. This is the real problem. Another problem of course is, that the method ranspositionTable::hashfull is now fooled to think, that the hash table is completely full. The info of hash utilisation is important for analysis too.
Don't underestimate the level of sophication of the stockfish code, there are literally cpu years of testing gone into every single line. If you change a line boldly, you can expect to mess it up somehow.

My patch treats infinite analysis, as if you were searching the starting position. It is still a change, but at least i don't mess with the hash replacement strategy.

[cdani]

The change proposed by Oliver Roese I think is to hide the "hash_full", nothing more.

I have worked as professional C++ programmer and i think i know a little bit, how to analyze source code.
Your patch screws up the hash replacement logic of TranspositionTable::probe, see https://github.com/official-stockfish/S ... tt.cpp#L95. This method is now called in a way, which has never occured before. This is the real problem. Another problem of course is, that the method ranspositionTable::hashfull is now fooled to think, that the hash table is completely full. The info of hash utilisation is important for analysis too.
Don't underestimate the level of sophication of the stockfish code, there are literally cpu years of testing gone into every single line. If you change a line boldly, you can expect to mess it up somehow.

My patch treats infinite analysis, as if you were searching the starting position. It is still a change, but at least i don't mess with the hash replacement strategy.

Thanks for explaining! Of course I know about the sophistication of Stockfish code, as I live with similar stuff every day in Andscacs.

The change is just an experiment to give analyzers a possibility to try HGM's suggestion. As they work with it if they wish, they will decide if is better or not than the standard way of doing of Stockfish, and consequently they will ask or not to have this in the official version.

Please, don't understand my phrase "The change proposed by Oliver Roese I think is to hide the "hash_full", nothing more" as if it has any aggressivity, nothing farter of my will.

[cdani]

So I updated the source and executable with the fix of Oliver. Thanks to him!

www.andscacs.com/downloads/stockfish_x6 ... alysis.zip
www.andscacs.com/downloads/stockfish-17 ... is_src.zip

[Rodolfo Leoni]

So I updated the source and executable with the fix of Oliver. Thanks to him!

www.andscacs.com/downloads/stockfish_x6 ... alysis.zip
www.andscacs.com/downloads/stockfish-17 ... is_src.zip

I don't exactly understand what had to be fixed. Both Stockfish forks behave the same in infinite analysys. Maybe Oliver was more precise but result doesnt change.

As for the has_full info, both versions start with 0%, and istantly show 99%.