Stockfish version with hash saving capability

[Rodolfo Leoni]

I compared results of Stockfish_x32_modern_170724_savehash: with Stockfish-TCEC6-32-PA_GTB:

the following position I suspect is a mate in 10. Stockfish_x32_modern_170724_savehash found a mate in 14 in 2 and half minutes on my computer depth 23. while Stockfish-TCEC6-32-PA_GTB takes over 9 minutes depth depth 25 to find a mate in 23.

so it seems for finding mates, latest stockfish is much faster. so it makes your work all the more valuable
thanks for doing this . much appreciated .

.......................................

Hi Duncan,

I'll run a test like this, to check if PHs will retain valuable entries: I'll set a complicated position up, then I'll try, in infinite move, to make moves, variations, to take them back, to try another variation, to take its moves back, watching if score gets propagated and if engine "remembers" scores of each variation (with an approx, as having the main hash persistent can't be as precise as a dedicated separate hash). Then, one could test saving and reloading that hash to check, again, if the engine kept all scores.

To test all above with different hash sizes will give us an idea of the min hash amount needed for long time analysis. If you analyze 1000 positions wiwh 4 GB hash each you'd need a 4 TBs HD.... So the lower the reliable size, the better the system.

I'll run a similar test on next sunday. It'd be nice if you do something similar so we can compare results.

Rodolfo

[cdani]

If someone wants to propose this change for official Stockfish, please go ahead. I don't plan to do it anytime soon. You can take this code or make your own version, as you wish.

Also I forgot to tell that in this code there is the small change proposed by Hgm about maintaining the age during infinite analysis session. As the other change, you can submit it to official Stockfish if you want if someone has not already done it.

[Rodolfo Leoni]

If someone wants to propose this change for official Stockfish, please go ahead. I don't plan to do it anytime soon. You can take this code or make your own version, as you wish.

Also I forgot to tell that in this code there is the small change proposed by Hgm about maintaining the age during infinite analysis session. As the other change, you can submit it to official Stockfish if you want if someone has not already done it.

Hi Daniel,

I agree with your solution. Your effort has been great, thanks.

I wish to specify this change would affect PHs only in infinite mode, so playing strenght wouldn't be affected any way. And it's only an UCI option, so users can decide to activate it or not.

(Am I right with it?)

[Dann Corbit]

I followed the steps you specified.
Unfortunately it doesn't work - neither in the Fritz 15 GUI nor in Aquarium 2017 - although the hash-file is created, but after loading the hashfile the analysis starts again at depth 0 - not 40 where I saved the analysis.
What could be wrong here? Any idea?

Nothing is wrong.
Look at how long it takes to analyze the same position compared to the first time.

The engine has the old hash values, but it still has to perform the search.

Thanks for explaining! This has given me an idea, to save also the whole state of the engine, so to allow it to recover the search from the point it was. I will try to see if is feasible in Andscacs.

It would be incredible if you could do that. It's almost like accomplishing a storable fork.

Imagine this:
You have a giant database of piece signatures. You load the storable fork for each signature when you analyze. It would only get smarter and smarter and if the hash was big enough (say, 16 GB per signature) it would hold a great amount of useful analysis.

[Rodolfo Leoni]

Imagine this:
You have a giant database of piece signatures. You load the storable fork for each signature when you analyze. It would only get smarter and smarter and if the hash was big enough (say, 16 GB per signature) it would hold a great amount of useful analysis.

Imagine this, Dann..... if that huge amount of useful analysis could be hosted, and anybody could contribute with some cpu time....

I need to go to work... I want to go! I need to turn it off!
Rodolfo! GO TO WORK!!!

[Dann Corbit]

Imagine this:
You have a giant database of piece signatures. You load the storable fork for each signature when you analyze. It would only get smarter and smarter and if the hash was big enough (say, 16 GB per signature) it would hold a great amount of useful analysis.

Imagine this, Dann..... if that huge amount of useful analysis could be hosted, and anybody could contribute with some cpu time....

I need to go to work... I want to go! I need to turn it off!
Rodolfo! GO TO WORK!!!

We don't have a storeable fork yet, so far as I know.

But as far as general analytics, I have lots of machines doing this around the clock.

It's a sickness.

[cdani]

I wish to specify this change would affect PHs only in infinite mode, so playing strenght wouldn't be affected any way. And it's only an UCI option, so users can decide to activate it or not.

(Am I right with it?)

Yes, only for analysis mode.

[cdani]

You have a giant database of piece signatures. You load the storable fork...

Sorry, I don't understand. What is a piece signature? and an storable fork?

I think in reality you mean simply that you will reuse the engine state many times to have better and better analysis, and that you will have various of those states stored, but I'm not sure exactly what you mean.

[retep1]

Can someone give an time-example - f.e. without hash it takes 30 sec. to reach depth 30 - with loaded hashfile it takes 5 sek. (depending on hardware? Of course).
My experience with 2 GB hash is that I see no significant difference. So I seem to be doing something wrong.

[cdani]

Can someone give an time-example - f.e. without hash it takes 30 sec. to reach depth 30 - with loaded hashfile it takes 5 sek. (depending on hardware? Of course).
My experience with 2 GB hash is that I see no significant difference. So I seem to be doing something wrong.

First the analysis session:

Code: Select all

setoption name Hash value 256
setoption name HashFile value f:\tmp\aa.shs
go infinite
info depth 1 seldepth 1 multipv 1 score cp 91 nodes 20 nps 10000 tbhits 0 time 2
 pv e2e4
...
info depth 25 seldepth 34 multipv 1 score cp 27 nodes 34905692 nps 1657046 hashf
ull 819 tbhits 0 time 21065 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8b4 c1g5 b8d7
 c4d5 e6d5 g5f6 d8f6 e2e3 e8g8 f1d3 c7c5 d1c2 b4c3 b2c3 h7h6 c3c4 c5d4 f3d4
info depth 26 currmove d2d4 currmovenumber 1
stop
info depth 26 seldepth 34 multipv 1 score cp 27 nodes 36525909 nps 1655978 hashf
ull 835 tbhits 0 time 22057 pv d2d4 g8f6 c2c4 e7e6 g1f3 d7d5 b1c3 f8b4 c1g5 b8d7
 c4d5 e6d5 g5f6 d8f6 e2e3 e8g8 f1d3 c7c5 d1c2 b4c3 b2c3 h7h6 c3c4 c5d4 f3d4
bestmove d2d4 ponder g8f6
setoption name SaveHashtoFile

So 22 seconds to depth 26 (single thread).

Now closing the engine and opening again:

Code: Select all

setoption name HashFile value f:\tmp\aa.shs
setoption name LoadHashfromFile
go infinite
...
info depth 25 seldepth 31 multipv 1 score cp 18 nodes 7249357 nps 1674603 hashfu
ll 886 tbhits 0 time 4329 pv g1f3 g8f6 d2d4 e7e6 c2c4 d7d5 b1c3 f8b4 c1g5 b8d7 e
2e3 e8g8 c4d5 e6d5 f1d3 h7h6 g5h4 g7g5 h4g3 f6e4 d1b3 b4c3 b2c3 d7b6 d3e4 d5e4 f
3d2
info depth 26 currmove g1f3 currmovenumber 1
info depth 26 seldepth 31 multipv 1 score cp 25 lowerbound nodes 8810309 nps 166
5150 hashfull 896 tbhits 0 time 5291 pv g1f3
info depth 26 currmove g1f3 currmovenumber 1
stop
info depth 26 seldepth 31 multipv 1 score cp 25 nodes 9785014 nps 1661574 hashfu
ll 898 tbhits 0 time 5889 pv g1f3

Now 5 seconds to depth 26.

Of course the win varies much if there is more or less hash, or how deep has gone, also the threads randomize a lot, so not always will pick much of the saved analysis. Hence having an option to save the whole state of the engine will be more useful. This does not negate some value for the current save hash option. Maybe ppl that use often engines for analysis can give a better opinion.