Stockfish Save Hash version that loads EPD

[Dann Corbit]

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

[cdani]

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

No problema at all!!! Thanks for working on it

[giovanni]

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.

[Dann Corbit]

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.

Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.

[Dann Corbit]

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

It might be interesting to have two hash tables.
One is the normal hash table, seeded from a special pv hash table.
When the engine starts up, it could take the preloaded pv hash table and seed the main hash from it.
When the engine is done playing, it would scan the normal hash table looking for pv nodes. If it finds any new nodes or any improved nodes, it would store them into the pv hash.

This would be another kind of learning.



Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.

Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.

[Dann Corbit]

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

It might be interesting to have two hash tables.
One is the normal hash table, seeded from a special pv hash table.
When the engine starts up, it could take the preloaded pv hash table and seed the main hash from it.
When the engine is done playing, it would scan the normal hash table looking for pv nodes. If it finds any new nodes or any improved nodes, it would store them into the pv hash.

This would be another kind of learning.



Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.

Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.

[giovanni]

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.

Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.

Thanks again, Dann. Very exciting. I have just one technical question. Which one is the best way to get epd positions from these pgn files? I guess you need to preserve the evaluation and pheraps even the depth.

[Rodolfo Leoni]

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.

It looks great. The only inconvenience could be the fact stored evaluations come from engines which are very different each other. The improvement by your SF savehash engine will be absolutely relevant because that minimax needs to be "hammered".

I think it's the first time someone performs such an experiment. It'll be interesting to see how it evolves.

[duncan]

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

were you planning on doing this.?and at this stage do you have an estimate for elo gain ?

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.

what do you call a long period of time. ?

[Dann Corbit]

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

were you planning on doing this.?and at this stage do you have an estimate for elo gain ?

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.

what do you call a long period of time. ?

I might do the experiment. I have a box with 64 cores and 128 GB RAM, and I have a database with a billion analyzed positions.

I can always do some mining to increase the data.

I think I will want to add other information to the hash, so I may want to customize it.