Stockfish Save Hash version that loads EPD

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Stockfish Save Hash version that loads EPD

Post by Dann Corbit » Wed Aug 23, 2017 6:21 pm

You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

User avatar
cdani
Posts: 2104
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: Stockfish Save Hash version that loads EPD

Post by cdani » Wed Aug 23, 2017 7:46 pm

Dann Corbit wrote:You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.
No problema at all!!! :-) Thanks for working on it :-)

giovanni
Posts: 120
Joined: Wed Jul 08, 2015 10:30 am

Re: Stockfish Save Hash version that loads EPD

Post by giovanni » Wed Aug 23, 2017 7:50 pm

Dann Corbit wrote:You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.

Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Stockfish Save Hash version that loads EPD

Post by Dann Corbit » Wed Aug 23, 2017 7:55 pm

giovanni wrote:
Dann Corbit wrote:You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.
Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Stockfish Save Hash version that loads EPD

Post by Dann Corbit » Wed Aug 23, 2017 7:57 pm

Dann Corbit wrote:
giovanni wrote:
Dann Corbit wrote:You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.
It might be interesting to have two hash tables.
One is the normal hash table, seeded from a special pv hash table.
When the engine starts up, it could take the preloaded pv hash table and seed the main hash from it.
When the engine is done playing, it would scan the normal hash table looking for pv nodes. If it finds any new nodes or any improved nodes, it would store them into the pv hash.

This would be another kind of learning.



Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.
Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Stockfish Save Hash version that loads EPD

Post by Dann Corbit » Wed Aug 23, 2017 7:58 pm

Dann Corbit wrote:
giovanni wrote:
Dann Corbit wrote:You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.
It might be interesting to have two hash tables.
One is the normal hash table, seeded from a special pv hash table.
When the engine starts up, it could take the preloaded pv hash table and seed the main hash from it.
When the engine is done playing, it would scan the normal hash table looking for pv nodes. If it finds any new nodes or any improved nodes, it would store them into the pv hash.

This would be another kind of learning.



Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.
Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

giovanni
Posts: 120
Joined: Wed Jul 08, 2015 10:30 am

Re: Stockfish Save Hash version that loads EPD

Post by giovanni » Wed Aug 23, 2017 8:26 pm

Dann Corbit wrote:
giovanni wrote:
Dann Corbit wrote:You can get it here:
http://rybkaforum.net/cgi-bin/rybkaforu ... pid=574711

No attachments in this forum.

I had to make a lot of changes so that I could make a working version that I understood.

Something appeared to be wrong with the EPD parsing, so I made a simplified C like version. Hopefully, Daniel will not be insulted at my crude carpentry of his pretty code.

There is a binary in the src folder.

Hi, Dann. Thanks for the program. It seems that the savehash option of Stockfish is generatiing a flurry of new idea!
I am sure you already did, but could you briefly summarize how you would envision that this option could serve chessplayers and programs? Even better, it would be fantastic if you could provide a motivating example, like Rodolfo recently did for his correspondence games.

Consider things like CCRL and CEGT, TCEC, etc.
These contests create thousands and thousands of compute hours generated by strong computers at various time controls.

Now, it is possible to mine this data to get computer analysis for more than a billion EPD positions.

Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.

Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.
Thanks again, Dann. Very exciting. I have just one technical question. Which one is the best way to get epd positions from these pgn files? I guess you need to preserve the evaluation and pheraps even the depth.

Rodolfo Leoni
Posts: 544
Joined: Tue Jun 06, 2017 2:49 pm
Location: Italy

Re: Stockfish Save Hash version that loads EPD

Post by Rodolfo Leoni » Wed Aug 23, 2017 8:39 pm

Dann Corbit wrote: Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.
It looks great. The only inconvenience could be the fact stored evaluations come from engines which are very different each other. The improvement by your SF savehash engine will be absolutely relevant because that minimax needs to be "hammered". :)

I think it's the first time someone performs such an experiment. It'll be interesting to see how it evolves.
F.S.I. Chess Teacher

duncan
Posts: 10202
Joined: Mon Jul 07, 2008 8:50 pm

Re: Stockfish Save Hash version that loads EPD

Post by duncan » Wed Aug 23, 2017 8:41 pm

Dann Corbit wrote:
Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.
were you planning on doing this.?and at this stage do you have an estimate for elo gain ?
wrote: Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.
what do you call a long period of time. ?

Dann Corbit
Posts: 9994
Joined: Wed Mar 08, 2006 7:57 pm
Location: Redmond, WA USA
Contact:

Re: Stockfish Save Hash version that loads EPD

Post by Dann Corbit » Wed Aug 23, 2017 8:56 pm

duncan wrote:
Dann Corbit wrote:
Suppose that we make a huge hash file (e.g. 64 GB) and load it with this data. The file would contain an enormous number of pv nodes with perfect scores.
were you planning on doing this.?and at this stage do you have an estimate for elo gain ?
wrote: Now, take a really strong engine and have it play with (say) 64 cores at long time control for a long period of time. It is going to minimax this data naturally, vastly improving it.

Eventually, that file would contain nearly perfect information and the engines attached would play nearly perfect chess.
what do you call a long period of time. ?
I might do the experiment. I have a box with 64 cores and 128 GB RAM, and I have a database with a billion analyzed positions.

I can always do some mining to increase the data.

I think I will want to add other information to the hash, so I may want to customize it.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.

Post Reply