hgm wrote:The main ugliness of DTZ play is that it starts with sacrificing all material you don't need, to zero the 50-move counter as fast as possible. E.g. if you have KQRK, it will sacrifice Q (because it is usually easier to force the bare King to take the Q than the R: you just drive it towards the edge with contact checks until he has no choice but to take it). Then it starts io optimally play KRK (because there are no ways left to zero the 50-move counter without drawing).
This could be prevented by only consider zeroing because of captures of the winning side, and not by captures of the losing side (which usually do not represent 'progress').
I think that you cannot prove that using this idea is not going to generate wrong tablebases.
It is possible that the only way of the winner to win is by a sacrifice.
For example suppose that you have mate in 80 when you play 40 moves sacrifice a pawn that the loser has to capture not to be mated in the next move and play again 39 moves(no captures except one capture and no pawn move)
If you decide that the capture of the losing side does not zero the 50 move counter of the 50 move rule then the position may be a draw.
I do not know about a case when something like that happens but I cannot prove that it does not happen.
Thinking about it again I think that the best solution is to start with generating tablebases only for DTM(distance to mate) when DTM<=50
if you find that DTM<=50 for all the positions with the relevant configuration(for example KQR vs K) then use DTM
If you find that there is at least one position when DTM>50 then use DTZ
This is going to solve at least most of the ugliness of DTZ and material configuration when there is at least one position when DTM>50 are usually not a win by some sacrifice.
I still can think of some ugliness when the computer may sacrifice a queen in KQRB vs KR because there may be position of KQRB vs KR that is mate in more than 50 moves(when the weaker side simply wins the queen in the next move).
Uri Blass wrote:I think that you cannot prove that using this idea is not going to generate wrong tablebases.
It is possible that the only way of the winner to win is by a sacrifice.
For example suppose that you have mate in 80 when you play 40 moves sacrifice a pawn that the loser has to capture not to be mated in the next move and play again 39 moves(no captures except one capture and no pawn move)
If you decide that the capture of the losing side does not zero the 50 move counter of the 50 move rule then the position may be a draw.
I do not know about a case when something like that happens but I cannot prove that it does not happen.
It can be proven after the fact. If you generate a tablebase in this metric (let's call it Distance To Progress, where losing your own pieces is not considered progress), and all won positions have DTP < 50, you know that there are no problems like the one you describe. If there are some positions with DTP >= 50 in the end-game you are generating the table for, you can also generate DTZ for it, and then check if there exist positions with a DTZ < DTP. If there aren't, there is nothing you can do. If there are, you simply use DTZ for that particular end-game. Giving priority to zeroing the 50-move counter in that case is apparently an essential part of the winning method, so it would not lead to any unnatural play.
syzygy wrote:
With some minor changes it should be possible to generate 7-piece tables on a system with 1 TB of shared RAM. If anyone wants to give that a try let me know.
What about using 1TB of SSD as swap? Know it would be a lot slower, but possible.
Thanks for the explanation. This is quite different from how I do it, in multiple ways. So it gives me a lot to think about. I probably should re-evaluate my ram/speed tradeoffs also, the last time I did serious tablebase work my system had 512meg of memory.
syzygy wrote:
With some minor changes it should be possible to generate 7-piece tables on a system with 1 TB of shared RAM. If anyone wants to give that a try let me know.
What about using 1TB of SSD as swap? Know it would be a lot slower, but possible.
SSDs are great for storing tablebases, but terrible for use as swap for the generation of tablebases. Writing to flash means erasing a whole block and rewriting it. The erase block size may be as large as 2MB (OCZ Vertex 3 SSD, says Google). During generation of a tablebase using my generator lots and lots of memory writes are made, many of which are in random order which means that caching won't help much and they have to be physically written to flash. There could easily be 500 writes per block during generation and another 500 during compression. The flash memory of recent SSDs has a maximum of about 3000 rewrites. Cheaper (larger capacity) TLC flash lasts from 500 to 1000 writes.
The firmware algorithms of SSD are probably pretty smart with how they continuously remap 4K "disk sectors" to flash erase blocks, so that it may not be as bad as it seems, but I would not count on it that a 1TB SSD survives the generation of more than a few 7-piece tables.
Angrim wrote:Thanks for the explanation. This is quite different from how I do it, in multiple ways. So it gives me a lot to think about. I probably should re-evaluate my ram/speed tradeoffs also, the last time I did serious tablebase work my system had 512meg of memory.
Are you thinking of generating the 6-piece suicide tables?
syzygy wrote:
Are you thinking of generating the 6-piece suicide tables?
Yes, although there is quite a ways between "thinking of" and actually doing it. I've been thinking about it since shortly before I finished the 5 piece tables, but back then the disk space needed would have cost a fortune. Now I finally have enough disk space that I could actually store the entire 6 piece table set compressed, but there's still a bit of coding to do before it becomes practical. And I haven't been very motivated the last few years.
I have now also removed the popcnt requirement. See the Makefile.
It should be possible to compile and run the generator on any x86-64 system with a 64-bit gcc and 16 GB of RAM (much less for tables with up to 5 pieces).
