Stockfish Natural TB loses heavily to Stockfish master

[Laskos]

Funny thing with "Natural". Marco gets excellent results for 5-men root positions with 5-men Syzygy. "Natural" beats the master on 5-men positions at root . In his 5-men suite he mixes up some 6-men positions (having only 5-men Syzygy), and the only failures I see in his test on 5-men at root are those of "Natural". This is the state of "Natural" as of now:

Code: Select all

[Event "My Tournament"]                                                               
[Site "?"]                                                                            
[Date "2017.09.14"]                                                                   
[Round "564"]                                                                         
[White "stockfish_ms"]                                                                
[Black "stockfish"]                                                                   
[Result "1-0"]                                                                        
[FEN "8/8/8/8/1K3p2/8/3rk3/Q7 w - - 0 1"]                                             
[PlyCount "121"]                                                                      
[SetUp "1"]                                                                           
[TimeControl "10+0.1"]                                                                
                                                                                      
1. Qe5+ {0.00/18 0.34s} Kf3 {0.00/70 0.39s} 2. Qf5 {+0.49/21 0.15s}                   
Ke3 {0.00/51 0.16s} 3. Qg5 {+0.49/1 0s} Ke4 {0.00/17 0.33s}                           
4. Kc3 {+132.79/16 0.11s} Re2 {-79.00/22 0.19s} 5. Qb5 {+132.79/26 0.23s}             
Re3+ {-79.00/22 0.27s} 6. Kd2 {+132.79/28 1.3s} Kf3 {-92.00/23 0.56s}                 
7. Qh5+ {+132.79/28 0.19s} Ke4 {-93.00/26 0.42s} 8. Qg4 {+132.79/30 0.20s}            
Ke5 {-98.00/26 0.055s} 9. Qg5+ {+132.79/29 0.053s} Ke4 {-99.00/32 0.23s}              
10. Qh5 {+132.79/30 0.20s} Rd3+ {-100.00/28 0.22s} 11. Ke1 {+132.79/27 0.26s}         
Re3+ {-102.00/22 0.052s} 12. Kf2 {+132.79/27 0.044s} Rg3 {-102.00/24 0.19s}           
13. Qh7+ {+132.79/34 0.22s} Ke5 {-93.00/19 0.27s} 14. Qc7+ {+132.79/38 0.24s}         
Kf5 {-91.00/21 0.29s} 15. Qc5+ {+132.79/39 0.22s} Kg4 {-90.00/30 0.69s}               
16. Qc6 {+132.79/40 0.24s} Kf5 {-91.00/32 0.30s} 17. Qd7+ {+132.79/39 0.23s}          
Kg5 {-92.00/36 0.25s} 18. Qe7+ {+132.79/40 0.27s} Kg4 {-91.00/35 0.25s}               
19. Qd7+ {+132.79/36 0.24s} Kg5 {-92.00/37 0.27s} 20. Qg7+ {+132.79/1 0.001s}         
Kf5 {-93.00/35 0.25s} 21. Qh6 {+132.79/1 0.001s} Ke5 {-94.00/36 0.39s}                
22. Qf8 {+132.79/1 0.001s} Re3 {-95.00/37 0.24s} 23. Kg2 {+132.79/32 0.29s}           
Ke4 {-101.00/31 0.25s} 24. Qf6 {+132.79/1 0.001s} Rg3+ {-107.00/22 0.43s}             
25. Kh2 {+132.79/1 0s} Kf3 {-108.00/31 0.26s} 26. Qf5 {+132.79/1 0.001s}              
Rg2+ {-110.00/25 1.6s} 27. Kh3 {+132.79/1 0s} Rd2 {-112.00/25 0.31s}                  
28. Qb1 {+132.79/22 0.20s} Rd4 {-113.00/22 0.20s} 29. Qb3+ {+132.79/18 0.18s}         
Kf2 {-114.00/24 0.34s} 30. Kg4 {+132.79/1 0.001s} f3+ {-117.00/23 0.20s}              
31. Kh3 {+132.79/1 0.003s} Ke2 {-112.00/24 0.31s} 32. Kg3 {+132.79/19 0.18s}          
Rd3 {-114.00/26 0.27s} 33. Qc2+ {+132.79/21 0.48s} Ke3 {-115.00/24 0.038s}            
34. Qf2+ {+132.79/21 0.094s} Ke4 {-116.00/1 0s} 35. Qc5 {+132.79/25 0.34s}            
f2+ {-117.00/24 0.37s} 36. Kxf2 {+132.79/28 0.41s} Rd7 {-93.00/25 0.20s}              
37. Qc4+ {+132.79/24 0.34s} Ke5 {-97.00/24 0.18s} 38. Ke3 {+132.79/22 0.63s}          
Kf6 {-99.00/22 0.17s} 39. Qc5 {+132.79/23 0.25s} Ke6 {-102.00/22 0.21s}               
40. Kf4 {+132.79/23 0.63s} Rd6 {-100.00/22 0.30s} 41. Qf5+ {+132.79/20 0.47s}         
Ke7 {-102.00/1 0s} 42. Ke5 {+132.79/20 0.044s} Rb6 {-103.00/17 0.75s}                 
43. Qh7+ {+132.79/20 0.16s} Kd8 {-104.00/19 0.13s} 44. Qf7 {+132.79/23 0.27s}         
Ra6 {-105.00/17 0.24s} 45. Qc4 {+132.79/22 0.22s} Rb6 {-105.00/18 0.24s}              
46. Qc5 {+132.79/23 0.24s} Ra6 {-106.00/19 0.14s} 47. Qc4 {+132.79/23 0.041s}         
Ra7 {-105.00/20 0.089s} 48. Qb5 {+132.79/1 0.002s} Kc7 {-108.00/20 0.044s}            
49. Kd5 {+132.79/21 0.12s} Rb7 {-109.00/23 0.14s} 50. Qc5+ {+132.79/1 0s}             
Kd8 {-110.00/24 0.18s} 51. Ke6 {+132.79/1 0.001s} Rb1 {-M24/26 0.18s}                 
52. Qd4+ {+M23/25 0.13s} Kc7 {-M22/28 0.088s} 53. Qe5+ {+M21/1 0.002s}                
Kb6 {-M20/30 0.11s} 54. Qb8+ {+M19/1 0.002s} Kc5 {-M18/27 0.017s}                     
55. Qxb1 {+M17/1 0.001s} Kc4 {-M16/29 0.090s} 56. Qd1 {+M11/29 0.19s}                 
Kc3 {-M10/31 0.094s} 57. Kd5 {+M9/39 0.19s} Kb4 {-M8/31 0.018s}                       
58. Qd3 {+M7/42 0.042s} Ka5 {-M6/89 0.22s} 59. Qb3 {+M5/100 0.19s}                    
Ka6 {-M26/1 0s} 60. Kc6 {+M3/127 0.013s} Ka5 {-M2/127 0.005s}                         
61. Qb5# {+M1/127 0.004s, White mates} 1-0

Returning to the more mundane things with Texel approach, I tested at 60 seconds per position 100 hard 6-men Wins with 5-men TBs.

Mates resolved:

Texel_No_TB:
4/100

Texel_Syzygy5_Gaviota5:
54/100



Stockfish_No_TB:
19/100

Stockfish_Early_Mate2:
15/100

I bet Stockfish, with Texel implementation of TBs, would have solved as mates maybe 70-90 out of 100 hard 6-men Wins at this testing time (60 seconds per position).

[kranium]

Hi Kai-

I've been reading this topic with great interest.
I want to thank you for all the work/testing and analysis you're doing on this.

I think Marco's idea of more 'natural' endgame play is excellent, and I applaud the effort he is making.
In the end, I'm sure he will be able to make some improvements in this regard.

My 2 cents (for what it's worth):

It's hard to see his efforts so harshly criticized by some.
I think he deserves our full support, and the benefit of the doubt (at least most of time!).

Keep up the good work!
Norm

[syzygy]

It's hard to see his efforts so harshly criticized by some.
I think he deserves our full support, and the benefit of the doubt (at least most of time!).

Ehm, no. All the criticism is well deserved. His efforts are completely misguided.

Just one example: he has been very vocally declaring that DTZ tables are totally useless. Why did he say that? Because he really believed it. How could he really believe it? Because he does not really have a good grasp of the issues, to put it very friendly.

He has now been trying courageously to get optimal DTZ play implemented correctly (yes, he realises now that DTZ tables are not totally useless). It is an immense struggle for him. After about a dozen tries it seems he still didn't get it right.

Now, the funny thing is... there is code that simply works. And if something needs to be improved about it, there are people that can quite effortlessly do that. But no, he wants to do it himself... dismissing any and all other attempts as "bad design". And breaking SF in the process (well, once committed).

[kranium]

It's hard to see his efforts so harshly criticized by some.
I think he deserves our full support, and the benefit of the doubt (at least most of time!).

Ehm, no. All the criticism is well deserved. His efforts are completely misguided.

Just one example: he has been very vocally declaring that DTZ tables are totally useless. Why did he say that? Because he really believed it. How could he really believe it? Because he does not really have a good grasp of the issues, to put it very friendly.

He has now been trying courageously to get optimal DTZ play implemented correctly (yes, he realises now that DTZ tables are not totally useless). It is an immense struggle for him. After about a dozen tries it seems he still didn't get it right.

Now, the funny thing is... there is code that simply works. And if something needs to be improved about it, there are people that can quite effortlessly do that. But no, he wants to do it himself... dismissing any and all other attempts as "bad design". And breaking SF in the process (well, once committed).

Well, many people 'tinker' with things as a way to learn how they work.
Take it apart, put it back together, break it, fix it, etc.

I don't see the harm of him tinkering with the TB code...his intentions are good, he'll figure it out.
I guess I don't understand why that's skin off your back.

Norm

[syzygy]

I don't see the harm of him tinkering with the TB code...his intentions are good, he'll figure it out.

If it were his private project, then sure. But "official-stockfish" is now more than a private project.

And then there is the part where one tries to helpfully explain that 1+1=2 and get back blabla about "illusions of the mind". Of course here the solution is to simply give up and stop wasting my time on a hopeless cause. So that's what I did.

[mcostalba]

Kay, do you want to get some fun? Get ready to have fun, I will post shortly the version with included no-silly sacrifices.

Ronald, working by attempt is my way to develop (and in case you didn't notice it is 8 years that SF development works in this way).

Ronald, rewriting code that already works even 100 times it is my way to improve (and in case you didn't notice all people but you and few totally biased trolls consider current syzygy implementation much better than original)

I am sorry for you but your code will fall, resistance is futile. But it will fall not because SF is my private project, but because new alternative it will be better.

[Ralph Stoesser]

Poor Kai.

[Michel]

and few totally biased trolls

I assume you refer to me here since you have accused me many times of bias.

I've asked you once before and I am doing it again. Please give me a single example that shows my bias.

But we should agree on definitions: bias is criticism not supported by evidence.

[Toadofsky]

Did Marco rip off DTZ from the probing?

Yes, he did. And he will probably commit it.

So serious users will have to look for alternatives.

In his words "The reason why DTZ are ripped out it is because they proved to be totally useless in real games". Anything else is an illusion (his words).

While I do appreciate Marco's publishing his attempts so others can provide feedback, I don't understand his vague subjective criticism of your attempts. I could understand a claim that costs/benefits of DTZ in "real" games are challenging to test... but "Natural TB" seems equally difficult to test.

I suppose the good news is that (assuming an Elo loss on some or all machines) Git makes it possible for developers to revert or cherry-pick commits.

[Uri Blass]

Kay, do you want to get some fun? Get ready to have fun, I will post shortly the version with included no-silly sacrifices.

Ronald, working by attempt is my way to develop (and in case you didn't notice it is 8 years that SF development works in this way).

Ronald, rewriting code that already works even 100 times it is my way to improve (and in case you didn't notice all people but you and few totally biased trolls consider current syzygy implementation much better than original)

I am sorry for you but your code will fall, resistance is futile. But it will fall not because SF is my private project, but because new alternative it will be better.

Calling "totally biased trolls" to people who disagree with you
seems to me a personal attack.

How many people expressed an unbiased opinion in this subject?

I admit that I did not read (and reading is not enough if you do not understand) the relevant changes so you can consider me as biased but I believe that the same can be also for people who support you.

I tend to believe Ronald for some reasons:
1)He is the developer of the syzygy implementation and I trust him to know better than you what works and what does not work and what is the results of changes in the code.
2)I saw bad results of NTB in test games in this thread.