Stockfish 10 was released 29.11.2018

[Ovyron]

I was wondering the same thing, if regressions were a thing that was happening it'd be trivial to just add the code back and have an engine stronger than Stockfish dev, and then you just need to add the improvements of Stockfish dev to always be ahead of it.

[Vinvin]

Look at the regression test here : https://nextchessmove.com/dev-builds
20K games for each version.

[Zenmastur]

I did have something weird with Stockfish recently. I chose the latest build at the time for 4040 testing, and after 100 games it was -30 Elo to Stockfish 10. Yes I know that is a tiny sample, but the chances of it recovering that 30 Elo and gaining a further +30-40 Elo over the next couple of hundred games looked remote. So I went back to the Oct 9th build that gave me such good results at blitz, and it is performing well. After I've finished that I'll take the latest build again and see how it goes, to see if the previous result a was just an aberration, or wait for SF11.

I'm not surprised. I used to always use the latest but so many are regressions it's better to wait. Unfortunately there are so many simplification regressions that they cancel out any progress made in the past several months.

Regards,

Zenmastur

How do you know that there are many simplification regressions?
usually simplifications pass with more than 50%.

Can you release a significantly stronger stockfish if you delete the simplifications?
I would like to see a version of stockfish that show at least 5 elo improvement based on tests with 40000 games when to achieve this target you simply "delete" simplifications that of course mean adding code.

A good question, but I don't think there is a simple answer to it. I think it would be better to change the bounds on simplifications to make regressions harder to pass the simplification testing. I think this would be the easiest way to solve the problem.

[Modern Times]

Anyway, in due course I'll pick up the latest version and try again.

[Jouni]

Before release finally fix this

[d]8/8/6p1/6Pp/4R2P/2K4k/8/3r4 w - - 91 111

SF played just 111.Kb4?? in CCC and lost.

[Vinvin]

Before release finally fix this

[d]8/8/6p1/6Pp/4R2P/2K4k/8/3r4 w - - 91 111

SF played just 111.Kb4?? in CCC and lost.

Stockfish eval Kb4 as a draw but Black-Diamond-XR7 eval Kb4 around -2.
SF is buggy there.

[Nay Lin Tun]

Before release finally fix this

[d]8/8/6p1/6Pp/4R2P/2K4k/8/3r4 w - - 91 111

SF played just 111.Kb4?? in CCC and lost.

"After one year we have again +50 ELO gain! +46 single and +54 multicore in framework test. How much has Lc0 gained in last year ."

Well, I am pretty sure such kind of positions were Leela's Kryptonite in last year. Leela blundered and Stockfish immediately saw it. But now " SF blundered and Leela immediately saw it".

"One remarkable thing is that Leela's endgame become so much better that she is not too far way from Stockfish."

[MikeB]

Before release finally fix this

[d]8/8/6p1/6Pp/4R2P/2K4k/8/3r4 w - - 91 111

SF played just 111.Kb4?? in CCC and lost.

Stockfish eval Kb4 as a draw but Black-Diamond-XR7 eval Kb4 around -2.
SF is buggy there.

ON a clean start , SF get this correct and sees Kb4 is major blunder. It's the GHI raising its ugly head again.

GHI is as old as the hills with respect to chess, you don't hear much about it anymore, but that doesn't mean it has gone away.
https://www.chessprogramming.org/Graph_ ... nteraction

A more scholarly discussion: http://webdocs.cs.ualberta.ca/~mmueller/ps/aaai-ghi.pdf

30 second Elevator Explanation: Transposition tables (TT) do not contain game state, most importantly , the 50 move rule counter in this case. So if Stockfish or any other chess program has it stored as mate when the 50 move rule counter did not indicate the draw, then on a lookup down the road, where the 50 move rule counter has been incremented, the game state has changed, but when the lookup is performed, stockfish , or any other engine does not know that and still calls it mate, even though it now may be a draw ( or vica versa as below) . If you clear the TT on every game state change , you lose the benefits of TT so much that it loses Elo. All attempts to fix it involve ignoring TT after the move counter, say hits 90 ( 90 == 45 moves for each player) are either Elo losses or neutral and also new issues appear as well - specifically , those long mates that you like engines to find also disappear.

Code: Select all

dep	score	nodes	time	(not shown:  tbhits	knps	seldep)
 64	+149.04 	236.3M	0:07.66	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 63	+149.04 	115.7M	0:03.95	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 62	+149.04 	109.6M	0:03.74	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 61	+149.04 	101.5M	0:03.49	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 60	+149.04 	95.1M  	0:03.28	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 59	+149.04 	87.3M  	0:03.02	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 58	+149.04 	83.9M  	0:02.90	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 57	+149.04 	82.2M  	0:02.84	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 56	+149.04 	78.9M  	0:02.72	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 55	+149.04 	77.9M  	0:02.69	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 54	+149.04 	77.4M  	0:02.67	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 53	+149.04 	77.1M  	0:02.66	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 52	+149.04 	76.8M  	0:02.65	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 51	+149.04 	76.5M  	0:02.64	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 50	+149.04 	26.0M  	0:00.93	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kc1 h2 Rxh2 
 49	+149.04 	24.0M  	0:00.86	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kc1 h2 Rxh2 
 48	+149.04 	22.9M  	0:00.82	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Kd1 h2 Rxh2 
 47	+149.04 	21.2M  	0:00.76	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Kd2 h2 Rf1 h1=Q Rxh1 
 46	+149.04 	11.5M  	0:00.48	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Rf1 h1=Q Rxh1 
 45	+149.04 	11.0M  	0:00.46	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Rf1 h1=Q Rxh1 
 44	+149.04 	10.7M  	0:00.45	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Rf1 h1=Q Rxh1 
 43	+149.04 	9.79M  	0:00.42	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 42	+149.04 	9.21M  	0:00.40	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 41	+149.04 	8.75M  	0:00.38	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Kd1 Rg1+ Rxg1+ 
 40	+149.04 	8.16M  	0:00.36	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 39	+149.04 	7.81M  	0:00.34	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 38	+149.04 	7.34M  	0:00.33	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 37	+149.04 	6.96M  	0:00.31	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 36	+149.04 	6.75M  	0:00.30	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 35	+149.04 	6.52M  	0:00.29	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 34	+149.04 	6.22M  	0:00.28	Rc1 Ka3 Rh1 Kb4 Kg3 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb4 Kg3 Kc3 g4 Kb2 Kh3 Ka1 g3 Kb1 g2 
 33	+149.04 	3.03M  	0:00.15	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb5 Kg3 Rg1+ hxg1=Q Kb4 g4 Kc3 
 32	+149.04 	2.91M  	0:00.14	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb4 Kg3 Rg1+ hxg1=Q Kc3 g4 Kb2 
 31	+149.04 	2.78M  	0:00.14	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb5 
 30	+149.04 	2.68M  	0:00.13	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb4 
 29	+149.04 	2.58M  	0:00.13	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb3 
 28	+149.04 	2.45M  	0:00.12	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb3 
 27	+149.04 	2.33M  	0:00.12	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 
 26	+149.04 	2.23M  	0:00.11	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 
 25	+149.04 	1.99M  	0:00.10	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 
 24	+149.04 	1.90M  	0:00.09	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Re4 Kd5 Re5+ Kc4 Rxg5 Rf1+ 
 23	+149.04 	1.80M  	0:00.09	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 22	+149.04 	1.69M  	0:00.08	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 21	+149.04 	1.58M  	0:00.08	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 20	+149.04 	1.47M  	0:00.07	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 19	+149.04 	1.33M  	0:00.07	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 18	+149.04 	1.20M  	0:00.06	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 
 17	+149.04 	1.08M  	0:00.06	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 
 16	+149.04 	976135	0:00.05	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 
 15	+149.04 	879998	0:00.05	Rc1 Kb3 Kg3 Ra4 Rh1 Kc2 Rxh4 Ra3+ Kg4 Ra5 Rh2+ Kd3 h4 Rb5 
 14	+149.04 	759471	0:00.04	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 
 13	+149.04 	658636	0:00.04	Rc1 Kb3 Kg3 Ka3 Rh1 Rc4 Rxh4 Rc6 
 12	+149.04 	552712	0:00.03	Rc1 Ka3 Rg1 Ka4 Rg4 Kb3 Rxh4 Re3+ 
 11	+149.04 	439904	0:00.03	Rc1 Ka3 Rg1 Kb4 Rg4 Kc5 Rxh4 Re5 Ra4 Re6 
 10	+149.04 	206401	0:00.02	Rc1 Kb3 Kg3 Ka3 Rh1 Rc4 Rxh4 Rc2 Rh1 
  9	+147.92 	44303  	0:00.01	Rc1 Ka3 Rg1 Rb4 Rg4 Rb8 
  8	+2.51 	37533  	0:00.01	Rc1 Kb3 Kg3 Ka4 Rh1 Re3+ Kxh4 Re4+ Kxg5 
  7	+2.51 	34515  	0:00.01	Rc1 Kb3 Kg3 Ka4 Rh1 Re3+ Kxh4 Re4+ Kxg5 
  6	+2.04 	32858  	0:00.01	Rc1 Kb3 Kg3 Ka4 
  5	+1.04 	7497    	0:00.00	Kg3 Kc5 Rh1 Kd5 Rxh4 
  4	+2.31 	3508    	0:00.00	Rc1 Rd4 Rc2 
  3	+2.17 	3105    	0:00.00	Rc1 Re2 Kxh4 
  2	+2.17 	1472    	0:00.00	Rc1 Re2 
  1	+0.46 	784      	0:00.00	Kg3 
  1	Found 1511 tablebases 

Note : My score output is from side to move , black is winning here.

[Uri Blass]

Before release finally fix this

[d]8/8/6p1/6Pp/4R2P/2K4k/8/3r4 w - - 91 111

SF played just 111.Kb4?? in CCC and lost.

Stockfish eval Kb4 as a draw but Black-Diamond-XR7 eval Kb4 around -2.
SF is buggy there.

ON a clean start , SF get this correct and sees Kb4 is major blunder. It's the GHI raising its ugly head again.

GHI is as old as the hills with respect to chess, you don't hear much about it anymore, but that doesn't mean it has gone away.
https://www.chessprogramming.org/Graph_ ... nteraction

A more scholarly discussion: http://webdocs.cs.ualberta.ca/~mmueller/ps/aaai-ghi.pdf

30 second Elevator Explanation: Transposition tables (TT) do not contain game state, most importantly , the 50 move rule counter in this case. So if Stockfish or any other chess program has it stored as mate when the 50 move rule counter did not indicate the draw, then on a lookup down the road, where the 50 move rule counter has been incremented, the game state has changed, but when the lookup is performed, stockfish , or any other engine does not know that and still calls it mate, even though it now may be a draw ( or vica versa as below) . If you clear the TT on every game state change , you lose the benefits of TT so much that it loses Elo. All attempts to fix it involve ignoring TT after the move counter, say hits 90 ( 90 == 45 moves for each player) are either Elo losses or neutral and also new issues appear as well - specifically , those long mates that you like engines to find also disappear.

Code: Select all

dep	score	nodes	time	(not shown:  tbhits	knps	seldep)
 64	+149.04 	236.3M	0:07.66	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 63	+149.04 	115.7M	0:03.95	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 62	+149.04 	109.6M	0:03.74	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 61	+149.04 	101.5M	0:03.49	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 60	+149.04 	95.1M  	0:03.28	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 59	+149.04 	87.3M  	0:03.02	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 58	+149.04 	83.9M  	0:02.90	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 57	+149.04 	82.2M  	0:02.84	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 56	+149.04 	78.9M  	0:02.72	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 55	+149.04 	77.9M  	0:02.69	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 54	+149.04 	77.4M  	0:02.67	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 53	+149.04 	77.1M  	0:02.66	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 52	+149.04 	76.8M  	0:02.65	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 51	+149.04 	76.5M  	0:02.64	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kd3 h2 Rxh2 
 50	+149.04 	26.0M  	0:00.93	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kc1 h2 Rxh2 
 49	+149.04 	24.0M  	0:00.86	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Kc1 h2 Rxh2 
 48	+149.04 	22.9M  	0:00.82	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Kd1 h2 Rxh2 
 47	+149.04 	21.2M  	0:00.76	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Kd2 h2 Rf1 h1=Q Rxh1 
 46	+149.04 	11.5M  	0:00.48	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Rf1 h1=Q Rxh1 
 45	+149.04 	11.0M  	0:00.46	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Rf1 h1=Q Rxh1 
 44	+149.04 	10.7M  	0:00.45	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Rf1 h1=Q Rxh1 
 43	+149.04 	9.79M  	0:00.42	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 42	+149.04 	9.21M  	0:00.40	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 41	+149.04 	8.75M  	0:00.38	Rc1 Ka3 Rg1 Rf4 Rg4 Rf6 Kxh4 Kb2 Kxg5 Rf2 h4 Kc1 h3 Rh2 Rg2 Rh1 h2 Kd1 Rg1+ Rxg1+ 
 40	+149.04 	8.16M  	0:00.36	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 39	+149.04 	7.81M  	0:00.34	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 38	+149.04 	7.34M  	0:00.33	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 37	+149.04 	6.96M  	0:00.31	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 36	+149.04 	6.75M  	0:00.30	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 35	+149.04 	6.52M  	0:00.29	Rc1 Ka3 Rg1 Rf4 Rg4 Rf2 Kxh4 Kb2 Kxg5 Kc1 h4 Kc2 h3 Rh2 Rg2+ Rxg2+ 
 34	+149.04 	6.22M  	0:00.28	Rc1 Ka3 Rh1 Kb4 Kg3 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb4 Kg3 Kc3 g4 Kb2 Kh3 Ka1 g3 Kb1 g2 
 33	+149.04 	3.03M  	0:00.15	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb5 Kg3 Rg1+ hxg1=Q Kb4 g4 Kc3 
 32	+149.04 	2.91M  	0:00.14	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb4 Kg3 Rg1+ hxg1=Q Kc3 g4 Kb2 
 31	+149.04 	2.78M  	0:00.14	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb5 
 30	+149.04 	2.68M  	0:00.13	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb4 
 29	+149.04 	2.58M  	0:00.13	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb3 
 28	+149.04 	2.45M  	0:00.12	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 Kg4 Kc4 g5 Kb3 
 27	+149.04 	2.33M  	0:00.12	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 
 26	+149.04 	2.23M  	0:00.11	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 h2 Ra1 Kxg5 Rh1 
 25	+149.04 	1.99M  	0:00.10	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Rd8 Rg2 h3 Rg1 
 24	+149.04 	1.90M  	0:00.09	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 Re4 Kd5 Re5+ Kc4 Rxg5 Rf1+ 
 23	+149.04 	1.80M  	0:00.09	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 22	+149.04 	1.69M  	0:00.08	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 21	+149.04 	1.58M  	0:00.08	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 20	+149.04 	1.47M  	0:00.07	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 19	+149.04 	1.33M  	0:00.07	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 Rd4 Rg1+ Kf4 Kb6 h4 Kc5 
 18	+149.04 	1.20M  	0:00.06	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 
 17	+149.04 	1.08M  	0:00.06	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 
 16	+149.04 	976135	0:00.05	Rc1 Kb3 Kg3 Ka4 Rh1 Ka5 Rxh4 Re1 
 15	+149.04 	879998	0:00.05	Rc1 Kb3 Kg3 Ra4 Rh1 Kc2 Rxh4 Ra3+ Kg4 Ra5 Rh2+ Kd3 h4 Rb5 
 14	+149.04 	759471	0:00.04	Rc1 Kb3 Kg3 Kb4 Rh1 Ka5 Rxh4 Re1 
 13	+149.04 	658636	0:00.04	Rc1 Kb3 Kg3 Ka3 Rh1 Rc4 Rxh4 Rc6 
 12	+149.04 	552712	0:00.03	Rc1 Ka3 Rg1 Ka4 Rg4 Kb3 Rxh4 Re3+ 
 11	+149.04 	439904	0:00.03	Rc1 Ka3 Rg1 Kb4 Rg4 Kc5 Rxh4 Re5 Ra4 Re6 
 10	+149.04 	206401	0:00.02	Rc1 Kb3 Kg3 Ka3 Rh1 Rc4 Rxh4 Rc2 Rh1 
  9	+147.92 	44303  	0:00.01	Rc1 Ka3 Rg1 Rb4 Rg4 Rb8 
  8	+2.51 	37533  	0:00.01	Rc1 Kb3 Kg3 Ka4 Rh1 Re3+ Kxh4 Re4+ Kxg5 
  7	+2.51 	34515  	0:00.01	Rc1 Kb3 Kg3 Ka4 Rh1 Re3+ Kxh4 Re4+ Kxg5 
  6	+2.04 	32858  	0:00.01	Rc1 Kb3 Kg3 Ka4 
  5	+1.04 	7497    	0:00.00	Kg3 Kc5 Rh1 Kd5 Rxh4 
  4	+2.31 	3508    	0:00.00	Rc1 Rd4 Rc2 
  3	+2.17 	3105    	0:00.00	Rc1 Re2 Kxh4 
  2	+2.17 	1472    	0:00.00	Rc1 Re2 
  1	+0.46 	784      	0:00.00	Kg3 
  1	Found 1511 tablebases 

Note : My score output is from side to move , black is winning here.

I think that one of the problem with testing in the stockfish framework is that you adjudicate games as a draw when the evaluation is 0.00 for some moves and there is no progress and do not continue until you see a draw by the 50 move rule.

It means that even if some patch make an improvement then it will probably not pass the tests in the framework because the games are adjudicated too early as a draw.

[Zenmastur]

I think that one of the problem with testing in the stockfish framework is that you adjudicate games as a draw when the evaluation is 0.00 for some moves and there is no progress and do not continue until you see a draw by the 50 move rule.

It means that even if some patch make an improvement then it will probably not pass the tests in the framework because the games are adjudicated too early as a draw.

I'm not sure that's the problem.

The bounds used for simplification tests are [-3.00,1.00]. This allows to many regressions. It would be much more balanced if they changed the bounds to [-2.00,2.00]. Even [-2.50,1.50] would help.