7 Man Syzygy and SSD

[Nordlandia]

No need to generate whole 8-man tables. Few 5-10 might be doable right, or is Bojun Guo's hardware limited by RAM in this case?

I remember EG magazine has mentioned before that few 8-man tablebases in other format than syzygy has been generated earlier.

[duncan]

We should be incredibly grateful for the efforts of Bojun Guo (aka noobpwnftw) and Ronald de Man for the stupendous effort of building the 7 man syzygy files and for making the code public and freely available for use.
It was very altruistic on the part of both parties.

I am sure every one will second that.

[syzygy]

About 4.7% of all games include 5-man TB positions.

But this is the wrong measure. In most games an engine will reach 5-men positions in the search and when they do, they can replace searching a whole subtree with a single probe.

[syzygy]

I remember EG magazine has mentioned before that few 8-man tablebases in other format than syzygy has been generated earlier.

I also seem to remember than a few have been generated. They must have been pawnless ones with a lot of duplicate pieces (like KQQQvKQQQ). After removing symmetries, these tables have about as many positions as some 7-men TBs. (My generator does not remove such symmetries, though.)

[Nordlandia]

I remember EG magazine has mentioned before that few 8-man tablebases in other format than syzygy has been generated earlier.

I also seem to remember than a few have been generated. They must have been pawnless ones with a lot of duplicate pieces (like KQQQvKQQQ). After removing symmetries, these tables have about as many positions as some 7-men TBs. (My generator does not remove such symmetries, though.)

Is it feasible to generate few 8-man tables in pawnless state or do it involve sophisticated modification of the generator.

[Zenmastur]

About 4.7% of all games include 5-man TB positions.

But this is the wrong measure. In most games an engine will reach 5-men positions in the search and when they do, they can replace searching a whole subtree with a single probe.

You seem to be saying that there is low/no correlation correlation between the number of TB hits that occur in the search and what shows up on the board. I'm not buying it. I'm assuming that what appears on the board has a VERY high correlation with what appeared in the search. But, if you have evidence to the contrary I'd be happy to consider it.

Your statement that whole sub-trees can be skipped by a TB hit is clearly true but, I don't have a program that will sum all TB accesses across each TB file for a 7M game database in a “reasonable” period of time. i.e 7,000,000 games * 150 plies per game * 60 second search per position = approximately 2,000 years. Even with a modest database with short games and short searches, 500,000 x 100 x 1 = ~579 days. The limit case would be to have no search at all, which is what I chose.

If you have a better method to obtain the required data please share! Until we have better data, I think the metric I chose is a reasonable one.

I remember EG magazine has mentioned before that few 8-man tablebases in other format than syzygy has been generated earlier.

I also seem to remember than a few have been generated. They must have been pawnless ones with a lot of duplicate pieces (like KQQQvKQQQ). After removing symmetries, these tables have about as many positions as some 7-men TBs. (My generator does not remove such symmetries, though.)

Is it feasible to generate few 8-man tables in pawnless state or do it involve sophisticated modification of the generator.

Even if it were feasible to generate 8-man pawnless EGTB why would you want to. I checked a few high quality game databases and found less than 1% of games end up in all piece endgames and >95% those are 6 man or less. So something less than 0.04% of games will see an all piece 8-man ending.

Pawnful endgames require less memory to generate because they can be broken up into pawn “slices”. (i.e. with a single pawn on the board the TB can be split into 8 separate parts with each part being calculated separately. With two pawns on the board the Tb can be split into 64 separate pieces etc. Note: this doesn't count any reduction due to symmetries.) This can aid in both generation and distribution of the files.



By far the majority of the conversions from one EGTB to another occur by a piece being removed from the game due to capture. A quick look at game databases shows something like 85%+ will convert by captures.

The promotions of a pawn leading to an all-piece endgame which hasn't been calculated is of course possible but it's not the end of the world. In some cases it's not even possible. E.g. Any positions with opposing pawns, one pawn of the opposite color on the same file and in front of the other pawn, results in a position in which neither pawn can promote until a piece or a pawn has been removed from the board thus reducing the position to a 7-man EGTB.

Even in the case were a promotion does happen it's not the end. Many of these positions the promoted piece can be captured in one or more ways. ALL of these captures result in a 7-man TB hit and most of these will resolve the position. Only the rare case of a promotion with no captures or no “good” captures possible will the search have to continue.

This seems to indicate that generating 8-man all-piece endgames can be omitted and the pawnful endgames will still be quite useful.

Regards,

Zenmastur

[Zenmastur]

I fixed a few errors and extended the list of RTBW files to fill out an SSD. This assumes that
162GB is used for all syzygy tb 6-man and less. Notice the different order of the first few files
and the removal of some files that were a result of some unusual games in the database I used.

The Sum GB column should be used to fill the drive as nearly as possible. RTBZ files for 7-man
can be stored on a conventional hard drive. The second % column shows the % of all games in
the database that contain a 7-man EGTB position will be found if all files in that row the and
the rows above it are included on the disk.

Code: Select all

		Pieces	# in DB	  %	  SUM	  %	SUM GB	GB	#/MB
1	4+3	kpppkpp	60,353	0.85%	60,353	9.58%	2.107	2.107	28.639
2	5+2	krpppkr	20,016	0.28%	80,369	12.76%	3.183	1.076	18.606
3	5+2	kppppkp	3,933	0.06%	84,302	13.39%	3.520	0.337	11.672
4	5+2	kqpppkp	1,934	0.03%	86,236	13.69%	3.709	0.189	10.231
5	4+3	krppkrp	165,068	2.33%	251,304	39.90%	26.971	23.261	7.096
6	4+3	kqppkpp	7,116	0.10%	258,420	41.03%	29.080	2.110	3.373
7	5+2	krpppkp	3,337	0.05%	261,757	41.56%	30.173	1.093	3.054
8	5+2	krpppkn	552	0.01%	262,309	41.65%	30.362	0.189	2.914
9	5+2	kbpppkn	1,770	0.03%	264,079	41.93%	31.026	0.664	2.666
10	5+2	krpppkb	676	0.01%	264,755	42.04%	31.283	0.257	2.629
11	4+3	krppkpp	12,457	0.18%	277,212	44.02%	36.338	5.055	2.464
12	5+2	knpppkn	2,322	0.03%	279,534	44.38%	37.298	0.960	2.419
13	4+3	kpppkqp	5,008	0.07%	284,542	45.18%	39.568	2.270	2.206
14	5+2	krrppkr	903	0.01%	285,445	45.32%	40.037	0.469	1.926
15	4+3	kpppkrp	8,882	0.13%	294,327	46.73%	44.682	4.645	1.912
16	4+3	kbppkpp	15,047	0.21%	309,374	49.12%	52.655	7.974	1.887
17	5+2	kbpppkp	2,746	0.04%	312,120	49.56%	54.147	1.492	1.840
18	5+2	krbppkr	3,125	0.04%	315,245	50.05%	56.078	1.930	1.619
19	5+2	kbpppkb	2,812	0.04%	318,057	50.50%	57.875	1.798	1.564
20	4+3	knppkpp	13,315	0.19%	331,372	52.61%	67.565	9.690	1.374
21	5+2	kppppkn	596	0.01%	331,968	52.71%	68.005	0.440	1.355
22	5+2	kqrppkp	744	0.01%	332,712	52.83%	68.561	0.556	1.338
23	5+2	knpppkb	1,883	0.03%	334,595	53.13%	70.008	1.447	1.301
24	4+3	kpppkbp	9,839	0.14%	344,434	54.69%	77.592	7.584	1.297
25	4+3	kpppknp	8,578	0.12%	353,012	56.05%	84.273	6.681	1.284
26	5+2	knpppkp	2,214	0.03%	355,226	56.40%	86.004	1.732	1.278
27	5+2	kqbppkp	745	0.01%	355,971	56.52%	86.610	0.606	1.229
28	5+2	kqpppkr	314	0.00%	356,285	56.57%	86.886	0.276	1.139
29	5+2	krnppkr	2,604	0.04%	358,889	56.98%	89.291	2.405	1.083
30	5+2	kppppkb	587	0.01%	359,476	57.08%	89.923	0.632	0.928
31	4+3	kbppkbp	26,184	0.37%	385,660	61.23%	120.670	30.746	0.852
32	4+3	krppkbp	9,633	0.14%	395,293	62.76%	132.105	11.435	0.842
33	5+2	kqrppkr	569	0.01%	395,862	62.85%	132.814	0.709	0.803
34	5+2	kqnppkp	495	0.01%	396,357	62.93%	133.448	0.634	0.781
35	4+3	knppknp	21,387	0.30%	417,744	66.33%	162.535	29.088	0.735
36	4+3	knppkbp	20,993	0.30%	438,737	69.66%	193.491	30.956	0.678
37	4+3	kbppknp	19,074	0.27%	457,811	72.69%	222.219	28.728	0.664
38	4+3	krppknp	6,275	0.09%	464,086	73.69%	231.852	9.632	0.651
39	5+2	kppppkr	1,090	0.02%	465,176	73.86%	233.614	1.763	0.618
40	4+3	kqppkqp	13,494	0.19%	478,670	76.00%	256.246	22.632	0.596
41	5+2	kqpppkq	2,006	0.03%	480,676	76.32%	259.992	3.746	0.536
42	4+3	kqppkrp	2,364	0.03%	483,040	76.70%	265.499	5.507	0.429
43	4+3	kqrpkrp	2,570	0.04%	485,610	77.10%	271.556	6.058	0.424
44	4+3	kbppkrp	13,357	0.19%	498,967	79.22%	303.163	31.607	0.423
45	4+3	kqrpkpp	558	0.01%	499,525	79.31%	304.485	1.322	0.422
46	5+2	krbppkb	342	0.00%	499,867	79.37%	305.304	0.818	0.418
47	4+3	krppkqr	3,095	0.04%	502,962	79.86%	312.994	7.690	0.402
48	4+3	krbppkn	215	0.00%	503,177	79.89%	313.575	0.581	0.370
49	5+2	kppppkq	426	0.01%	503,603	79.96%	314.774	1.199	0.355
50	5+2	kbnppkn	308	0.00%	503,911	80.01%	315.673	0.899	0.343
51	5+2	kbpppkr	2,243	0.03%	506,154	80.37%	322.570	6.897	0.325
52	5+2	kbnppkb	455	0.01%	506,609	80.44%	324.021	1.451	0.314
53	4+3	krnppkn	186	0.00%	506,795	80.47%	324.647	0.627	0.297
54	4+3	knppkrp	8,395	0.12%	515,190	81.80%	354.559	29.911	0.281
55	4+3	kqppkbp	776	0.01%	515,966	81.92%	357.625	3.066	0.253
56	4+3	krnppkb	175	0.00%	516,141	81.95%	358.321	0.696	0.251
57	4+3	krppkqp	4,188	0.06%	520,329	82.62%	375.011	16.690	0.251
58	4+3	kqbpkpp	699	0.01%	521,028	82.73%	377.801	2.790	0.251
59	4+3	krbpkrp	11,999	0.17%	533,027	84.63%	426.182	48.381	0.248
60	5+2	krbppkp	750	0.01%	533,777	84.75%	429.249	3.067	0.245
61	4+3	kpppkqr	254	0.00%	534,031	84.79%	430.320	1.071	0.237
62	4+3	kqppknp	766	0.01%	534,797	84.91%	433.636	3.316	0.231
63	5+2	knpppkr	1,461	0.02%	536,258	85.15%	440.948	7.312	0.200
64	4+3	krrppkp	215	0.00%	536,473	85.18%	442.051	1.103	0.195
65	4+3	kqnpkpp	629	0.01%	537,102	85.28%	445.395	3.344	0.188
66	4+3	krrpkrp	1,653	0.02%	538,755	85.54%	454.761	9.366	0.176
67	4+3	krnpkrp	10,056	0.14%	548,811	87.14%	512.786	58.025	0.173
68	4+3	krppkrb	4,784	0.07%	553,595	87.90%	541.473	28.687	0.167
69	5+2	krnppkp	559	0.01%	554,154	87.99%	544.981	3.508	0.159
70	4+3	krppkrn	4,111	0.06%	558,265	88.64%	572.816	27.835	0.148
71	4+3	krppkrr	1,279	0.02%	559,544	88.84%	584.584	11.768	0.109
72	4+3	knppkqp	1,158	0.02%	560,702	89.03%	596.022	11.438	0.101
73	4+3	kbppkqp	1,090	0.02%	561,792	89.20%	608.161	12.139	0.090
74	4+3	krbpkpp	951	0.01%	562,743	89.35%	619.313	11.152	0.085
75	5+2	kbnppkr	1,206	0.02%	563,949	89.54%	635.522	16.209	0.074
76	4+3	kpppkqn	213	0.00%	564,162	89.58%	638.727	3.205	0.066
77	4+3	kbnpkpp	1,074	0.02%	565,236	89.75%	655.047	16.320	0.066
78	4+3	kpppkqb	199	0.00%	565,435	89.78%	658.080	3.033	0.066
79	5+2	kbnppkp	475	0.01%	565,910	89.85%	665.343	7.263	0.065
80	4+3	krpppkq	694	0.01%	566,604	89.96%	676.454	11.111	0.062
81	4+3	kqbpkbp	571	0.01%	567,175	90.05%	685.603	9.149	0.062
82	4+3	krrpkrr	1,271	0.02%	568,446	90.26%	706.257	20.655	0.062
83	4+3	kqrpkbp	202	0.00%	568,648	90.29%	709.541	3.283	0.062
84	4+3	krnpkpp	773	0.01%	569,421	90.41%	722.504	12.964	0.060
85	4+3	kpppkrb	333	0.00%	569,754	90.46%	728.664	6.160	0.054
86	4+3	kqnpkbp	501	0.01%	570,255	90.54%	738.215	9.551	0.052
87	4+3	krrpkpp	172	0.00%	570,427	90.57%	741.794	3.579	0.048
88	4+3	kqppkrb	451	0.01%	570,878	90.64%	751.415	9.621	0.047
89	4+3	knnpknp	419	0.01%	571,297	90.71%	760.638	9.223	0.045
90	4+3	krrpkbp	284	0.00%	571,581	90.75%	767.369	6.730	0.042
91	4+3	kqbpkrp	499	0.01%	572,080	90.83%	779.640	12.271	0.041
92	4+3	krppkbn	1,144	0.02%	573,224	91.02%	808.636	28.996	0.039
93	4+3	kbbppkb	235	0.00%	573,459	91.05%	814.627	5.991	0.039
94	4+3	kpppkrn	285	0.00%	573,744	91.10%	821.915	7.288	0.039
95	4+3	krbpkrb	3,105	0.04%	576,849	91.59%	902.196	80.280	0.039
96	4+3	kpppkbn	381	0.01%	577,230	91.65%	912.505	10.309	0.037
97	4+3	krbpkbp	970	0.01%	578,200	91.81%	939.504	26.999	0.036
98	4+3	kqnpknp	365	0.01%	578,565	91.86%	949.713	10.210	0.036
99	4+3	kqppkrn	304	0.00%	578,869	91.91%	958.279	8.566	0.035
100	4+3	krrpkqp	1,226	0.02%	580,095	92.11%	993.723	35.444	0.035

Regards,

Zenmastur

[syzygy]

By far the majority of the conversions from one EGTB to another occur by a piece being removed from the game due to capture. A quick look at game databases shows something like 85%+ will convert by captures.

Just as chess is about reaching a winning pawn endgame, so are pawn endgames about getting a pawn to promote.

That your database doesn't show many promotions is not because the previous sentence is wrong but because games are typically resigned when promotion by the other side becomes inevitable.

What one could do to avoid generating pawnless TBs is assume any promotion is winning. The resulting TB would be very unreliable, but good luck with that.

[Zenmastur]

By far the majority of the conversions from one EGTB to another occur by a piece being removed from the game due to capture. A quick look at game databases shows something like 85%+ will convert by captures.

Just as chess is about reaching a winning pawn endgame, so are pawn endgames about getting a pawn to promote.

Nice strawman!

But I don't believe it's true. Direct mating attacks, capturing pieces to gain a decisive material advantage, perpetual checks, three-move repetitions, and stalemates are all valid motifs in the endgame.

With an odd number of men (3,5, or 7 men) then it's harder to get a close balance between the two sides. So, many of the odd man EGTB's will naturally lead to more decisive outcomes. That however doesn't diminish the validity of using the EGTB to obtain a draw. With an even number of men (4,6, or 8 men) the positions can be exactly materially balanced. One would expect a much higher percentage of drawn positions in balanced or nearly balanced EGTBs.

So, I don't think it's all about winning. And it certainly isn't all about winning by promoting pawns even if pawns are present on the board.

That your database doesn't show many promotions is not because the previous sentence is wrong but because games are typically resigned when promotion by the other side becomes inevitable.

Again I disagree. More on this later.

What one could do to avoid generating pawnless TBs is assume any promotion is winning. The resulting TB would be very unreliable, but good luck with that.

What makes you think I would have to “assume” anything. E.G.

[d]2b5/2rk4/3p4/3P4/2BRK3/8/8/8 w KQkq - 0 1

I don't see any assumptions about promotions that need to be made to resolve this endgame or the many other ones like it. In an 8-man EGTB there are positions like this that contain 2, 3, 4, 5, and 6 pawns. It's not possible for any of these to be converted to 8-man pawnless endgame as the pawns are blocked. I'll let you work out the number of permutations of such endgames. It's not a small number. Even if not all pawns are blocked there are a huge number of positions where neither side can make progress!

Anytime there are two or more pawns on the board the is no single move that can convert the endgame to a pawnless 8-man endgame. The more pawns there are the less and less likely it is to ever convert to an 8-man pawnless endgame. There are simply too many other ways to win or draw.

Like I said earlier, 85% (or more) of all 8-man endgames that I looked at covert to a 7-man (or less) endgame by one or more pieces being captured. So, for the vast majority of endgames NO assumptions need be made. They will naturally convert to a 7-man endgame with best play.

Here is an example:

[d]8/8/8/5pR1/2n1k3/6P1/3r1BK1/8 b - - 0 63

Who knows how many positions like this there are. One side can AND should force the conversion to a 7-man EGTB. Absolutely no assumptions need to be made about what happens if a pawn is promoted since the best and direct method to end the game doesn't involve a promotion. I do know there are an absolutely HUGE number of positions like this.

Another one of those positions where no promotion is possible while remaining an 8-man position:

[d]8/1r1k4/R1n5/6K1/1pB5/1P6/8/8 b - - 0 49

A position where a promotion occurs but there is still not a pawnless endgame:

[d]3r4/P7/8/2R5/1K3k2/1B4n1/6p1/8 w - - 0 62


A couple of modified positions where a conversion by promotion to a 8-man pawnless endgame could occur:

[d]3r4/P7/8/2R5/1K3k2/1B4n1/6n1/8 w - - 0 62

[d]3r4/P7/8/2R5/1K3k2/1B4n1/6b1/8 w - - 0 62

Notice that even if the promotion occurs and you now have an 8-man pawnless endgame it doesn't make much of a difference. If the pawnless endgame file doesn't exist and the EGTB routine exits back to the search on the very next ply a capture will be tried and this will convert it to a 7-man pawnless endgame which will immediately resolve the issue without further search.


All of these positions came from “real” human games with the exception of the last two which I derived from a single human game by replacing a pawn with either a night or a bishop so when the promotion occurred it would become a pawnless 8-man EGTB position. It took more time to write this post than it did to find these positions. So the idea that resignation by human players is the reason more pawn promotions aren't seen seem to be unsubstanciated.

As far as I'm concerned the notion that pawnful EGTB become worthless “just” because the pawnless EGTB aren't available is ridiculous. My guess would be that they retain >95% of there full value EVEN if no pawless EGTB are generated.

Of the 8-man EGTB that appear to be worth generating all have at least three pawns. This has a few implications. There is no way to directly convert any of them to a pawnless 8-man EGTB. They can all be broken in to at least 512 pawn slices. This yields smaller files generated and less memory required to do the generations. This indicates that the exact same hardware used to generate the 7-man EGTB would be more than sufficient to generate any of these 8-man EGTB's. Some of the pawn slices will be more important than others, which opens the possibility to forego generating some of the least important slices. So most of these endgames would have very low memory requirements if sufficient pawn slices were used!

Of the top 10 files 2 have 6 pawns each, 2 have 5 pawns each, 5 have 4 pawns each and 1 has 3 pawns.

Code: Select all

#	endgame		# occ.	Sum # occ.
1	krppkrpp	128,688	128,688
2	krpppkrp	55,780	184,468
3	krpppkrp	55,780	240,248
4	kpppkppp	48,227	288,475
5	kbppkbpp	23,218	311,693
6	kbpppkbp	10,955	322,648
7	kppppkpp	10,235	332,883
8	kpppkbpp	8,543	341,426
9	krppkrbp	8,266	349,692
10	kpppknpp	6,829	356,521
11	krppkrnp	6,429	362,950
12	krbppkrp	5,987	368,937
13	kpppkrpp	5,442	374,379
14	krnppkrp	4,899	379,278
15	krpppkpp	4,155	383,433
16	krpppkpp	4,155	387,588
17	kbpppkpp	4,022	391,610
18	knpppkpp	3,588	395,198
19	kpppkqpp	3,472	398,670
20	krppkqrp	2,146	400,816
21	krppppkr	2,056	402,872
22	kqpppkpp	2,025	404,897
23	krrppkrp	1,805	406,702
24	krppkrrp	1,681	408,383
25	krpppkrb	1,597	409,980
26	krpppkrn	1,374	411,354
27	kbppknbp	1,244	412,598
28	kbppkrbp	1,025	413,623

This is the play out of a game between a couple of 2200 class players. It was the very first game in the list of games that contained 8-man EGTB positions. And it seems "typical" of such games

[pgn][Event "?"] [Site "?"] [Date "????.??.??"] [Round "?"] [White "?"] [Black "?"] [Result "1/2-1/2"] [FEN "1B6/4r3/2K2Rp1/1P6/6k1/5p1n/8/8 w - - 21 67"] 67.Rxg6+ Kh4 68.Bg3+ Kh5 69.Rf6 Kg4 70.Bd6 Re2 71.b6 Rc2+ 72.Bc5 Ng5 73.b7 Ne4 74.Rf4+ Kxf4 75.b8=Q+ Kg4 76.Qg8+ Kf5 77.Qh7+ Ke5 78.Qh5+ Kf4 79.Qf7+ Ke5 80.Qc7+ Kf5 81.Qd7+ Kf4 1/2-1/2[/pgn]

Notice no 8-man pawnless endgames is reached, the game converts to a 7-man EGTB before the promotion, there is no resignation prior to the promotion and the point of the promotion is to force a draw by perpetual check or exchange pieces to force insufficient material.

It's early in the morning again so I hope I didn't screw any of the positions up as I modifed the post several times and changed several of the positions.

Regards,

Zenmastur

[syzygy]

By far the majority of the conversions from one EGTB to another occur by a piece being removed from the game due to capture. A quick look at game databases shows something like 85%+ will convert by captures.

Just as chess is about reaching a winning pawn endgame, so are pawn endgames about getting a pawn to promote.

Nice strawman!

But I don't believe it's true. Direct mating attacks, capturing pieces to gain a decisive material advantage, perpetual checks, three-move repetitions, and stalemates are all valid motifs in the endgame.

Fine, so you don't really know much about chess. Then we can end the conversation here.