7-men Syzygy attempt

[Nordlandia]

6-men adjudication works perfectly fine on HDD. HDD wake up from idle once approaching adjudication limit. Probing 5-men + 6-men adj is good solution for weaker machines.

[syzygy]

Code: Select all

vm.swappiness=0
kernel.nmi_watchdog=0
[b]kernel.numa_balancing=0[/b]
kernel.sched_migration_cost_ns=5000000
kernel.sched_autogroup_enabled=0

Running without thread binding(-a) and having the above tweaks drops CPU sys% usage on top from 60% to 2%, thread binding has some negative impact on performance due to far side NUMA.

Looks like the workaround is to not letting Linux migrate memory pages across NUMA nodes since we use a lot of huge pages and accessed randomly in general, but to migrate threads among cores gently to reduce overall far side NUMA rate.

Inter-node memory accesses cannot be avoided entirely, but I think there are some relatively easy ways to make the generator more NUMA friendly.

[noobpwnftw]

Perf shows a lot of extremely slow software interrupts, has something to do with spin locks in memory page management.

I think there is no need to fix on the generator side, it must be a kernel problem, or it wouldn't not work so smoothly on virtual machines.

I tried many different things and it will basically hang on everything under heavy load.

Not feeling like to figure out why, do you want to play around with it?

[syzygy]

2) Many positions are easy to solve without a table. For example, kbk and knk 3-men combinations are always drawn and you do not need a table to resolve them. On the other hand, krk and kqk are mostly won, apart for a few trivial cases that a 1 ply search can decipher. Only the kpk combination with 111,282 wins (from the pawn side) and 54,394 draws are complex enough to make a table worth.

Let's see how much space is "wasted" by those useless KNvK, KBvK, KRvK, KQvK WDL files:

Code: Select all

table    size
KNvK     80 bytes
KBvK     80 bytes
KRvK    208 bytes
KQvK    272 bytes

(KPvK takes up 7824 bytes.)

[abulmo2]

2) Many positions are easy to solve without a table. For example, kbk and knk 3-men combinations are always drawn and you do not need a table to resolve them. On the other hand, krk and kqk are mostly won, apart for a few trivial cases that a 1 ply search can decipher. Only the kpk combination with 111,282 wins (from the pawn side) and 54,394 draws are complex enough to make a table worth.

Let's see how much space is "wasted" by those useless KNvK, KBvK, KRvK, KQvK WDL files:

Code: Select all

table    size
KNvK     80 bytes
KBvK     80 bytes
KRvK    208 bytes
KQvK    272 bytes

(KPvK takes up 7824 bytes.)

You forgot the dtz50 table:

Code: Select all

KNvK   80 bytes
KBvK   80 bytes
KRvK 7632 bytes
KQvK 5392 bytes
KPvK 4176 bytes

So 53% of the necessary space is "wasted". Moreover, you cut my quote where the important sentence was the following one:

Of course the same can be said for many 4-men to 8-men tables

For example, are the 5-men positions with an isolated king usefull? Are KRRvKN and KRRvKB table really usefull? For the latter cases, I know some drawing positions could occur and even some mate by the knight side, but a shallow 1 or 2 ply search should trivially eliminate them.
I understand that as a table maker, you want to be exhaustive; but I guess you could agree that an incomplete 7-men table restricted to hard to decipher positions are somewhat more usefull than a bunch of trivial 3,4,5 and 6-men positions.

[noobpwnftw]

Further perf digging shows kernel scheduling issues on bare hardware(tried 3.14, 4.15, referenced various guides, still no luck), falling back to ESXi + VM where everything works but running on less cores, I heard that it's the one and only system of this model(SYS-7089P-TR4T) ever built from the hardware vendor so far, so no surprise when something doesn't work.


About usefulness of certain TB piece combinations, that really depends on the use case: for engines a partial set is usually sufficient, for defining a TB format and making its generator a full set is more desirable. With the current latest generator code, you can build most 7-piece TBs(except some that overflows 8-bit DTZ value) with the -d option in about 900GB memory(pawnless) and about 760GB(pawnful).

Further work may reduce memory requirements for some combinations with the same piece while doubling memory usage for building ones that require 16-bit of DTZ values.

After statistics are available for the full set you can decide which are useful for engines and which are just for the sake of making a full set for future 8-piece attempts.

[Nordlandia]

Bojun Guo: i guess a partial set is satisfactory for many as lomonsov-7 already exist. Albeit can't be probed with engines, just locally at the russian university with custom stockfish according rumours.

[noobpwnftw]

Jon: my goal is to make 7-piece tablebases available for all forms of usage. Syzygy is by far the most compact format. Hopefully this can be done within this year.

[Nordlandia]

Jon: my goal is to make 7-piece tablebases available for all forms of usage./quote]

By all means fine. It's just that 7-piece is expected to be enormous for most engine testers. Bigger HDD and SSDs have to be invested. So if users can choose necessary files to keep that's wonderful.

[noobpwnftw]

Jon: I think this is always possible. Even a missing tablebase after capture is fine because of DTZ resets. The dependency only affects the building process.
And all TB files are distributed individually so users can choose what to download.