DTM50

Discussion of chess software programming and technical issues.

Moderators: hgm, Rebel, chrisw

User avatar
Nordlandia
Posts: 2821
Joined: Fri Sep 25, 2015 9:38 pm
Location: Sortland, Norway

Re: DTM50

Post by Nordlandia »

It's also nice to have the vast majority stored on external drive. Much easier to copy back and forth instead of downloading every file individually. Sata speed is usually faster than download speed. And downloading one file at a time is a headache, although it has to be done once. I had to do so with the 6-piece nalimov. Boring and tiresome but it has to be done.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: DTM50

Post by MikeB »

Nordlandia wrote: Mon Jul 08, 2019 7:47 pm It's also nice to have the vast majority stored on external drive. Much easier to copy back and forth instead of downloading every file individually. Sata speed is usually faster than download speed. And downloading one file at a time is a headache, although it has to be done once. I had to do so with the 6-piece nalimov. Boring and tiresome but it has to be done.
I wrote a script that downloads them automatically - I believe they let you have two connections at same time. It's a bash script. Let me see if I can find it.
Image
Joost Buijs
Posts: 1563
Joined: Thu Jul 16, 2009 10:47 am
Location: Almere, The Netherlands

Re: DTM50

Post by Joost Buijs »

Nordlandia wrote: Mon Jul 08, 2019 7:47 pm It's also nice to have the vast majority stored on external drive. Much easier to copy back and forth instead of downloading every file individually. Sata speed is usually faster than download speed. And downloading one file at a time is a headache, although it has to be done once. I had to do so with the 6-piece nalimov. Boring and tiresome but it has to be done.
True, of course I have a backup of 6-piece Nalimov and syzygy on my NAS.

I still remember when I first downloaded the 1.2 TB Nalimov TB, it was about 19 years ago, internet was really slow, at that time I had an 18 Mbit connection, so I could download at max. with 1.8 MB/sec. Now I have glass-fiber, 1 Gbit up/down, 50 times faster.
User avatar
MikeB
Posts: 4889
Joined: Thu Mar 09, 2006 6:34 am
Location: Pen Argyl, Pennsylvania

Re: DTM50

Post by MikeB »

MikeB wrote: Mon Jul 08, 2019 8:06 pm
Nordlandia wrote: Mon Jul 08, 2019 7:47 pm It's also nice to have the vast majority stored on external drive. Much easier to copy back and forth instead of downloading every file individually. Sata speed is usually faster than download speed. And downloading one file at a time is a headache, although it has to be done once. I had to do so with the 6-piece nalimov. Boring and tiresome but it has to be done.
I wrote a script that downloads them automatically - I believe they let you have two connections at same time. It's a bash script. Let me see if I can find it.
label this 'get.sh'

Code: Select all

#!/bin/bash

while true; do
  wget -r -np -nc -R "index.html*" -e robots=off  ftp://ftp.chessdb.cn/pub/syzygy//3-6men/ &
  wait
done
exit
example output

Code: Select all

Mac-Pro:~ cd /Volumes/S3/Z3
Mac-Pro:Z3 $ get.sh
--2019-07-08 14:30:43--  ftp://ftp.chessdb.cn/pub/syzygy//3-6men/
           => 'ftp.chessdb.cn/pub/syzygy/3-6men/.listing'
Resolving ftp.chessdb.cn... 112.73.74.24
Connecting to ftp.chessdb.cn|112.73.74.24|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /pub/syzygy//3-6men ... done.
==> PASV ... done.    ==> LIST ... done.

    [   <=>                                                               ] 71,523      73.0KB/s   in 1.0s   

2019-07-08 14:30:48 (73.0 KB/s) - 'ftp.chessdb.cn/pub/syzygy/3-6men/.listing' saved [71523]

Removed 'ftp.chessdb.cn/pub/syzygy/3-6men/.listing'.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBBvK.rtbw' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBBvK.rtbz' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBNvK.rtbw' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBNvK.rtbz' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBPvK.rtbw' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBPvK.rtbz' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvK.rtbw' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvK.rtbz' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKB.rtbw' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKB.rtbz' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKN.rtbw' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKN.rtbz' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKP.rtbw' already there; not retrieving.
File 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKP.rtbz' already there; not retrieving.
--2019-07-08 14:30:48--  ftp://ftp.chessdb.cn/pub/syzygy//3-6men/KBBBvKQ.rtbw
           => 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKQ.rtbw'
==> CWD not required.
==> PASV ... done.    ==> RETR KBBBvKQ.rtbw ... done.
Length: 123568656 (118M)

100%[====================================================================>] 123,568,656 2.59MB/s   in 57s    

2019-07-08 14:31:46 (2.07 MB/s) - 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKQ.rtbw' saved [123568656]

--2019-07-08 14:31:46--  ftp://ftp.chessdb.cn/pub/syzygy//3-6men/KBBBvKQ.rtbz
           => 'ftp.chessdb.cn/pub/syzygy/3-6men/KBBBvKQ.rtbz'
==> CWD not required.
==> PASV ... done.    ==> RETR KBBBvKQ.rtbz ... done.
If the connection breaks it will know where it left off ( while using the flags in the get.sh script)....

the tricky part is when your disk gets full ( should not be a problem here 3-6men) , and you have to start on another drive, how it will know that you have already downloaded the files ? in the new drive, use the touch command on the new drive to create empty files for all the ones you have already downloaded and are stored somewhere else - make sure you put them in the correct directory where the new files will be stored . The is only an issue when you dowmloading the 7 men files. Of course if you have a 20 tb external drive, you are good to go with no worries. Just run the above get.sh one comment for each of the subfolders ( modify the script for each subfolder)

Of course the script here assumes you want to download them all ...
Image
User avatar
Nordlandia
Posts: 2821
Joined: Fri Sep 25, 2015 9:38 pm
Location: Sortland, Norway

Re: DTM50

Post by Nordlandia »

My 5960X 4.5GHz 8-Core managed 18788 kN/s without probing tablebases at 4096 MB hash. At 1024 MB hash it was 20624 kN/s, 10% increase.

[d]1q6/1r2k1p1/4pp1p/1P1b1P2/3Q4/7P/4B1P1/2R3K1 w - - 0 1

Analysis by Stockfish 010719 64 BMI2:

1.Qg4 g5 2.fxg6 Kf8 3.h4 Qe5 4.Rc2 Ra7 5.Bf1 h5 6.Qe2 Kg7 7.Qxe5 fxe5 8.b6 Ra3 9.Rb2 Bb7 10.Bb5 Kxg6 11.Be8+ Kh6 12.Kf2 Ra8 13.Bd7 Rf8+ 14.Ke1 Rf4 15.g3 Rf3 16.Rb5 Rxg3 17.Rxe5 Kg7 18.Rxe6 Rg2 19.Be8 Rh2 20.Re7+ Kf8 21.Rxb7 Kxe8 22.Rh7 Rxh4 23.b7 Rb4 24.b8Q+ Rxb8 25.Rh8+ Ke7 26.Rxb8
White is better: +/- (1.07 --) Depth: 37/59 00:01:01 1152MN
(, 09.07.2019)
Joost Buijs
Posts: 1563
Joined: Thu Jul 16, 2009 10:47 am
Location: Almere, The Netherlands

Re: DTM50

Post by Joost Buijs »

It is all a bit off topic, but since we both use the same motherboard I want to make this remark.

With versions of Windows 10 after Spectre mitigations overclocking my i7-6950X did not work anymore, even with the latest 1902 beta BIOS it didn't work. I could enter overclocks in the BIOS, no problem, but when I looked at it with CPU-Z it was just at base frequency. It turned out to be a discrepancy between the microcode table in the BIOS and Windows 10.

I could solve the problem by replacing the microcode table in Windows 10 by an older one, but this was not very satisfactory because it breaks Spectre mitigations and after most Windows updates I had to replace the table again.

Asus really has bad support, they are aware of this problem, but they refuse to release a BIOS for the X99 Deluxe II with the latest Intel microcode in it (mainly because they want you to buy their new stuff), so I finally had to patch the BIOS with the latest Intel microcode myself which solved the problem indeed.

Only Broadwell processors seem to be affected by this, since yours is Haswell you probably don't have this problem, at least my 5960X overclocks fine on my old X99 Deluxe (which has the same outdated microcode table in the BIOS).