Discussion of chess software programming and technical issues.
Moderators: hgm , Rebel , chrisw
xr_a_y
Posts: 1871 Joined: Sat Nov 25, 2017 2:28 pm
Location: France
Post
by xr_a_y » Sat Aug 10, 2019 6:49 pm
File is available here for some time :
https://serv.x-ray.fr/chess/CCRL-quiet- ... edp.tar.gz (my home server)
Here is also a python script to randomly split it, because it is maybe to huge to be used ...
Code: Select all
import random
import sys
def split_huge_file(file,out1,out2,percentage=0.75,seed=42):
random.seed(seed)
with open(file, 'r') as fin, \
open(out1, 'w') as foutBig, \
open(out2, 'w') as foutSmall:
for line in fin:
r = random.random()
if r < percentage:
foutBig.write(line)
else:
foutSmall.write(line)
split_huge_file(sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4]), int(sys.argv[5]))
fabianVDW
Posts: 146 Joined: Fri Mar 15, 2019 8:46 pm
Location: Germany
Full name: Fabian von der Warth
Post
by fabianVDW » Sat Aug 10, 2019 7:17 pm
zenpawn wrote: ↑ Sat Aug 10, 2019 5:40 pm
fabianVDW wrote: ↑ Sat Aug 10, 2019 4:31 pm
Lichess Quiet( no illegal positions there) ) k: 1.0887, Error: 0.06516 Residual: 0.25526, STC Tests: +244 =261 -203 +20.14 +/- 25.61 So that seems to be a success
I will start LTC regression tests for the tuned params of Lichess Quiet against Version 1.9 and Version 1.9.2 (private version)
Is "Lichess Quiet" the second link from Jon above
Yes, it is
(perhaps with non-quiet positions removed?) or something else?
No, I don't remove non-quiets. I replace every position with the leaf of their q-search (which for quiet positions, is the position itself).
xr_a_y
Posts: 1871 Joined: Sat Nov 25, 2017 2:28 pm
Location: France
Post
by xr_a_y » Sat Aug 10, 2019 9:06 pm
xr_a_y wrote: ↑ Sat Aug 10, 2019 6:49 pm
File is available here for some time :
https://serv.x-ray.fr/chess/CCRL-quiet- ... edp.tar.gz (my home server)
Here is also a python script to randomly split it, because it is maybe to huge to be used ...
Code: Select all
import random
import sys
def split_huge_file(file,out1,out2,percentage=0.75,seed=42):
random.seed(seed)
with open(file, 'r') as fin, \
open(out1, 'w') as foutBig, \
open(out2, 'w') as foutSmall:
for line in fin:
r = random.random()
if r < percentage:
foutBig.write(line)
else:
foutSmall.write(line)
split_huge_file(sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4]), int(sys.argv[5]))
The file has been updated, 35millions positions (instead of 38), removing duplicates.
fabianVDW
Posts: 146 Joined: Fri Mar 15, 2019 8:46 pm
Location: Germany
Full name: Fabian von der Warth
Post
by fabianVDW » Sat Aug 10, 2019 9:45 pm
Thanks Vivien, I will download tomorrow
fabianVDW
Posts: 146 Joined: Fri Mar 15, 2019 8:46 pm
Location: Germany
Full name: Fabian von der Warth
Post
by fabianVDW » Sun Aug 11, 2019 1:58 pm
fabianVDW wrote: ↑ Sat Aug 10, 2019 4:31 pm
Lichess Quiet( no illegal positions there) ) k: 1.0887, Error: 0.06516 Residual: 0.25526, STC Tests: +244 =261 -203 +20.14 +/- 25.61 So that seems to be a success
I will start LTC regression tests for the tuned params of Lichess Quiet against Version 1.9 and Version 1.9.2 (private version)
Fabi
LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.
Will soon test again with part of the CCRL data.
Fabi
zenpawn
Posts: 349 Joined: Sat Aug 06, 2016 8:31 pm
Location: United States
Post
by zenpawn » Sun Aug 11, 2019 2:50 pm
fabianVDW wrote: ↑ Sun Aug 11, 2019 1:58 pm
LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.
Will soon test again with part of the CCRL data.
Fabi
Very nice indeed. Would you mind sharing your modified version of lichess-quiet?
Erin Dame
Author of RookieMonster
fabianVDW
Posts: 146 Joined: Fri Mar 15, 2019 8:46 pm
Location: Germany
Full name: Fabian von der Warth
Post
by fabianVDW » Sun Aug 11, 2019 4:11 pm
zenpawn wrote: ↑ Sun Aug 11, 2019 2:50 pm
fabianVDW wrote: ↑ Sun Aug 11, 2019 1:58 pm
LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.
Will soon test again with part of the CCRL data.
Fabi
Very nice indeed. Would you mind sharing your modified version of lichess-quiet?
https://drive.google.com/file/d/1ZtR1My ... sp=sharing File is there.
I am afraid It is not your typical epd encoding, it has historical (dumb) reasons why it is not. It should still be pretty easy to parse just split every line at "|" and you can extract FEN and Game Result.
Fabi
zenpawn
Posts: 349 Joined: Sat Aug 06, 2016 8:31 pm
Location: United States
Post
by zenpawn » Sun Aug 11, 2019 9:46 pm
fabianVDW wrote: ↑ Sun Aug 11, 2019 4:11 pm
I am afraid It is not your typical epd encoding, it has historical (dumb) reasons why it is not. It should still be pretty easy to parse just split every line at "|" and you can extract FEN and Game Result.
Thank you. I did a few search and replace operations, and now it's all set.
Erin Dame
Author of RookieMonster
fabianVDW
Posts: 146 Joined: Fri Mar 15, 2019 8:46 pm
Location: Germany
Full name: Fabian von der Warth
Post
by fabianVDW » Mon Aug 12, 2019 8:40 pm
Tried now with 20% of CCRL data (around 7mil positions):
k: 0.6089, Error: 0.09379 Residual: 0.30625 STC Tests: Cancelled early +38 =46 -106 -130.1 +/-52.7 Elo
Data seems to be too badly labelled.
xr_a_y
Posts: 1871 Joined: Sat Nov 25, 2017 2:28 pm
Location: France
Post
by xr_a_y » Mon Aug 12, 2019 10:17 pm
fabianVDW wrote: ↑ Mon Aug 12, 2019 8:40 pm
Tried now with 20% of CCRL data (around 7mil positions):
k: 0.6089, Error: 0.09379 Residual: 0.30625 STC Tests: Cancelled early +38 =46 -106 -130.1 +/-52.7 Elo
Data seems to be too badly labelled.
ok thanks for testing, these are only >2800 elo games. The result shall be quite reliable.