Page 3 of 4

Re: Texel tuning Zurichess quiet like generator

Posted: Sat Aug 10, 2019 4:49 pm
by xr_a_y
File is available here for some time : https://serv.x-ray.fr/chess/CCRL-quiet- ... edp.tar.gz (my home server)

Here is also a python script to randomly split it, because it is maybe to huge to be used ...

Code: Select all

import random
import sys

def split_huge_file(file,out1,out2,percentage=0.75,seed=42):
   random.seed(seed)
   with open(file, 'r') as fin, \
      open(out1, 'w') as foutBig, \
      open(out2, 'w') as foutSmall:
      for line in fin:
         r = random.random() 
         if r < percentage:
             foutBig.write(line)
         else:
             foutSmall.write(line)

split_huge_file(sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4]), int(sys.argv[5]))

Re: Texel tuning Zurichess quiet like generator

Posted: Sat Aug 10, 2019 5:17 pm
by fabianVDW
zenpawn wrote:
Sat Aug 10, 2019 3:40 pm
fabianVDW wrote:
Sat Aug 10, 2019 2:31 pm

Lichess Quiet( no illegal positions there) ) k: 1.0887, Error: 0.06516 Residual: 0.25526, STC Tests: +244 =261 -203 +20.14 +/- 25.61 So that seems to be a success

I will start LTC regression tests for the tuned params of Lichess Quiet against Version 1.9 and Version 1.9.2 (private version)
Is "Lichess Quiet" the second link from Jon above
Yes, it is
(perhaps with non-quiet positions removed?) or something else?
No, I don't remove non-quiets. I replace every position with the leaf of their q-search (which for quiet positions, is the position itself).

Re: Texel tuning Zurichess quiet like generator

Posted: Sat Aug 10, 2019 7:06 pm
by xr_a_y
xr_a_y wrote:
Sat Aug 10, 2019 4:49 pm
File is available here for some time : https://serv.x-ray.fr/chess/CCRL-quiet- ... edp.tar.gz (my home server)

Here is also a python script to randomly split it, because it is maybe to huge to be used ...

Code: Select all

import random
import sys

def split_huge_file(file,out1,out2,percentage=0.75,seed=42):
   random.seed(seed)
   with open(file, 'r') as fin, \
      open(out1, 'w') as foutBig, \
      open(out2, 'w') as foutSmall:
      for line in fin:
         r = random.random() 
         if r < percentage:
             foutBig.write(line)
         else:
             foutSmall.write(line)

split_huge_file(sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4]), int(sys.argv[5]))
The file has been updated, 35millions positions (instead of 38), removing duplicates.

Re: Texel tuning Zurichess quiet like generator

Posted: Sat Aug 10, 2019 7:45 pm
by fabianVDW
Thanks Vivien, I will download tomorrow

Re: Texel tuning Zurichess quiet like generator

Posted: Sun Aug 11, 2019 11:58 am
by fabianVDW
fabianVDW wrote:
Sat Aug 10, 2019 2:31 pm

Lichess Quiet( no illegal positions there) ) k: 1.0887, Error: 0.06516 Residual: 0.25526, STC Tests: +244 =261 -203 +20.14 +/- 25.61 So that seems to be a success

I will start LTC regression tests for the tuned params of Lichess Quiet against Version 1.9 and Version 1.9.2 (private version)

Fabi
LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.

Will soon test again with part of the CCRL data.

Fabi

Re: Texel tuning Zurichess quiet like generator

Posted: Sun Aug 11, 2019 12:50 pm
by zenpawn
fabianVDW wrote:
Sun Aug 11, 2019 11:58 am

LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.

Will soon test again with part of the CCRL data.

Fabi
Very nice indeed. Would you mind sharing your modified version of lichess-quiet?

Re: Texel tuning Zurichess quiet like generator

Posted: Sun Aug 11, 2019 2:11 pm
by fabianVDW
zenpawn wrote:
Sun Aug 11, 2019 12:50 pm
fabianVDW wrote:
Sun Aug 11, 2019 11:58 am

LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.

Will soon test again with part of the CCRL data.

Fabi
Very nice indeed. Would you mind sharing your modified version of lichess-quiet?
https://drive.google.com/file/d/1ZtR1My ... sp=sharing File is there.

I am afraid It is not your typical epd encoding, it has historical (dumb) reasons why it is not. It should still be pretty easy to parse just split every line at "|" and you can extract FEN and Game Result.

Fabi

Re: Texel tuning Zurichess quiet like generator

Posted: Sun Aug 11, 2019 7:46 pm
by zenpawn
fabianVDW wrote:
Sun Aug 11, 2019 2:11 pm
I am afraid It is not your typical epd encoding, it has historical (dumb) reasons why it is not. It should still be pretty easy to parse just split every line at "|" and you can extract FEN and Game Result.
Thank you. I did a few search and replace operations, and now it's all set. :)

Re: Texel tuning Zurichess quiet like generator

Posted: Mon Aug 12, 2019 6:40 pm
by fabianVDW
Tried now with 20% of CCRL data (around 7mil positions):

k: 0.6089, Error: 0.09379 Residual: 0.30625 STC Tests: Cancelled early +38 =46 -106 -130.1 +/-52.7 Elo

Data seems to be too badly labelled.

Re: Texel tuning Zurichess quiet like generator

Posted: Mon Aug 12, 2019 8:17 pm
by xr_a_y
fabianVDW wrote:
Mon Aug 12, 2019 6:40 pm
Tried now with 20% of CCRL data (around 7mil positions):

k: 0.6089, Error: 0.09379 Residual: 0.30625 STC Tests: Cancelled early +38 =46 -106 -130.1 +/-52.7 Elo

Data seems to be too badly labelled.
ok thanks for testing, these are only >2800 elo games. The result shall be quite reliable.