Texel tuning Zurichess quiet like generator

[xr_a_y]

File is available here for some time : https://serv.x-ray.fr/chess/CCRL-quiet- ... edp.tar.gz (my home server)

Here is also a python script to randomly split it, because it is maybe to huge to be used ...

Code: Select all

import random
import sys

def split_huge_file(file,out1,out2,percentage=0.75,seed=42):
   random.seed(seed)
   with open(file, 'r') as fin, \
      open(out1, 'w') as foutBig, \
      open(out2, 'w') as foutSmall:
      for line in fin:
         r = random.random() 
         if r < percentage:
             foutBig.write(line)
         else:
             foutSmall.write(line)

split_huge_file(sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4]), int(sys.argv[5]))

[fabianVDW]

Lichess Quiet( no illegal positions there) ) k: 1.0887, Error: 0.06516 Residual: 0.25526, STC Tests: +244 =261 -203 +20.14 +/- 25.61 So that seems to be a success

I will start LTC regression tests for the tuned params of Lichess Quiet against Version 1.9 and Version 1.9.2 (private version)

Is "Lichess Quiet" the second link from Jon above

Yes, it is

(perhaps with non-quiet positions removed?) or something else?

No, I don't remove non-quiets. I replace every position with the leaf of their q-search (which for quiet positions, is the position itself).

[xr_a_y]

File is available here for some time : https://serv.x-ray.fr/chess/CCRL-quiet- ... edp.tar.gz (my home server)

Here is also a python script to randomly split it, because it is maybe to huge to be used ...

Code: Select all

import random
import sys

def split_huge_file(file,out1,out2,percentage=0.75,seed=42):
   random.seed(seed)
   with open(file, 'r') as fin, \
      open(out1, 'w') as foutBig, \
      open(out2, 'w') as foutSmall:
      for line in fin:
         r = random.random() 
         if r < percentage:
             foutBig.write(line)
         else:
             foutSmall.write(line)

split_huge_file(sys.argv[1], sys.argv[2], sys.argv[3], float(sys.argv[4]), int(sys.argv[5]))

The file has been updated, 35millions positions (instead of 38), removing duplicates.

[fabianVDW]

Thanks Vivien, I will download tomorrow

[fabianVDW]

Lichess Quiet( no illegal positions there) ) k: 1.0887, Error: 0.06516 Residual: 0.25526, STC Tests: +244 =261 -203 +20.14 +/- 25.61 So that seems to be a success

I will start LTC regression tests for the tuned params of Lichess Quiet against Version 1.9 and Version 1.9.2 (private version)

Fabi

LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.

Will soon test again with part of the CCRL data.

Fabi

[zenpawn]

LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.

Will soon test again with part of the CCRL data.

Fabi

Very nice indeed. Would you mind sharing your modified version of lichess-quiet?

[fabianVDW]

LTC tests passed. +289 =363 -211 +31.49 +/- 23.26 ELO. Nice.

Will soon test again with part of the CCRL data.

Fabi

Very nice indeed. Would you mind sharing your modified version of lichess-quiet?

https://drive.google.com/file/d/1ZtR1My ... sp=sharing File is there.

I am afraid It is not your typical epd encoding, it has historical (dumb) reasons why it is not. It should still be pretty easy to parse just split every line at "|" and you can extract FEN and Game Result.

Fabi

[zenpawn]

I am afraid It is not your typical epd encoding, it has historical (dumb) reasons why it is not. It should still be pretty easy to parse just split every line at "|" and you can extract FEN and Game Result.

Thank you. I did a few search and replace operations, and now it's all set.

[fabianVDW]

Tried now with 20% of CCRL data (around 7mil positions):

k: 0.6089, Error: 0.09379 Residual: 0.30625 STC Tests: Cancelled early +38 =46 -106 -130.1 +/-52.7 Elo

Data seems to be too badly labelled.

[xr_a_y]

Tried now with 20% of CCRL data (around 7mil positions):

k: 0.6089, Error: 0.09379 Residual: 0.30625 STC Tests: Cancelled early +38 =46 -106 -130.1 +/-52.7 Elo

Data seems to be too badly labelled.

ok thanks for testing, these are only >2800 elo games. The result shall be quite reliable.