txt: automated chess engine tuning

Discussion of chess software programming and technical issues.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

txt: automated chess engine tuning

Post by brtzsnr » Wed Mar 18, 2015 11:13 am

Hi, all!

I wrote a small framework to do automated chess tuning based on Texel's Tuning method (https://chessprogramming.wikispaces.com ... ing+Method). You can find the source code and instructions how to use it here https://bitbucket.org/brtzsnr/txt.

I'm still experimenting with it, so I cannot yet report any success. Nevertheless, feel free to experiment with txt and if you find it useful, please consider contributing.

Regards,

Engin
Posts: 831
Joined: Mon Jan 05, 2009 6:40 pm
Location: Germany
Full name: Engin Üstün
Contact:

Re: txt: automated chess engine tuning

Post by Engin » Wed Mar 18, 2015 10:57 pm

Great !

is there a compiled exe file for windows allready ?

i have allready in my Tornado the texel method but i am interesting if your method is much easier to use or if its really working right.

the texel method are working only for my material score, but if i want to tune some other values like pawn its give a wrong sign of the value, i don't know why is this happen.

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Thu Mar 19, 2015 11:18 am

Engin wrote: is there a compiled exe file for windows allready ?
I can't provide an exe because you have to write a small program to configure which variables to tune. See the instructions. It's like a configuration file, but it is much easier to write and allows you to do some more complicated stuff.

Code: Select all

package main
import "bitbucket.org/brtzsnr/txt/txt"
func main() {
    opts := make(txt.Options)
    opts["EndGameMaterial.BishopPairBonus"] = txt.NewIntegerRange(10, 100)
    opts["MidGameMaterial.BishopPairBonus"] = txt.NewIntegerRange(10, 100)
    txt.Run("setvalue {{.Name}} {{.Value}}\n", opts)
}

jdart
Posts: 3719
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: txt: automated chess engine tuning

Post by jdart » Thu Mar 19, 2015 7:05 pm

I am giving up on this method after spending quite some time experimenting with it.

See http://arasanchess.org/blog.shtml.

--Jon

Engin
Posts: 831
Joined: Mon Jan 05, 2009 6:40 pm
Location: Germany
Full name: Engin Üstün
Contact:

Re: txt: automated chess engine tuning

Post by Engin » Thu Mar 19, 2015 7:18 pm

yes, agree with you, in this case the tuning method is unlogical, some values become very high or low values that i can not understand how this can be, so better tuning with own logical brain. Maybe i have an bug anywhere, i have checked everywhere and did not found it.

Engin
Posts: 831
Joined: Mon Jan 05, 2009 6:40 pm
Location: Germany
Full name: Engin Üstün
Contact:

Re: txt: automated chess engine tuning

Post by Engin » Thu Mar 19, 2015 7:36 pm

i said i have allready implemented the texel method with some help from Martin Sedlak and Peter Österlund it self in my Tornado, so i am not need your code to run. just i was interested if that would be really work. it is not easy because it make differences between wich PGN games you are using CC games, master games or self played games from own engine. everytime i started i get other strange values and K is each time different. Peter checked after i send him my code that i have many doubled positions in there, so i have remove repeated position out too. of course there are no mates and i save only after 10 book moves in my filtered EPD file. What i am not understanding is that is work for material values good and why its not working for my pawns where i give them negative values and its giving me positive values what is nonsense. for example doubled pawns = -0.10cp or isolated pawns = -0.20cp i have and i am sure that all other programmers giving penalty values for double, isolated, backward pawns too. the final best value it say give me some kind of +1.5 cp what is incredible nonsense

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Thu Mar 19, 2015 8:10 pm

I tried generating ~100.000 positions from stockfish games and optimized a few values. Initial results after 500 games were promising (10-40ELO increase) but after 2000 games all elo increase disappeared.

Now I'm trying the same with 900.000 positions from 12.000 self played games.

I removed all duplicate positions (so this should remove most of the openings), but I did not remove positions close to mate. I couldn't find how to do that with pgn-extract. Maybe somebody can suggest a way to cleanup more of the positions?

I know that most engine authors have difficulty tuning more than a few values, and I'm hoping that txt will fill some of the void. I need, however, feed back of more experienced programmers :)

Ferdy
Posts: 3813
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: txt: automated chess engine tuning

Post by Ferdy » Thu Mar 19, 2015 9:09 pm

brtzsnr wrote: I removed all duplicate positions (so this should remove most of the openings), but I did not remove positions close to mate. I couldn't find how to do that with pgn-extract. Maybe somebody can suggest a way to cleanup more of the positions?
I have this window's executable where it will analyze a fen positions in a file with uci engine, say sf6 and any position with search score within a given window say [-50/+3000] will be saved. Just let the engine analyze at 50ms/position for example.
I use that window to generate my training sets. If interested I will upload it.

brtzsnr
Posts: 433
Joined: Fri Jan 16, 2015 3:02 pm
Contact:

Re: txt: automated chess engine tuning

Post by brtzsnr » Thu Mar 19, 2015 10:13 pm

I wonder why you chose such an asymmetric score window [-50/+3000]. Is there a good reason behind it?

User avatar
michiguel
Posts: 6386
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: txt: automated chess engine tuning

Post by michiguel » Thu Mar 19, 2015 10:52 pm

Ferdy wrote:
brtzsnr wrote: I removed all duplicate positions (so this should remove most of the openings), but I did not remove positions close to mate. I couldn't find how to do that with pgn-extract. Maybe somebody can suggest a way to cleanup more of the positions?
I have this window's executable where it will analyze a fen positions in a file with uci engine, say sf6 and any position with search score within a given window say [-50/+3000] will be saved. Just let the engine analyze at 50ms/position for example.
I use that window to generate my training sets. If interested I will upload it.
Gaviota uses ALL positions that are quiet (no captures in the next 6 plies of the game) from engines games. I have like 3,000,000 positions. I do not think that being selective is a good idea. The whole thing is probabilistic, so the more the merrier. It should not be only games from top engines! that would completely defeat the purpose and narrow the landscape.

Miguel


Miguel

Post Reply