In praise of git, cutechess-cli, bayeselo, and pgn-extract

Discussion of chess software programming and technical issues.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Aaron Becker
Posts: 292
Joined: Tue Jul 07, 2009 2:56 am

In praise of git, cutechess-cli, bayeselo, and pgn-extract

Post by Aaron Becker » Tue Mar 02, 2010 4:33 am

I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.
  • Every change I make is recorded in version control, and all testing results can be associated with a git commit-id that allows me to see exactly what the code looked like when I ran the tests.
  • I test my changes using several thousand games against multiple at 1s+0.1s time control. Cutechess-cli makes this very easy, and its concurrency feature simplifies things a lot.
  • Bayeselo makes it very easy to quantify the effect of my changes, complete with likelihood-of-superiority numbers.
  • PGN-Extract lets me look at the kinds of games that Daydreamer tends to win or lose and lets me home in on the areas that need work. For instance, it's easy to isolate just the games that feature pawn endgames, or just the games that end in 20 moves or fewer.
These tools are all free, and all open source, and all incredibly useful to me. I wanted to create this thread for two reasons. First, to thank the authors of these tools for their efforts--thanks to David Barnes, Rémi Coulom, Ilari Pihlajisto, Arto Jonsson, and all the authors of git. I appreciate all your hard work. Second, to promote pervasive high-speed testing as a road to engine improvement. It seems perverse to test at such fast time controls, but as I've migrated to faster and faster games and more and more of them, my results have been consistently good.

It's certainly possible to create an engine that's weak at fast time controls and strong at longer time controls, but I've never seen an engine that's extremely strong at fast time controls that's not strong in other scenarios as well. When you combine the large number of games from high speed testing with a reliable way to sort through those games and extract useful information about engine strengths and weaknesses, you have an incredibly powerful tool to improve your engine, and it's completely practical to run on modest desktop hardware.

swami
Posts: 6535
Joined: Thu Mar 09, 2006 3:21 am

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by swami » Tue Mar 02, 2010 4:42 am

Aaron Becker wrote:I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.
That's good to know, Aaron. I'm pleased to hear that one of the favorite engines of mine is frequently being experimented and improved. :)

Aaron Becker
Posts: 292
Joined: Tue Jul 07, 2009 2:56 am

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by Aaron Becker » Tue Mar 02, 2010 4:45 am

swami wrote:
Aaron Becker wrote:I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.
That's good to know, Aaron. I'm pleased to hear that one of the favorite engines of mine is frequently being worked on. :)
I should add that the strategic test suite is a great tool when you're working on tuning evaluation parameters like mobility and outposts. It's much faster to run through STS than to play thousands of test games, which lets you try ideas a lot faster, and then only subject the most promising ideas to a full gauntlet of tests. So thanks for your work too, Swami :)

User avatar
michiguel
Posts: 6388
Joined: Thu Mar 09, 2006 7:30 pm
Location: Chicago, Illinois, USA
Contact:

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by michiguel » Tue Mar 02, 2010 6:16 am

Aaron Becker wrote:I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.
  • Every change I make is recorded in version control, and all testing results can be associated with a git commit-id that allows me to see exactly what the code looked like when I ran the tests.
  • I test my changes using several thousand games against multiple at 1s+0.1s time control. Cutechess-cli makes this very easy, and its concurrency feature simplifies things a lot.
  • Bayeselo makes it very easy to quantify the effect of my changes, complete with likelihood-of-superiority numbers.
  • PGN-Extract lets me look at the kinds of games that Daydreamer tends to win or lose and lets me home in on the areas that need work. For instance, it's easy to isolate just the games that feature pawn endgames, or just the games that end in 20 moves or fewer.
These tools are all free, and all open source, and all incredibly useful to me. I wanted to create this thread for two reasons. First, to thank the authors of these tools for their efforts--thanks to David Barnes, Rémi Coulom, Ilari Pihlajisto, Arto Jonsson, and all the authors of git. I appreciate all your hard work. Second, to promote pervasive high-speed testing as a road to engine improvement. It seems perverse to test at such fast time controls, but as I've migrated to faster and faster games and more and more of them, my results have been consistently good.

It's certainly possible to create an engine that's weak at fast time controls and strong at longer time controls, but I've never seen an engine that's extremely strong at fast time controls that's not strong in other scenarios as well. When you combine the large number of games from high speed testing with a reliable way to sort through those games and extract useful information about engine strengths and weaknesses, you have an incredibly powerful tool to improve your engine, and it's completely practical to run on modest desktop hardware.
Thanks God for Git.

Git: Gold Medal
The ability to work on separate branches so easily is absolutely priceless. I have no idea how the heck things are combined so easily. It scares me.

Silver:
Another software that I fell in love with = Rake ("make" but uses a real Ruby language) + Ruby. Now I do not hate to write scripts. For PGO compiles I need to type "rake pgo" and I wait 20 minutes. I made one 1/2 hour before in one of the Monthly blitz.

Bronze: Doxygen (Thanks Dann Corbit, for recommending it).

Miguel

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by ilari » Tue Mar 02, 2010 5:11 pm

Thank you Aaron, I'm humbled by your kind words. Ironically the development of Cute Chess (and its "cli" subproject) has taken away most of the time that I would've otherwise spent developing my chess engine. But it's been a lot more rewarding as well.

I haven't played with Daydreamer enough to decide if I like the kind of chess it plays, but after browsing its git repository for a while I knew this engine would be a success. In most open source chess engines I see code that makes me think "if I wrote something like that I'd get fired", but not Daydreamer. The code is well-written and thoroughly documented, commit history is easy to follow, and the developer just seems to know what he's doing.

Aaron Becker
Posts: 292
Joined: Tue Jul 07, 2009 2:56 am

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by Aaron Becker » Tue Mar 02, 2010 7:29 pm

ilari wrote:Thank you Aaron, I'm humbled by your kind words. Ironically the development of Cute Chess (and its "cli" subproject) has taken away most of the time that I would've otherwise spent developing my chess engine. But it's been a lot more rewarding as well.

I haven't played with Daydreamer enough to decide if I like the kind of chess it plays, but after browsing its git repository for a while I knew this engine would be a success. In most open source chess engines I see code that makes me think "if I wrote something like that I'd get fired", but not Daydreamer. The code is well-written and thoroughly documented, commit history is easy to follow, and the developer just seems to know what he's doing.
This is way too flattering; I'm doing my best not to blush. The knowledge that anyone can look through your code is a powerful motivating force for documentation and code cleanup, although I always wonder whether or not anyone actually looks at it. I've never read through Sloppy's source, but I feel a little kinship with it just based on the fact that both it and Daydreamer are written in C99, version controlled with git, and have slightly self-deprecating names. :) I have spent a little time with Cute Chess's source, and it's extremely well-engineered. I think the way that it separates chess logic from interface logic is excellent.

I think Daydreamer's overall style is a weak point right now, but I've stayed away from working on it because I think it will be more productive to work on less subjective areas first, and only tune up the style once the other parts of the engine are more mature. Getting king safety right is especially challenging.

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by jshriver » Tue Mar 02, 2010 8:28 pm

Where can you download cutechess-cli?
I came across a git repo but it was empty.

-Josh

Ron Murawski
Posts: 397
Joined: Sun Oct 29, 2006 3:38 am
Location: Schenectady, NY
Contact:

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by Ron Murawski » Tue Mar 02, 2010 9:58 pm

jshriver wrote:Where can you download cutechess-cli?
I came across a git repo but it was empty.

-Josh
A good place to look for stuff like that is on my Chess News List
http://computer-chess.org/doku.php?id=c ... _news_list

Look for the date 2009/11/30 when cutechess-cli was last updated.

Ron

User avatar
jshriver
Posts: 1105
Joined: Wed Mar 08, 2006 8:41 pm
Location: Morgantown, WV, USA
Contact:

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by jshriver » Tue Mar 02, 2010 10:11 pm

Do you know where to fine the source? I can run this under wine but was hoping to try for a linux build. Either way appreciate your help at least have the windows version.

Nice site as well :)
-Josh

User avatar
ilari
Posts: 750
Joined: Mon Mar 27, 2006 5:45 pm
Location: Finland
Contact:

Re: In praise of git, cutechess-cli, bayeselo, and pgn-extra

Post by ilari » Tue Mar 02, 2010 10:35 pm

jshriver wrote:Do you know where to fine the source? I can run this under wine but was hoping to try for a linux build. Either way appreciate your help at least have the windows version.

Nice site as well :)
-Josh
The source code is here: http://repo.or.cz/w/sloppygui.git
The git repository still carries the legacy name "sloppygui", but the program doesn't have anything to do with Sloppy anymore.
Note that the repository contains 3 projects: cutechess-lib, cutechess-gui, and cutechess-cli. The gui project is still at an early state, and not very useful for anything but experimentation.

If you just want the binaries, go here: http://koti.mbnet.fi/~ilaripih/bin/

Post Reply