In praise of git, cutechess-cli, bayeselo, and pgn-extract

[Aaron Becker]

I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.

• Every change I make is recorded in version control, and all testing results can be associated with a git commit-id that allows me to see exactly what the code looked like when I ran the tests.
• I test my changes using several thousand games against multiple at 1s+0.1s time control. Cutechess-cli makes this very easy, and its concurrency feature simplifies things a lot.
• Bayeselo makes it very easy to quantify the effect of my changes, complete with likelihood-of-superiority numbers.
• PGN-Extract lets me look at the kinds of games that Daydreamer tends to win or lose and lets me home in on the areas that need work. For instance, it's easy to isolate just the games that feature pawn endgames, or just the games that end in 20 moves or fewer.

These tools are all free, and all open source, and all incredibly useful to me. I wanted to create this thread for two reasons. First, to thank the authors of these tools for their efforts--thanks to David Barnes, Rémi Coulom, Ilari Pihlajisto, Arto Jonsson, and all the authors of git. I appreciate all your hard work. Second, to promote pervasive high-speed testing as a road to engine improvement. It seems perverse to test at such fast time controls, but as I've migrated to faster and faster games and more and more of them, my results have been consistently good.

It's certainly possible to create an engine that's weak at fast time controls and strong at longer time controls, but I've never seen an engine that's extremely strong at fast time controls that's not strong in other scenarios as well. When you combine the large number of games from high speed testing with a reliable way to sort through those games and extract useful information about engine strengths and weaknesses, you have an incredibly powerful tool to improve your engine, and it's completely practical to run on modest desktop hardware.

[swami]

I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.

That's good to know, Aaron. I'm pleased to hear that one of the favorite engines of mine is frequently being experimented and improved.

[Aaron Becker]

I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.

That's good to know, Aaron. I'm pleased to hear that one of the favorite engines of mine is frequently being worked on.

I should add that the strategic test suite is a great tool when you're working on tuning evaluation parameters like mobility and outposts. It's much faster to run through STS than to play thousands of test games, which lets you try ideas a lot faster, and then only subject the most promising ideas to a full gauntlet of tests. So thanks for your work too, Swami

[michiguel]

I'm really happy with the progress I've been making with Daydreamer lately, and I credit my success to the fast testing scheme I've adopted.

• Every change I make is recorded in version control, and all testing results can be associated with a git commit-id that allows me to see exactly what the code looked like when I ran the tests.
• I test my changes using several thousand games against multiple at 1s+0.1s time control. Cutechess-cli makes this very easy, and its concurrency feature simplifies things a lot.
• Bayeselo makes it very easy to quantify the effect of my changes, complete with likelihood-of-superiority numbers.
• PGN-Extract lets me look at the kinds of games that Daydreamer tends to win or lose and lets me home in on the areas that need work. For instance, it's easy to isolate just the games that feature pawn endgames, or just the games that end in 20 moves or fewer.

These tools are all free, and all open source, and all incredibly useful to me. I wanted to create this thread for two reasons. First, to thank the authors of these tools for their efforts--thanks to David Barnes, Rémi Coulom, Ilari Pihlajisto, Arto Jonsson, and all the authors of git. I appreciate all your hard work. Second, to promote pervasive high-speed testing as a road to engine improvement. It seems perverse to test at such fast time controls, but as I've migrated to faster and faster games and more and more of them, my results have been consistently good.

It's certainly possible to create an engine that's weak at fast time controls and strong at longer time controls, but I've never seen an engine that's extremely strong at fast time controls that's not strong in other scenarios as well. When you combine the large number of games from high speed testing with a reliable way to sort through those games and extract useful information about engine strengths and weaknesses, you have an incredibly powerful tool to improve your engine, and it's completely practical to run on modest desktop hardware.

Thanks God for Git.

Git: Gold Medal
The ability to work on separate branches so easily is absolutely priceless. I have no idea how the heck things are combined so easily. It scares me.

Silver:
Another software that I fell in love with = Rake ("make" but uses a real Ruby language) + Ruby. Now I do not hate to write scripts. For PGO compiles I need to type "rake pgo" and I wait 20 minutes. I made one 1/2 hour before in one of the Monthly blitz.

Bronze: Doxygen (Thanks Dann Corbit, for recommending it).

Miguel

[ilari]

Thank you Aaron, I'm humbled by your kind words. Ironically the development of Cute Chess (and its "cli" subproject) has taken away most of the time that I would've otherwise spent developing my chess engine. But it's been a lot more rewarding as well.

I haven't played with Daydreamer enough to decide if I like the kind of chess it plays, but after browsing its git repository for a while I knew this engine would be a success. In most open source chess engines I see code that makes me think "if I wrote something like that I'd get fired", but not Daydreamer. The code is well-written and thoroughly documented, commit history is easy to follow, and the developer just seems to know what he's doing.

[Aaron Becker]

Thank you Aaron, I'm humbled by your kind words. Ironically the development of Cute Chess (and its "cli" subproject) has taken away most of the time that I would've otherwise spent developing my chess engine. But it's been a lot more rewarding as well.

I haven't played with Daydreamer enough to decide if I like the kind of chess it plays, but after browsing its git repository for a while I knew this engine would be a success. In most open source chess engines I see code that makes me think "if I wrote something like that I'd get fired", but not Daydreamer. The code is well-written and thoroughly documented, commit history is easy to follow, and the developer just seems to know what he's doing.

This is way too flattering; I'm doing my best not to blush. The knowledge that anyone can look through your code is a powerful motivating force for documentation and code cleanup, although I always wonder whether or not anyone actually looks at it. I've never read through Sloppy's source, but I feel a little kinship with it just based on the fact that both it and Daydreamer are written in C99, version controlled with git, and have slightly self-deprecating names. I have spent a little time with Cute Chess's source, and it's extremely well-engineered. I think the way that it separates chess logic from interface logic is excellent.

I think Daydreamer's overall style is a weak point right now, but I've stayed away from working on it because I think it will be more productive to work on less subjective areas first, and only tune up the style once the other parts of the engine are more mature. Getting king safety right is especially challenging.

[jshriver]

Where can you download cutechess-cli?
I came across a git repo but it was empty.

-Josh

[Ron Murawski]

Where can you download cutechess-cli?
I came across a git repo but it was empty.

-Josh

A good place to look for stuff like that is on my Chess News List
http://computer-chess.org/doku.php?id=c ... _news_list

Look for the date 2009/11/30 when cutechess-cli was last updated.

Ron

[jshriver]

Do you know where to fine the source? I can run this under wine but was hoping to try for a linux build. Either way appreciate your help at least have the windows version.

Nice site as well
-Josh

[ilari]

Do you know where to fine the source? I can run this under wine but was hoping to try for a linux build. Either way appreciate your help at least have the windows version.

Nice site as well
-Josh

The source code is here: http://repo.or.cz/w/sloppygui.git
The git repository still carries the legacy name "sloppygui", but the program doesn't have anything to do with Sloppy anymore.
Note that the repository contains 3 projects: cutechess-lib, cutechess-gui, and cutechess-cli. The gui project is still at an early state, and not very useful for anything but experimentation.

If you just want the binaries, go here: http://koti.mbnet.fi/~ilaripih/bin/