The last week or so I went down a bit of a rabbit hole trying to make a bot on Twitter that could identify chess problems and try to post a solution - using Vice as the solver.
Here's a picture of it actually replying..
It turns out that identifying a chess board in an image is quite a challenge! (for me, anyway).
The process it goes through is:
1. Uses the Twitter v2 api to search for tweets containing variations on 'white to play', 'black to play' etc.
2. Sees if there is an image with the text.
3 Tries to extract the board from the image. This is done in as follows:
- Convert the image to B&W trying various thresholds of levels
- Use template matching to find the 49 intersections between dark and light squares. Do some 'sanity checks' so assess whether the corners are fitting more or less a large squares
- Calculate the square size from the corners (which can differ by a pixel or two), and chop the board into 64 squares
Here's a picture:
4. Run each square through a NN model and identify the square
- This part was really tricky. I spent days trying all sorts - template matching, slicing sedges, greyscale template matching, 'cheating' and using the board colours to know which site it's likely from. None of it worked.
- In the end, I wrote a script to scrape sites like Chess.com and get a big collection of piece images and square colour options and design. A few, like marble and wood, I cut myself. I also used Blitzin, Dasher and some other chess software. I ended up with a lot of squares and pieces. Most of this was automated, thank goodness.
- Created a massive set of all combinations of pieces and board designs using all variations of square colour I'd seen / scraped. There were about 700k squares in the end - each piece + dark sq + light + sq for each design.
- **edit** some of these were augmented - so I coloured them artificially to make variations
- Trained these using a NN (well, Tensorflow did it for me) with training set about 500k images, Vaildation 100k, the rest for test. The images were 48x48 in size (RGB)
- Ran out of RAM a lot and got very angry **edit** Keras has Data Generator classes which turned out to be life savers. They can be used to batch load data from disk
- in the end the model finished with 0.9914 accuracy, validation 1.000 test 1.000
5. Create an FEN from the board
6. Start up Vice, send him the fen, give him 30s to find the best move (hah!)
7. Get this best move and tweet a reply.
The whole bot is actually four separate processes, the information stored in a MongoDB database. This was mainly for redundancy.
Process 1 - Periodically checks for tweets online using the Twitter Api, saves the imageUrl, the tweet_id and the username of the author (you need this to reply) into the database
Process 2 - Periodically checks for new tweets in the DB and tries to create an FEN from the image. If it manages, it saves this in the DB, otherwise flags the tweet as failed and I can take a look.
Process 3 - Periodically checks for new FEN in the DB and gets a move from Vice, saving it in the DB
Process - Periodically checks for new bestmoves and tried to reply to the original tweet.
Oh, all of the above is Python (apart from Vice, which is some beginner level C)
Anyway, maybe it's interesting to someone. I had a good time with this
Richard
.