CCLS: New tournament (league system) - Introduction
Posted: Fri Jan 05, 2018 6:39 pm
Hi guys, I'm Carlos from Argentina and I've been testing chess engines for a long time, but always in private fashion. This is about to change now and I will share the results with you all. After doing a couple of test for a while I've come up with a league system that looks reasonable (at least to me), with promotions and demotions, longer time controls and fewer engines as the leagues get "better", etc.
CCLS means "Carlos Chess League System", but you can also make a case for "Computer Chess League System" or "Carlos Canavessi League System" or whatever.
Season 1 has already started on December 18 and I'm using it as a real test to see how well or how bad things work out. It is a short season with 3 leagues and a superfinal with the top 3 engines (a total of 66 engines). From Season 2 and beyond there will be 4 leagues + the superfinal (with a total of 122 engines).
The idea is to test as many engines as possible, regardless of their strenght, giving priority to the actively developed ones. Every season (except the shortened 1st one) the engines finishing in the bottom 16 places will be replaced with different ones the following season, but that does not mean they couldn't make a return later on.
The league system is structured as follows:
Entry league (skipped in season 1): 64 engines, bottom 16 are replaced with new engines the following season, top 8 are promoted to the Junior league. Time control is 1 minutes + 1 second and matches will be played in 2 x round robin format changing colors every round. A total of 4032 matches will be played, and the whole run will take aprox. 13 days to complete (estimation).
Junior league: 32 engines, bottom 8 are demoted to the Entry league, top 4 are promoted to the Super league. Time control is 2 minutes + 1 second and matches will be played in 4 x round robin format changing colors every round. A total of 1984 matches will be played, and the whole run will take aprox. 9 days to complete (took 9,5 days for season 1, already finished).
Super league: 24 engines, bottom 4 are demoted, top 2 are promoted to the Elite league. Time control is 3 minutes + 2 seconds and matches will be played in 6 x round robin format changing colors every round. A total of 1656 matches will be played, and the whole run will take aprox. 13 days to complete (more than 60% played so far and it's right there with the estimation).
Elite league: 16 engines, bottom 2 are demoted to the Super league, top 3 play the Superfinal. Time control is 5 minutes + 5 seconds and matches will be played in 8 x round robin format changing colors every round. A total of 960 matches will be played, and the whole run will take aprox. 15 days to complete (estimation).
Superfinal: 3 engines finishing on top of the Elite league. Time control will be 15 minutes + 15 seconds and matches will be played in 20 x round robin format changing colors every round. A total of 60 matches will be played, and the whole run will take aprox. 7 days to complete (estimation).
General rules:
Only 1 thread for each engine
Ponder ON
Opening book is Libra8 from the Numpty engine author (click here for book info), though that may change in future seasons
4GB Hash or biggest size the engine can handle if less than 4GB
Using Arena GUI on Windows 10, Ryzen 1700 processor with 32 GB total Ram
I'm still deciding what to do with the commercial engines. I have Shredder 13, Fritz 16 and Hiarcs 14 but no Komodo or Houdini; so I might have to use Houdini 1.5a and Komodo 9.02 which are free. After writing this post here I'm gonna email Mark and Robert to see if they are willing to provide the latest public releases of their engines so I can test them. If not, free versions for all, which I know it's gonna be disappointing.
Also, I'll contact Ginkgo author and see if he wants to participate from Season 2 (will have to start from the entry league, but it will be easy to promote fast to the elite league).
I don't want to test "dev" versions of engines, as that would complicate things for me. Only public releases at least for now.
In the future I would love to broadcast the matches for everyone to watch, but I'm still testing different options on how to do that. I use Arena and that limits things a lot.
So with that said, I'm gonna go ahead and make another thread with the results that I have so far (Season 1 Junior league 100% complete, Super league still going).
Ideas and suggestions always welcome.
CCLS means "Carlos Chess League System", but you can also make a case for "Computer Chess League System" or "Carlos Canavessi League System" or whatever.
Season 1 has already started on December 18 and I'm using it as a real test to see how well or how bad things work out. It is a short season with 3 leagues and a superfinal with the top 3 engines (a total of 66 engines). From Season 2 and beyond there will be 4 leagues + the superfinal (with a total of 122 engines).
The idea is to test as many engines as possible, regardless of their strenght, giving priority to the actively developed ones. Every season (except the shortened 1st one) the engines finishing in the bottom 16 places will be replaced with different ones the following season, but that does not mean they couldn't make a return later on.
The league system is structured as follows:
Entry league (skipped in season 1): 64 engines, bottom 16 are replaced with new engines the following season, top 8 are promoted to the Junior league. Time control is 1 minutes + 1 second and matches will be played in 2 x round robin format changing colors every round. A total of 4032 matches will be played, and the whole run will take aprox. 13 days to complete (estimation).
Junior league: 32 engines, bottom 8 are demoted to the Entry league, top 4 are promoted to the Super league. Time control is 2 minutes + 1 second and matches will be played in 4 x round robin format changing colors every round. A total of 1984 matches will be played, and the whole run will take aprox. 9 days to complete (took 9,5 days for season 1, already finished).
Super league: 24 engines, bottom 4 are demoted, top 2 are promoted to the Elite league. Time control is 3 minutes + 2 seconds and matches will be played in 6 x round robin format changing colors every round. A total of 1656 matches will be played, and the whole run will take aprox. 13 days to complete (more than 60% played so far and it's right there with the estimation).
Elite league: 16 engines, bottom 2 are demoted to the Super league, top 3 play the Superfinal. Time control is 5 minutes + 5 seconds and matches will be played in 8 x round robin format changing colors every round. A total of 960 matches will be played, and the whole run will take aprox. 15 days to complete (estimation).
Superfinal: 3 engines finishing on top of the Elite league. Time control will be 15 minutes + 15 seconds and matches will be played in 20 x round robin format changing colors every round. A total of 60 matches will be played, and the whole run will take aprox. 7 days to complete (estimation).
General rules:
Only 1 thread for each engine
Ponder ON
Opening book is Libra8 from the Numpty engine author (click here for book info), though that may change in future seasons
4GB Hash or biggest size the engine can handle if less than 4GB
Using Arena GUI on Windows 10, Ryzen 1700 processor with 32 GB total Ram
I'm still deciding what to do with the commercial engines. I have Shredder 13, Fritz 16 and Hiarcs 14 but no Komodo or Houdini; so I might have to use Houdini 1.5a and Komodo 9.02 which are free. After writing this post here I'm gonna email Mark and Robert to see if they are willing to provide the latest public releases of their engines so I can test them. If not, free versions for all, which I know it's gonna be disappointing.
Also, I'll contact Ginkgo author and see if he wants to participate from Season 2 (will have to start from the entry league, but it will be easy to promote fast to the elite league).
I don't want to test "dev" versions of engines, as that would complicate things for me. Only public releases at least for now.
In the future I would love to broadcast the matches for everyone to watch, but I'm still testing different options on how to do that. I use Arena and that limits things a lot.
So with that said, I'm gonna go ahead and make another thread with the results that I have so far (Season 1 Junior league 100% complete, Super league still going).
Ideas and suggestions always welcome.