SIM new version
Posted: Mon Apr 01, 2019 6:53 am
Recently (after 5 years) I tested the current top-engines on similarity (once again) because I wanted to know 2 things:
1. if the crazy days of the Fruit cloning (2005-2010) and Rybka3 cloning (2010-2015) were over and how the situation anno 2019 is now that Stockfish is the ideal target. And so I ran SIM3 from Don Daily checking the CCRL TOP-10. And the results were amazingly good, as it seems computer chess healed itself.
#2. I also wanted to know something else, since the immense elo progress that have been made the last 5 years, is it not likely that the top-engines increase in similarity? And if that is the case then: how much. In other words, a test if SIM3 is still a reliable tool. And looking at the situation of 2019 the numbers are looking good, the top-engines nicely curdle between 45-50%. So far so good.
And then something somewhat strange happened, as last 3 engines I added the latest versions of Fire, Booot and Fizbo to the test and Fizbo showed an incredible low similarity of 24% average. That's not necessarily wrong but a bit strange it is for 3300+ elo rated engine.
And I decided to have a closer look, long story short, I found a small (but mean) bug in SIM3. In Fail-high situations it not always recorded the right move. I fixed the bug and ran the test again and the result is remarkable, softly speaking that is, view the comparison between SIM3 and SIM4. Suddenly according to the legenda Xiphos with 64% is a clone / derivative of Stockfish, even Komodo with 61% is in the red zone. High similarity between the top-engines. And of course they are not clones / derivatives of each other. Several are open source, easy to compare.
As next step I ran several engines from the Rybka3 cloning (2010-2015) period to see how they would perform with the corrected version. And suddenly they are no clones / derivatives any longer! View the comparison, similarity is about in the same range as today.
This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??
Page including links and download - http://rebel13.nl/home/sim.html
1. if the crazy days of the Fruit cloning (2005-2010) and Rybka3 cloning (2010-2015) were over and how the situation anno 2019 is now that Stockfish is the ideal target. And so I ran SIM3 from Don Daily checking the CCRL TOP-10. And the results were amazingly good, as it seems computer chess healed itself.
#2. I also wanted to know something else, since the immense elo progress that have been made the last 5 years, is it not likely that the top-engines increase in similarity? And if that is the case then: how much. In other words, a test if SIM3 is still a reliable tool. And looking at the situation of 2019 the numbers are looking good, the top-engines nicely curdle between 45-50%. So far so good.
And then something somewhat strange happened, as last 3 engines I added the latest versions of Fire, Booot and Fizbo to the test and Fizbo showed an incredible low similarity of 24% average. That's not necessarily wrong but a bit strange it is for 3300+ elo rated engine.
And I decided to have a closer look, long story short, I found a small (but mean) bug in SIM3. In Fail-high situations it not always recorded the right move. I fixed the bug and ran the test again and the result is remarkable, softly speaking that is, view the comparison between SIM3 and SIM4. Suddenly according to the legenda Xiphos with 64% is a clone / derivative of Stockfish, even Komodo with 61% is in the red zone. High similarity between the top-engines. And of course they are not clones / derivatives of each other. Several are open source, easy to compare.
As next step I ran several engines from the Rybka3 cloning (2010-2015) period to see how they would perform with the corrected version. And suddenly they are no clones / derivatives any longer! View the comparison, similarity is about in the same range as today.
This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??
Page including links and download - http://rebel13.nl/home/sim.html