SIM new version

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
User avatar
Rebel
Posts: 4472
Joined: Thu Aug 18, 2011 10:04 am

SIM new version

Post by Rebel » Mon Apr 01, 2019 4:53 am

Recently (after 5 years) I tested the current top-engines on similarity (once again) because I wanted to know 2 things:

1. if the crazy days of the Fruit cloning (2005-2010) and Rybka3 cloning (2010-2015) were over and how the situation anno 2019 is now that Stockfish is the ideal target. And so I ran SIM3 from Don Daily checking the CCRL TOP-10. And the results were amazingly good, as it seems computer chess healed itself.

#2. I also wanted to know something else, since the immense elo progress that have been made the last 5 years, is it not likely that the top-engines increase in similarity? And if that is the case then: how much. In other words, a test if SIM3 is still a reliable tool. And looking at the situation of 2019 the numbers are looking good, the top-engines nicely curdle between 45-50%. So far so good.

And then something somewhat strange happened, as last 3 engines I added the latest versions of Fire, Booot and Fizbo to the test and Fizbo showed an incredible low similarity of 24% average. That's not necessarily wrong but a bit strange it is for 3300+ elo rated engine.

And I decided to have a closer look, long story short, I found a small (but mean) bug in SIM3. In Fail-high situations it not always recorded the right move. I fixed the bug and ran the test again and the result is remarkable, softly speaking that is, view the comparison between SIM3 and SIM4. Suddenly according to the legenda Xiphos with 64% is a clone / derivative of Stockfish, even Komodo with 61% is in the red zone. High similarity between the top-engines. And of course they are not clones / derivatives of each other. Several are open source, easy to compare.

As next step I ran several engines from the Rybka3 cloning (2010-2015) period to see how they would perform with the corrected version. And suddenly they are no clones / derivatives any longer! View the comparison, similarity is about in the same range as today.

This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??

Page including links and download - http://rebel13.nl/home/sim.html
Everybody is unique, except me.

Ferdy
Posts: 3938
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: SIM new version

Post by Ferdy » Mon Apr 01, 2019 5:32 am

Interesting discovery Ed. Lets try it.

carldaman
Posts: 1665
Joined: Sat Jun 02, 2012 12:13 am

Re: SIM new version

Post by carldaman » Mon Apr 01, 2019 6:39 am

Hmm, and the calendar says what? :o :twisted:

Sergio Martinez
Posts: 1709
Joined: Fri Oct 05, 2012 12:35 pm
Location: Spain
Contact:

Re: SIM new version

Post by Sergio Martinez » Mon Apr 01, 2019 7:08 am

Very very interesting!

Thanks Ed.
Member of the CCRL Group. Write me if you want I test your engine.

Ferdy
Posts: 3938
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: SIM new version

Post by Ferdy » Mon Apr 01, 2019 10:56 am

Rebel wrote:
Mon Apr 01, 2019 4:53 am
This begs the question if programmers have been wrongly accused of cloning in the past because of a bug in SIM3 ??

Page including links and download - http://rebel13.nl/home/sim.html
I tried to log sim4.1 output here is the difference to my analyzer.
1. sim4.1 does not send ucinewgame command or restart the engine. In my analyzer engine is restarted after every position has been analyzed.

2. sim4.1 sends go depth <depth> and stop commands, this has a drawback, it might happen that after sending stop, some engines may extend more time than others and you save the bestmove sent by engine. In my analyzer I use go movetime <timems> then save the bestmove sent by engine. This is more reliable as I am not interrupting the search of the engine. But this would only work for those engines that complies the movetime command which the majority of strong engines implemented correctly. go movetime does not work for Fizbo, so I use go infinite then send stop if analysis time is reached. The way I collect the bestmove is thru the pv sent by the engine. That pv line should not have upperbound and lowerbound text in it. After the stop command the engine would send bestmove bm, but I will not use that bm, I use the first move in the pv at a time that I sent the stop command, to comply with the analysis time specified by the user.

Perhaps these differences may affect the overall results of the tests.

Using sim4.1 here are my results.

Code: Select all

sim version 4.1

  Key:

  1) Booot 6.3.1 (time: 200 ms  scale: 1.0)
  2) Deuterium v2019.2.37.37 (time: 200 ms  scale: 1.0)
  3) Ethereal 11.25 (time: 200 ms  scale: 1.0)
  4) Fire 7.1 (time: 200 ms  scale: 1.0)
  5) Laser 1.7 (time: 200 ms  scale: 1.0)
  6) Stockfish 10 (time: 20 ms  scale: 1.0)
  7) Stockfish 10 (time: 200 ms  scale: 1.0)
  8) Stockfish 10 r1 (time: 200 ms  scale: 1.0)
  9) Xiphos 0.5 (time: 200 ms  scale: 1.0)

         1     2     3     4     5     6     7     8     9
  1.  ----- 36.87 36.39 35.86 36.00 37.16 35.08 34.69 35.52
  2.  36.87 ----- 48.62 46.26 48.48 47.52 47.95 47.40 48.36
  3.  36.39 48.62 ----- 47.67 50.96 48.08 51.66 51.76 52.82
  4.  35.86 46.26 47.67 ----- 47.21 46.53 49.20 48.85 47.37
  5.  36.00 48.48 50.96 47.21 ----- 47.75 51.03 50.39 50.06
  6.  37.16 47.52 48.08 46.53 47.75 ----- 53.92 53.48 48.28
  7.  35.08 47.95 51.66 49.20 51.03 53.92 ----- 64.53 52.88
  8.  34.69 47.40 51.76 48.85 50.39 53.48 64.53 ----- 52.71
  9.  35.52 48.36 52.82 47.37 50.06 48.28 52.88 52.71 -----

I ran Stockfish 10 twice at 200ms to establish a base line. This should be the highest similarity at 64.53.
1. Stockfish 10
2. Stockfish 10 r1

Result:

Code: Select all

sim version 4.1
------ Stockfish 10 (time: 200 ms  scale: 1.0) ------
 64.53  Stockfish 10 r1 (time: 200 ms  scale: 1.0)
 53.92  Stockfish 10 (time: 20 ms  scale: 1.0)
 52.88  Xiphos 0.5 (time: 200 ms  scale: 1.0)
 51.66  Ethereal 11.25 (time: 200 ms  scale: 1.0)
 51.03  Laser 1.7 (time: 200 ms  scale: 1.0)
 49.20  Fire 7.1 (time: 200 ms  scale: 1.0)
 47.95  Deuterium v2019.2.37.37 (time: 200 ms  scale: 1.0)
 35.08  Booot 6.3.1 (time: 200 ms  scale: 1.0)
Now with this sim version we get the highest similarity of 64.53 as 100% clone, on my computer with that given analysis time and 1 thread per engine with 128 MB hash.

User avatar
cdani
Posts: 2103
Joined: Sat Jan 18, 2014 9:24 am
Location: Andorra
Contact:

Re: SIM new version

Post by cdani » Mon Apr 01, 2019 9:34 pm

Rebel wrote:
Mon Apr 01, 2019 4:53 am
... Several are open source, easy to compare...
Even if the engine sources are different, they use more and more the same known to work techniques. I expect them to be more and more equal in the best lines that they find. So they are not clones or derivatives per se, but are partial derivatives in term of the ideas they use. Andscacs also, of course, even if I try to do everything in my own way.

Sergio Martinez
Posts: 1709
Joined: Fri Oct 05, 2012 12:35 pm
Location: Spain
Contact:

Re: SIM new version

Post by Sergio Martinez » Tue Apr 02, 2019 7:03 am

Hi Ed

Is there any possibility to edit the positions of the tool and add or remove positions?

Thank you
Member of the CCRL Group. Write me if you want I test your engine.

User avatar
Rebel
Posts: 4472
Joined: Thu Aug 18, 2011 10:04 am

Re: SIM new version

Post by Rebel » Tue Apr 02, 2019 9:32 am

Sergio Martinez wrote:
Tue Apr 02, 2019 7:03 am
Hi Ed

Is there any possibility to edit the positions of the tool and add or remove positions?

Thank you
Yes, but you need a special version of SIM3 including the source code. I am not allowed to share it.
Everybody is unique, except me.

Sergio Martinez
Posts: 1709
Joined: Fri Oct 05, 2012 12:35 pm
Location: Spain
Contact:

Re: SIM new version

Post by Sergio Martinez » Tue Apr 02, 2019 10:20 am

Rebel wrote:
Tue Apr 02, 2019 9:32 am
Sergio Martinez wrote:
Tue Apr 02, 2019 7:03 am
Hi Ed

Is there any possibility to edit the positions of the tool and add or remove positions?

Thank you
Yes, but you need a special version of SIM3 including the source code. I am not allowed to share it.
Ok.

Thanks.
Member of the CCRL Group. Write me if you want I test your engine.

Post Reply