Similarity tester - 2nd generation - BETA

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
User avatar
Rebel
Posts: 4547
Joined: Thu Aug 18, 2011 10:04 am

Similarity tester - 2nd generation - BETA

Post by Rebel » Fri Aug 09, 2019 9:32 pm

I am working on the documentation of SIMEX 2.0 which is meant to be the successor of Don Dailey's famous SIM03 Similarity Tester.

SIMEX works the same way but is more user-friendly, has more features but the main advantage is that you are no longer limited to the build-in 8238 positions of SIM03 but can create your own positions using EPD, in the download 7 EPD sets are provided for demonstration purposes. SIMEX uses MEA from Ferdinand Mosca as a base.

http://rebel13.nl/misc/simex.html

The documentation is not even half-way and by this thread I take the opportunity to receive comments, ideas before an official release. The page will be frequently updated the coming days.
Everybody is unique, except me.

Ferdy
Posts: 4051
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Sat Aug 10, 2019 3:57 pm

Is simex2 able to generate a matrix similar to sim03?
In sim03 I just use the -m option to generate the following.
Rename midgame.data to similarity.data then run

Code: Select all

sim03w32.exe -m >matrix.txt
matrix.txt

Code: Select all

sim version 3

  Key:

  1) Andscacs_0.93
  2) brainfish_2018
  3) D2019.1.36.50
  4) D2019.2.37.52
  5) Ethereal_11.25
  6) Laser_1.7
  7) Pedone_1.9
  8) Rodent_0.277
  9) RofChade_2.1
 10) RubiChess_1.5
 11) Schooner_2.0.34
 12) Stockfish_10
 13) Texel_1.07

         1     2     3     4     5     6     7     8     9    10    11    12    13
  1.  ----- 54.19 51.77 51.54 54.26 53.53 48.88 53.34 52.86 50.46 54.56 53.27 52.23
  2.  54.19 ----- 52.68 52.99 56.11 56.66 48.83 51.97 55.88 51.49 55.37 63.38 51.65
  3.  51.77 52.68 ----- 65.94 53.69 54.66 52.03 53.96 54.92 51.41 53.44 52.56 51.95
  4.  51.54 52.99 65.94 ----- 53.73 54.54 51.64 53.86 55.06 51.70 53.06 52.69 51.75
  5.  54.26 56.11 53.69 53.73 ----- 57.94 50.33 53.59 58.35 54.09 55.68 56.66 54.05
  6.  53.53 56.66 54.66 54.54 57.94 ----- 51.63 54.69 56.43 53.45 55.79 56.34 54.01
  7.  48.88 48.83 52.03 51.64 50.33 51.63 ----- 51.64 50.38 49.36 51.64 48.96 50.59
  8.  53.34 51.97 53.96 53.86 53.59 54.69 51.64 ----- 53.97 52.62 54.29 51.83 55.05
  9.  52.86 55.88 54.92 55.06 58.35 56.43 50.38 53.97 ----- 54.19 55.20 55.81 54.19
 10.  50.46 51.49 51.41 51.70 54.09 53.45 49.36 52.62 54.19 ----- 53.46 51.08 53.11
 11.  54.56 55.37 53.44 53.06 55.68 55.79 51.64 54.29 55.20 53.46 ----- 54.49 52.82
 12.  53.27 63.38 52.56 52.69 56.66 56.34 48.96 51.83 55.81 51.08 54.49 ----- 51.20
 13.  52.23 51.65 51.95 51.75 54.05 54.01 50.59 55.05 54.19 53.11 52.82 51.20 -----

I tried the midgame at 100ms, then use sim03 to generate the matrix. I need the matrix output to generate dendrogram.

Image


And this is from your html output.

Image

Deuteriums are suspects the same is true for Stockfish/Brainfish :). The dendrogram agrees with it.

Will release the sim2dendro.py later.

User avatar
Rebel
Posts: 4547
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Sat Aug 10, 2019 6:25 pm

I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.
Everybody is unique, except me.

User avatar
Guenther
Posts: 2960
Joined: Wed Oct 01, 2008 4:33 am
Location: Regensburg, Germany
Full name: Guenther Simon
Contact:

Re: Similarity tester - 2nd generation - BETA

Post by Guenther » Sat Aug 10, 2019 6:28 pm

Rebel wrote:
Fri Aug 09, 2019 9:32 pm
I am working on the documentation of SIMEX 2.0 which is meant to be the successor of Don Dailey's famous SIM03 Similarity Tester.

SIMEX works the same way but is more user-friendly, has more features but the main advantage is that you are no longer limited to the build-in 8238 positions of SIM03 but can create your own positions using EPD, in the download 7 EPD sets are provided for demonstration purposes. SIMEX uses MEA from Ferdinand Mosca as a base.

http://rebel13.nl/misc/simex.html

The documentation is not even half-way and by this thread I take the opportunity to receive comments, ideas before an official release. The page will be frequently updated the coming days.
Thanks for this Ed and Ferdy (MEA). Hopefully I can fiddle around with this after summer with own sets of epds.
Current foe list count : [91]
http://rwbc-chess.de/chronology.htm

Ferdy
Posts: 4051
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Sun Aug 11, 2019 1:25 am

Rebel wrote:
Sat Aug 10, 2019 6:25 pm
I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.
I think it is indeed better for specific input for sim2dendro.
Better is just a csv type formatting with or without a header. Example

engine1 name,-----,51.42,49.40, and so on
engine2 name,51.42,-----,60.66, and so on
and so on

You can just add it at the end of the report.txt so that they are just in one file. That should not be difficult to parse.

The repo is at https://github.com/fsmosca/Similarity-Dendrogram but it can only read sim3 matrix output currently.

If possible perhaps you can add in the report.txt the info on test epd and time used.

User avatar
Ovyron
Posts: 2348
Joined: Tue Jul 03, 2007 2:30 am

Re: Similarity tester - 2nd generation - BETA

Post by Ovyron » Sun Aug 11, 2019 5:28 am

Is it possible to "reverse engineer" this to make engine personalities that increase their similarity to other engines or to human players? Say, Stockfish already plays kind of similar to Laser, to maybe it'd be possible to make changes to the code that increase this similarity, so we'd get some "Laser Personality", and it'd be possible that it'd play like Laser but with a higher playing strength.
Make someone happy today.

User avatar
Rebel
Posts: 4547
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Sun Aug 11, 2019 7:16 am

Ferdy wrote:
Sun Aug 11, 2019 1:25 am
Rebel wrote:
Sat Aug 10, 2019 6:25 pm
I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.
I think it is indeed better for specific input for sim2dendro.
Better is just a csv type formatting with or without a header. Example

engine1 name,-----,51.42,49.40, and so on
engine2 name,51.42,-----,60.66, and so on
and so on

You can just add it at the end of the report.txt so that they are just in one file. That should not be difficult to parse.

The repo is at https://github.com/fsmosca/Similarity-Dendrogram but it can only read sim3 matrix output currently.

If possible perhaps you can add in the report.txt the info on test epd and time used.
Made the changes, while creating the HTML a file "dendogram.csv" is also made. Contents:

Code: Select all

epd\simex.epd,8238,100
Andscacs 0.93,-----,51.67,50.21,48.83,50.21,48.28,50.48,44.92,49.69,50.87
asmFish 2017,51.67,-----,60.21,50.07,53.00,49.06,53.38,45.74,52.27,60.65
brainfish 2018,50.21,60.21,-----,49.54,52.43,48.21,51.33,45.49,51.57,62.40
Equinox 3.30,48.83,50.07,49.54,-----,49.94,51.91,50.33,47.01,50.68,49.97
Ethereal 11.25,50.21,53.00,52.43,49.94,-----,48.22,53.40,49.03,51.66,52.74
Hannibal 1.7,48.28,49.06,48.21,51.91,48.22,-----,49.43,43.84,49.12,48.51
Laser 1.7,50.48,53.38,51.33,50.33,53.40,49.43,-----,48.21,51.21,52.31
RubiChess 1.4,44.92,45.74,45.49,47.01,49.03,43.84,48.21,-----,47.08,44.69
Schooner 2.0.34,49.69,52.27,51.57,50.68,51.66,49.12,51.21,47.08,-----,50.53
Stockfish 10,50.87,60.65,62.40,49.97,52.74,48.51,52.31,44.69,50.53,-----
Header contains used EPD, number positions, movetime.

If that's workable for you re-download SIMEX.

http://rebel13.nl/dl/simex2.7z
Everybody is unique, except me.

User avatar
Rebel
Posts: 4547
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Sun Aug 11, 2019 8:31 am

Added 8 new engines to the 2500ms statistic.

http://rebel13.nl/html/simex_2500ms_18_engines.html

Don't know what to think about the increasing orange markers. One can say that's only logical, more time (in general) means better moves.
Everybody is unique, except me.

Ferdy
Posts: 4051
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Similarity tester - 2nd generation - BETA

Post by Ferdy » Sun Aug 11, 2019 11:56 am

Rebel wrote:
Sun Aug 11, 2019 7:16 am
Ferdy wrote:
Sun Aug 11, 2019 1:25 am
Rebel wrote:
Sat Aug 10, 2019 6:25 pm
I completed the documentation.

From the command line:

simex2 data\similarity.data >report.txt

In report.txt you get something like SIM3:

Code: Select all

Andsc asmFi brain Equin Ether Hanni Laser Stock 
----- 51.42 49.40 48.07 50.08 47.41 50.42 49.83 
51.42 ----- 60.66 49.74 53.11 48.75 53.26 61.04 
49.40 60.66 ----- 49.53 52.48 48.07 51.58 62.28 
48.07 49.74 49.53 ----- 49.74 51.29 50.19 49.71 
50.08 53.11 52.48 49.74 ----- 47.89 53.43 52.74 
47.41 48.75 48.07 51.29 47.89 ----- 49.23 48.22 
50.42 53.26 51.58 50.19 53.43 49.23 ----- 52.18 
49.83 61.04 62.28 49.71 52.74 48.22 52.18 ----- 
But maybe it's more easy when I create an output file as input for your sim2dendro.
I think it is indeed better for specific input for sim2dendro.
Better is just a csv type formatting with or without a header. Example

engine1 name,-----,51.42,49.40, and so on
engine2 name,51.42,-----,60.66, and so on
and so on

You can just add it at the end of the report.txt so that they are just in one file. That should not be difficult to parse.

The repo is at https://github.com/fsmosca/Similarity-Dendrogram but it can only read sim3 matrix output currently.

If possible perhaps you can add in the report.txt the info on test epd and time used.
Made the changes, while creating the HTML a file "dendogram.csv" is also made. Contents:

Code: Select all

epd\simex.epd,8238,100
Andscacs 0.93,-----,51.67,50.21,48.83,50.21,48.28,50.48,44.92,49.69,50.87
asmFish 2017,51.67,-----,60.21,50.07,53.00,49.06,53.38,45.74,52.27,60.65
brainfish 2018,50.21,60.21,-----,49.54,52.43,48.21,51.33,45.49,51.57,62.40
Equinox 3.30,48.83,50.07,49.54,-----,49.94,51.91,50.33,47.01,50.68,49.97
Ethereal 11.25,50.21,53.00,52.43,49.94,-----,48.22,53.40,49.03,51.66,52.74
Hannibal 1.7,48.28,49.06,48.21,51.91,48.22,-----,49.43,43.84,49.12,48.51
Laser 1.7,50.48,53.38,51.33,50.33,53.40,49.43,-----,48.21,51.21,52.31
RubiChess 1.4,44.92,45.74,45.49,47.01,49.03,43.84,48.21,-----,47.08,44.69
Schooner 2.0.34,49.69,52.27,51.57,50.68,51.66,49.12,51.21,47.08,-----,50.53
Stockfish 10,50.87,60.65,62.40,49.97,52.74,48.51,52.31,44.69,50.53,-----
Header contains used EPD, number positions, movetime.

If that's workable for you re-download SIMEX.

http://rebel13.nl/dl/simex2.7z
Downloaded that file an hour ago, no changes.

Code: Select all

Variety (forced nature of positions) overview

 1. 2862 (28.62%)
 2. 4468 (44.68%)
 3. 2669 (26.69%)

Positions 10000 Andsc Deute Xipho 
Andscacs 0.93   ----- 52.73 39.45 
Deuterium v2019.1.36.50                                          52.73 ----- 41.64 
Xiphos 0.5      39.45 41.64 ----- 

 1. Engine Andscacs 0.93 and Deuterium v2019.1.36.50 show a similarity of 52.73%
 2. Engine Deuterium v2019.1.36.50 and Xiphos 0.5 show a similarity of 41.64%
 3. Engine Andscacs 0.93 and Xiphos 0.5 show a similarity of 39.45%


Done...

User avatar
Rebel
Posts: 4547
Joined: Thu Aug 18, 2011 10:04 am

Re: Similarity tester - 2nd generation - BETA

Post by Rebel » Sun Aug 11, 2019 2:36 pm

Ferdy wrote:
Sun Aug 11, 2019 11:56 am
Downloaded that file an hour ago, no changes.
Check the file dendogram.csv, should be present.
Everybody is unique, except me.

Post Reply