SIMEX 2.1

[Rebel]

After inspection I made some changes to the "ccrl.htm", removed the following engines:

Laser - doesn't support the UCI depth command.
Fizbo - doesn't support the UCI depth command.
Protector - doesn't support the UCI depth command.

Unrealistic low numbers
Smarthink - can not check it since the data came from an external source.
Pedone 1.8 - looking at 1.7 and 1.8 the number is too weird to make sense for similarity testing.
Senpai 2 - same as Pedone 1.8

Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;

Add (the last version) of ProDeo.

The new list - http://rebel13.nl/html/ccrl.html

[Ovyron]

To me, I see people trying to draw a line. But I do not clearly understand exactly what the line is supposed to mean.

The line can't be drawn.

Vasik Rajlich once said around the time Rybka 4 was released, that he could trivially produce some 20 engines that played near the strength of Rybka, but that played completely different from Rybka. He even added settings to the engine so people could change its material evaluation and do it themselves, and people like Mindbreaker managed to create settings stronger than default.

What this means is that anybody capable of improving some source code by someone else (like Robert Houdart improving Ippolit to Houdini 6 levels) is also capable of producing a stronger engine and then make changes so that its similarity with the source engine is arbitrarily low.

If no matter where you draw the line, someone can close Stockfish's source and sell it while fooling everyone because they can produce an engine that plays nothing like Stockfish, drawing the line is useless.

[chrisw]

To me, I see people trying to draw a line. But I do not clearly understand exactly what the line is supposed to mean.

The line can't be drawn.

Vasik Rajlich once said around the time Rybka 4 was released, that he could trivially produce some 20 engines that played near the strength of Rybka, but that played completely different from Rybka. He even added settings to the engine so people could change its material evaluation and do it themselves, and people like Mindbreaker managed to create settings stronger than default.

What this means is that anybody capable of improving some source code by someone else (like Robert Houdart improving Ippolit to Houdini 6 levels) is also capable of producing a stronger engine and then make changes so that its similarity with the source engine is arbitrarily low.

Well, let’s wait and see what the data has to say about that.

If no matter where you draw the line, someone can close Stockfish's source and sell it while fooling everyone because they can produce an engine that plays nothing like Stockfish, drawing the line is useless.

[Ferdy]

Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;

The ce -32000 and acd 0 mean that the engine did not send score and depth info. Default score and depth are used instead.

[Rebel]

Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;

The ce -32000 and acd 0 mean that the engine did not send score and depth info. Default score and depth are used instead.

Yep, I noticed... those are engines that hide their mainlines for the first x plies. There are also a lot of engines that don't support the UCI depth command. I will make a list of those in the final report.

[chrisw]

After inspection I made some changes to the "ccrl.htm", removed the following engines:

Laser - doesn't support the UCI depth command.
Fizbo - doesn't support the UCI depth command.
Protector - doesn't support the UCI depth command.

Unrealistic low numbers
Smarthink - can not check it since the data came from an external source.
Pedone 1.8 - looking at 1.7 and 1.8 the number is too weird to make sense for similarity testing.
Senpai 2 - same as Pedone 1.8

Booot and RofChade. Both produce indentical weird EPD output.

Code: Select all

Booot
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm Nbd2; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bd4; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Rxd4; ce -32000; acd 0;

Code: Select all

RofChade
3q1rk1/3n1ppp/1p1p1b2/1R1P4/8/r1pNP1P1/P1Q2PBP/5RK1 w - - bm Rb3; ce -32000; acd 0;
r2q1rk1/pppb1pbp/2np1np1/4p3/3PP3/2P2NP1/PP3PBP/RNBQR1K1 w - - bm d5; ce -32000; acd 0;
2r1k2r/4bp2/R6p/8/4n3/4BN2/1PP5/1K4R1 w k - bm Bxh6; ce -32000; acd 0;
8/6pk/p1q2pbp/1p6/PPnRrB2/1BQ4P/5PPK/8 b - - bm Re2; ce -32000; acd 0;

Add (the last version) of ProDeo.

The new list - http://rebel13.nl/html/ccrl.html

Quick piece of data mining on that table.
Lowest similarity is Hiarcs1 vs Merv at 29%
if you run a search on that page for '29.', '30.', '31.', '32.', '33.', '34.' you'll note the entire set is owned by Hiarcs14. No other engine gets that 'different' to any other engine until 35% similarity, when quite a few other engines kick in.
Meanwhile, the highest similarity Hiarcs 14 scores is only 37%.

This suggests to me that Hiarcs14 is 'different' (not-similar) in a different way to all the others.

From a NN Simex chart I saw recently, the 'difference' behaviour is similar, all 'NN-similarities' with AB engines are in the range 28-34%, quite similar profile to Hiarcs14.

Looking at other engines in general, it's rare for the minimum similarity to be less than 40%, and of course, the maximum similarity is stratospheric in some case.

Isn't that interesting?!

[Rebel]

I think Hiarcs, Gandalf, ProDeo regarding the evaluation belong to the old-class generation before the open source revolution of Fruit, Strelka, Robbolito and currently Stockfish began and were not influenced, hence low similarity.

I added a few oldies, Ruffian, Hiarcs 13 and Shredder 8.

Seems to confirm my line of thought.

http://rebel13.nl/html/ccrl.html

On NN engines you may notice that when running on time instead of depth=1 similarity normalizes.

http://rebel13.nl/html/lc0-1000ms.html