Why successive versions of Stockfish converge towards Lc0?

[Laskos]

Several successive versions of Stockfish should behave randomly under the Sim test with regard to similarity to Lc0, if the positional strength is not a factor in determining the similarity. Sim 8,000+ positions are quiet many-option positions with no tactical complications. Here under Sim are the similarities between 2 Lc0 nets and several Stockfish versions. Well, from the results the first question arises why all mature net version of Lc0 for all runs look like clones (I got this result previously)? Move ordering argument of the engine was given, not the positional strength. Well, that doesn't stand when seeing the coalescence of the successive versions of Stockfishes towards Lc0. So, maybe positional strength of Lc0 (and a tremendous one) might be the reason after all?

Here are Sim matrix results:

Code: Select all

sim version 3

  Key:

  1) Lc0 SV_3010 (time: 100 ms  scale: 20)
  2) Lc0 T70 (time: 100 ms  scale: 1)
  3) Stockfish 10 (time: 100 ms  scale: 1)
  4) Stockfish 11 (time: 100 ms  scale: 1)
  5) Stockfish 6 (time: 100 ms  scale: 1)
  6) Stockfish 8 (time: 100 ms  scale: 1)
  7) Stockfish dev (time: 100 ms  scale: 1)

         1     2     3     4     5     6     7
  1.  ----- 67.27 52.15 52.82 48.94 50.72 53.88
  2.  67.27 ----- 52.42 52.51 49.78 51.07 52.62
  3.  52.15 52.42 ----- 64.79 56.43 61.67 63.57
  4.  52.82 52.51 64.79 ----- 55.52 60.31 66.04
  5.  48.94 49.78 56.43 55.52 ----- 60.96 54.79
  6.  50.72 51.07 61.67 60.31 60.96 ----- 59.03
  7.  53.88 52.62 63.57 66.04 54.79 59.03 -----

If we eliminate the relationships between different versions of Stockfishes, the matrix will look like that:

Code: Select all

         1     2     3     4     5     6     7 
  1.  99.99 67.27 52.15 52.82 48.94 50.72 53.88
  2.  67.27 99.99 52.42 52.51 49.78 51.07 52.62
  3.  52.15 52.42 99.99 00.00 00.00 00.00 00.00
  4.  52.82 52.51 00.00 99.99 00.00 00.00 00.00
  5.  48.94 49.78 00.00 00.00 99.99 00.00 00.00
  6.  50.72 51.07 00.00 00.00 00.00 99.99 00.00
  7.  53.88 52.62 00.00 00.00 00.00 00.00 99.99 

And the dendrogram looks like that:





My impression is that both coalescence of successive Stockfishes towards Lc0 nets and the "clone" nature of all mature Lc0 nets show an extreme positional strength of mature Lc0 nets. The engine used was v0.25.1 on RTX2070. Stockfishes are on 4 i7 cores.

[Laskos]

If including several other engines, the picture looks like that:

Code: Select all

sim version 3

  Key:

  1) Lc0 SV_3010 (time: 100 ms  scale: 20)
  2) Lc0 T70 (time: 100 ms  scale: 1)
  3) Stockfish 10 (time: 100 ms  scale: 1)
  4) Stockfish 11 (time: 100 ms  scale: 1)
  5) Stockfish 6 (time: 100 ms  scale: 1)
  6) Stockfish 8 (time: 100 ms  scale: 1)
  7) Stockfish dev (time: 100 ms  scale: 1)
  8) Andscacs 0.95 (time: 100 ms  scale: 1)
  9) Komodo 14 (time: 100 ms  scale: 1)
 10) Ethereal 12.25 (time: 100 ms  scale: 1)

         1     2     3     4     5     6     7     8     9    10
  1.  ----- 67.27 52.15 52.82 48.94 50.72 53.88 49.54 49.90 49.07
  2.  67.27 ----- 52.62 52.61 49.78 51.07 52.62 49.73 50.44 50.30
  3.  52.15 52.62 ----- 64.79 56.43 61.67 63.57 56.05 55.83 56.25
  4.  52.82 52.61 64.79 ----- 55.52 60.31 66.04 55.55 55.90 54.95
  5.  48.94 49.78 56.43 55.52 ----- 60.96 54.79 56.02 54.90 54.05
  6.  50.72 51.07 61.67 60.31 60.96 ----- 59.03 58.55 56.19 55.45
  7.  53.88 52.62 63.57 66.04 54.79 59.03 ----- 54.88 54.60 53.96
  8.  49.54 49.73 56.05 55.55 56.02 58.55 54.88 ----- 53.91 53.44
  9.  49.90 50.44 55.83 55.90 54.90 56.19 54.60 53.91 ----- 53.86
 10.  49.07 50.30 56.25 54.95 54.05 55.45 53.96 53.44 53.86 -----

Aside all correlations between different A/B engines and taking only in consideration only the similarity to Lc0 engines




If the order can be taken as the positional strength, and this is some sort of ranking of positional strength of engines, is it plausible that Komodo 14 is positionally weaker than SF8? Does Stockfish generally has a tendency to be more similar to Lc0 independently of strength? Some parts of SF code was ported to Lc0 (move generator, move ordering etc.)?

[mjlef]

If including several other engines, the picture looks like that:

Code: Select all

sim version 3

  Key:

  1) Lc0 SV_3010 (time: 100 ms  scale: 20)
  2) Lc0 T70 (time: 100 ms  scale: 1)
  3) Stockfish 10 (time: 100 ms  scale: 1)
  4) Stockfish 11 (time: 100 ms  scale: 1)
  5) Stockfish 6 (time: 100 ms  scale: 1)
  6) Stockfish 8 (time: 100 ms  scale: 1)
  7) Stockfish dev (time: 100 ms  scale: 1)
  8) Andscacs 0.95 (time: 100 ms  scale: 1)
  9) Komodo 14 (time: 100 ms  scale: 1)
 10) Ethereal 12.25 (time: 100 ms  scale: 1)

         1     2     3     4     5     6     7     8     9    10
  1.  ----- 67.27 52.15 52.82 48.94 50.72 53.88 49.54 49.90 49.07
  2.  67.27 ----- 52.62 52.61 49.78 51.07 52.62 49.73 50.44 50.30
  3.  52.15 52.62 ----- 64.79 56.43 61.67 63.57 56.05 55.83 56.25
  4.  52.82 52.61 64.79 ----- 55.52 60.31 66.04 55.55 55.90 54.95
  5.  48.94 49.78 56.43 55.52 ----- 60.96 54.79 56.02 54.90 54.05
  6.  50.72 51.07 61.67 60.31 60.96 ----- 59.03 58.55 56.19 55.45
  7.  53.88 52.62 63.57 66.04 54.79 59.03 ----- 54.88 54.60 53.96
  8.  49.54 49.73 56.05 55.55 56.02 58.55 54.88 ----- 53.91 53.44
  9.  49.90 50.44 55.83 55.90 54.90 56.19 54.60 53.91 ----- 53.86
 10.  49.07 50.30 56.25 54.95 54.05 55.45 53.96 53.44 53.86 -----

Aside all correlations between different A/B engines and taking only in consideration only the similarity to Lc0 engines




If the order can be taken as the positional strength, and this is some sort of ranking of positional strength of engines, is it plausible that Komodo 14 is positionally weaker than SF8? Does Stockfish generally has a tendency to be more similar to Lc0 independently of strength? Some parts of SF code was ported to Lc0 (move generator, move ordering etc.)?

I thought lc0 code was cleaned up and efficient code removed.
Also, why was Stockfish 9 excluded?

[Raphexon]

Because a lot of eval patches are written with (losses against) Leela in mind.

[smatovic]

Because a lot of eval patches are written with (losses against) Leela in mind.

Yea, evolution and co-evolution. I guess LC0 and SF will become a kind of binary-star.

--
Srdja

[Laskos]

Because a lot of eval patches are written with (losses against) Leela in mind.

Hmmm, interesting, didn't know. How do they do it, a systematic analysis of errors made when playing Lc0? Doesn't seem easy. I think it's easier to do the opposite, to fit Lc0 to SF, either by training on SF games or implementing some tactical abilities of SF.

[chrisw]

Several successive versions of Stockfish should behave randomly under the Sim test with regard to similarity to Lc0, if the positional strength is not a factor in determining the similarity. Sim 8,000+ positions are quiet many-option positions with no tactical complications. Here under Sim are the similarities between 2 Lc0 nets and several Stockfish versions. Well, from the results the first question arises why all mature net version of Lc0 for all runs look like clones (I got this result previously)? Move ordering argument of the engine was given, not the positional strength. Well, that doesn't stand when seeing the coalescence of the successive versions of Stockfishes towards Lc0. So, maybe positional strength of Lc0 (and a tremendous one) might be the reason after all?

Here are Sim matrix results:

Code: Select all

sim version 3

  Key:

  1) Lc0 SV_3010 (time: 100 ms  scale: 20)
  2) Lc0 T70 (time: 100 ms  scale: 1)
  3) Stockfish 10 (time: 100 ms  scale: 1)
  4) Stockfish 11 (time: 100 ms  scale: 1)
  5) Stockfish 6 (time: 100 ms  scale: 1)
  6) Stockfish 8 (time: 100 ms  scale: 1)
  7) Stockfish dev (time: 100 ms  scale: 1)

         1     2     3     4     5     6     7
  1.  ----- 67.27 52.15 52.82 48.94 50.72 53.88
  2.  67.27 ----- 52.42 52.51 49.78 51.07 52.62
  3.  52.15 52.42 ----- 64.79 56.43 61.67 63.57
  4.  52.82 52.51 64.79 ----- 55.52 60.31 66.04
  5.  48.94 49.78 56.43 55.52 ----- 60.96 54.79
  6.  50.72 51.07 61.67 60.31 60.96 ----- 59.03
  7.  53.88 52.62 63.57 66.04 54.79 59.03 -----

If we eliminate the relationships between different versions of Stockfishes, the matrix will look like that:

Code: Select all

         1     2     3     4     5     6     7 
  1.  99.99 67.27 52.15 52.82 48.94 50.72 53.88
  2.  67.27 99.99 52.42 52.51 49.78 51.07 52.62
  3.  52.15 52.42 99.99 00.00 00.00 00.00 00.00
  4.  52.82 52.51 00.00 99.99 00.00 00.00 00.00
  5.  48.94 49.78 00.00 00.00 99.99 00.00 00.00
  6.  50.72 51.07 00.00 00.00 00.00 99.99 00.00
  7.  53.88 52.62 00.00 00.00 00.00 00.00 99.99 

And the dendrogram looks like that:





My impression is that both coalescence of successive Stockfishes towards Lc0 nets and the "clone" nature of all mature Lc0 nets show an extreme positional strength of mature Lc0 nets. The engine used was v0.25.1 on RTX2070. Stockfishes are on 4 i7 cores.

Because.
There are two adversarial networks fighting each other. One is composed of programmers and testers who, if they had a deep knowledge of chess, can’t really apply it, because their system doesn’t really allow for input of human chess knowledge, rather, it allows for input of engineering knowledge. The other is composed of programmers and testers who can usefully try and work out the chess reasons their program have not understood and can often work out a way to get that knowledge into their system (and proof it by testing). It’s a struggle therefore between human reasoning and philosophising (about chess) and somewhat brute force engineering. Curiously enough the brute force engineering makes a (possibly) more human/like player, while the reasoning/philosophy ends up more machine like.

[Dann Corbit]

The most interesting thing is that the development SF is closer to LC0 than it is to SF 11.
So SF dev is more like LC0 than it is like SF 11.
What?!

[Alayan]

Because a lot of eval patches are written with (losses against) Leela in mind.

This effect is EXTREMELY overrated.

At least 95% of elo gainers are unrelated to this.

The most interesting thing is that the development SF is closer to LC0 than it is to SF 11.
So SF dev is more like LC0 than it is like SF 11.
What?!

Laskos' dendeogram only looks at the similarity with the Leela nets, discarding similarity between the Stockfishes.

[Laskos]

The most interesting thing is that the development SF is closer to LC0 than it is to SF 11.
So SF dev is more like LC0 than it is like SF 11.
What?!

No, it's an artifact of the dendrogram. It exaggerates these things, just for illustration. Look at the complete similarity matrix. Then I discarded the similarities between Stockfishes in the second matrix. Maybe comparing several numbers was clearer than the dendrogram, but people like pictures. The fact is SF dev is significantly closer to Lc0 than SF 11 is to Lc0, but not by so much. And SF dev is still very close to SF 11.