Correlation test reliability concerning clone issue.

[Hood]

Hello,

I am wondering if it is trustable method.

Lets say we do have a position mate in #5,6,...,9 whatever and there are some ways to do that let say A, B, C, D.
The original engine selects path A.
The potential clone has a modification that selects path B.
In that case what is a correlation function value 0 ?

We can extend the example for the position where are some moves with the same evaluation value and the clone chooses other path then original.

I suppose that in that case correlation test will failed.
What are the opinions of experienced Posters and chessprogramers.

Regards
Chris

[Graham Banks]

Hello,

I am wondering if it is trustable method.

Lets say we do have a position mate in #5,6,...,9 whatever and there are some ways to do that let say A, B, C, D.
The original engine selects path A.
The potential clone has a modification that selects path B.
In that case what is a correlation function value 0 ?

We can extend the example for the position where are some moves with the same evaluation value and the clone chooses other path then original.

I suppose that in that case correlation test will failed.
What are the opinions of experienced Posters and chessprogramers.

Regards
Chris

I think that if such a test (using 1000 or more positions) can show an extreme similarity between all cases of already known clones, then there's no real reason to doubt its reliability.
Just my opinion, but I'm not a programmer.

[michiguel]

Hello,

I am wondering if it is trustable method.

Lets say we do have a position mate in #5,6,...,9 whatever and there are some ways to do that let say A, B, C, D.
The original engine selects path A.
The potential clone has a modification that selects path B.
In that case what is a correlation function value 0 ?

We can extend the example for the position where are some moves with the same evaluation value and the clone chooses other path then original.

I suppose that in that case correlation test will failed.
What are the opinions of experienced Posters and chessprogramers.

Regards
Chris

I must say this again, the "clone" detection is jumping to conclusions. The test we have ran based on Don's idea detects similarity of move selection by definition, no more, no less. Whatever you conclude from the similarity or lack thereof, is up to the observer.

Regarding your question, the test fails in your position, but it won't fail so easily in 1000 positions. It is statistical in nature. A good reliability test was that with different sets of position (Don's vs. Michael's), the conclusions still match. Moreover, re-sampling the data (taking only 10% of the data randomly, in 1000 different combinations), the conclusions are still the same.

Miguel

[michiguel]

Hello,

I am wondering if it is trustable method.

Lets say we do have a position mate in #5,6,...,9 whatever and there are some ways to do that let say A, B, C, D.
The original engine selects path A.
The potential clone has a modification that selects path B.
In that case what is a correlation function value 0 ?

We can extend the example for the position where are some moves with the same evaluation value and the clone chooses other path then original.

I suppose that in that case correlation test will failed.
What are the opinions of experienced Posters and chessprogramers.

Regards
Chris

I think that if such a test (using 1000 or more positions) can show an extreme similarity between all cases of already known clones, then there's no real reason to doubt its reliability.
Just my opinion, but I'm not a programmer.

Graham, just to be precise, what it has been demonstrated is that if an engine is similar to another (clone or not), that similarity will show over different sets of positions. i.e. The similarity seen is not a fluke. To correlate similarity with "families", we will need to run the test over a much larger number of engines. Still, it will be impossible to pin an engine as clone with only this.

However, if a clone hunter wants to use this technique, the proper use will be for a screening. For instance, you suspect that a engine is a clone. A clone of what? Then you run this test to tell you which engines you should look at the code first. OTOH, my instinct tells me that it may serve a purpose to speed up the search for blatant 100% copied clones. We had some of those in the past right? I will be nice to run this tests on those. Once one of those is flagged with an extremely high correlation, the clone hunter can focus on other more time consuming tests (RE, for instance).

Let me be clear once again, it may speed up the search for evidence, but this cannot be looked as "the" evidence. OTOH, I do not believe that any similarity can be dismissed as pure coincidence. The reason may not be cloning, but may be common features in eval that other engines do not have, tuning, etc.

Miguel

[Hood]

Regarding your question, the test fails in your position, but it won't fail so easily in 1000 positions. It is statistical in nature.

Miguel

1000 positions & statistical nature... It is not clear for me if the chess engines are behaving statistically. I think they shall behave logically.

Assumption that they are behaving statisticly is very strong assumption and weak point of the conclusions !

De Morgan rules concerning implication p => q tell if p is false then
p=>q is always true. It causes the saying that " by statistics you can prove what you want " The point is in the assumption.

Do we have someone with grade in statistical math ?

My thinking is that way.
Between thouse 1000 positions there are positions where:
- there are many moves equal,
- there is one move stronger
- there are moves with similar strength
- ...
i would ask for adding sth to the classification

So the positions where one move is stronger then remaining shall be excluded from the test because here all engines shall select it , so correlance here shall be maximal.

I do not understand why the game with the full information can be treated by statistical tools as statistical being. I would agree in the case of bridge, roullette etc.

Rgds
Chris

[michiguel]

Regarding your question, the test fails in your position, but it won't fail so easily in 1000 positions. It is statistical in nature.

Miguel

1000 positions & statistical nature... It is not clear for me if the chess engines are behaving statistically. I think they shall behave logically.

Assumption that they are behaving statisticly is very strong assumption and weak point of the conclusions !

It is not an assumption, it was tested by bootstrap analysis.

De Morgan rules concerning implication p => q tell if p is false then
p=>q is always true. It causes the saying that " by statistics you can prove what you want " The point is in the assumption.

Do we have someone with grade in statistical math ?

My thinking is that way.
Between thouse 1000 positions there are positions where:
- there are many moves equal,
- there is one move stronger
- there are moves with similar strength
- ...
i would ask for adding sth to the classification

So the positions where one move is stronger then remaining shall be excluded from the test because here all engines shall select it , so correlance here shall be maximal.

No, they do not have to be excluded because they do not harm the test at all. They just add more computing time. In fact, if you remove positions, you risk introducing bias. Positions should be removed if *all* the engines play the same move. That is ok, but, who guarantees that if you come with another engine it will play that move?

I do not understand why the game with the full information can be treated by statistical tools as statistical being. I would agree in the case of bridge, roullette etc.

Rgds
Chris

Because two engines that are "similar" are not guaranteed to play the same move 100% of the time. Any minor change could affect move selection. Then, maybe they will play the same move in ~95, 90% of the positions selected etc. The randomness here is not given by the move itself, but by the selection of positions.

Miguel

[jwes]

Regarding your question, the test fails in your position, but it won't fail so easily in 1000 positions. It is statistical in nature.

Miguel

1000 positions & statistical nature... It is not clear for me if the chess engines are behaving statistically. I think they shall behave logically.

Assumption that they are behaving statisticly is very strong assumption and weak point of the conclusions !

De Morgan rules concerning implication p => q tell if p is false then
p=>q is always true. It causes the saying that " by statistics you can prove what you want " The point is in the assumption.

Do we have someone with grade in statistical math ?

My thinking is that way.
Between thouse 1000 positions there are positions where:
- there are many moves equal,
- there is one move stronger
- there are moves with similar strength
- ...
i would ask for adding sth to the classification

So the positions where one move is stronger then remaining shall be excluded from the test because here all engines shall select it , so correlance here shall be maximal.

I do not understand why the game with the full information can be treated by statistical tools as statistical being. I would agree in the case of bridge, roullette etc.

Because engines treat chess as if it were a statistical game (except for positions with exact results). Evaluations are estimates, with means and standard deviations.

[Hood]

"Because engines treat chess as if it were a statistical game (except for positions with exact results). Evaluations are estimates, with means and standard deviations."

Statistical game treated by engines? Hard to believe. The chess engine algorithm shall not differ
too much from the optimization algorithm or controlling algorithm.

Such the algorithms are giving real control values for steering industry plants.
No one is treating such the objects as statistical ones although they are described by mathematical equations which are simulating reality.
If the control values wil be statistical game then i.e. many reactors of catalitic hydrockracking would explode.

Steering value is the same as choosen move more or less.

"It is not an assumption, it was tested by bootstrap analysis."

Bootstrap analysis, it is next statistical tool, statistics is proving that statistics is OK . All time p=>q and big problem with p.

Is there any prove that the bootstrap analysis is adequate tool for that problem.

The chess algorithm working on SP is deterministic one. It is the pure mathematical formula not the statistical animal

Rgds
Chris

[jwes]

"Because engines treat chess as if it were a statistical game (except for positions with exact results). Evaluations are estimates, with means and standard deviations."

Statistical game treated by engines? Hard to believe. The chess engine algorithm shall not differ
too much from the optimization algorithm or controlling algorithm.

Such the algorithms are giving real control values for steering industry plants.
No one is treating such the objects as statistical ones although they are described by mathematical equations which are simulating reality.
If the control values wil be statistical game then i.e. many reactors of catalitic hydrockracking would explode.

Steering value is the same as choosen move more or less.

If their measurements had as large a standard deviation as chess evaluations, they would regularly explode. Another major difference between chess and control algorithms is that in control algorithms there is no opponent trying to make it explode.

[michiguel]

"Because engines treat chess as if it were a statistical game (except for positions with exact results). Evaluations are estimates, with means and standard deviations."

Statistical game treated by engines? Hard to believe. The chess engine algorithm shall not differ
too much from the optimization algorithm or controlling algorithm.

Such the algorithms are giving real control values for steering industry plants.
No one is treating such the objects as statistical ones although they are described by mathematical equations which are simulating reality.
If the control values wil be statistical game then i.e. many reactors of catalitic hydrockracking would explode.

Steering value is the same as choosen move more or less.

"It is not an assumption, it was tested by bootstrap analysis."

Bootstrap analysis, it is next statistical tool, statistics is proving that statistics is OK . All time p=>q and big problem with p.

Is there any prove that the bootstrap analysis is adequate tool for that problem.

Yes.

Miguel

The chess algorithm working on SP is deterministic one. It is the pure mathematical formula not the statistical animal

Rgds
Chris