Adam Hair's article on Pairwise comparison of engines

[CRoberson]

Adam,

I scanned the article. Nice.

My thoughts concern: "There are 4851 pairs. The average matched move percentage was 45.16, the standard deviation of the data was 2.86. "

Looks like you included known clones/derivatives in the calculation of the standard deviation. I think their inclusion biases the standard deviation. Of course, it makes it smaller. Shouldn't the standard deviation be calculated without the known clones? It would make the number larger (maybe only slightly).

[Dann Corbit]

Adam,

I scanned the article. Nice.

My thoughts concern: "There are 4851 pairs. The average matched move percentage was 45.16, the standard deviation of the data was 2.86. "

Looks like you included known clones/derivatives in the calculation of the standard deviation. I think their inclusion biases the standard deviation. Of course, it makes it smaller. Shouldn't the standard deviation be calculated without the known clones? It would make the number larger (maybe only slightly).

In order to do that you have to presuppose the result you hope to establish (assuming that you do not really know every single clone).

An interesting experiment would be to take known clones and see the match rate and standard deviations.

A further interesting experiment would be to find the highest match rate and standard deviations for engines known definitely not to be clones.

Without doing these experiments, I wonder what the controls are.

[Adam Hair]

Adam,

I scanned the article. Nice.

Thanks, Charles.

My thoughts concern: "There are 4851 pairs. The average matched move percentage was 45.16, the standard deviation of the data was 2.86. "

Looks like you included known clones/derivatives in the calculation of the standard deviation. I think their inclusion biases the standard deviation. Of course, it makes it smaller. Shouldn't the standard deviation be calculated without the known clones? It would make the number larger (maybe only slightly).

Actually, their inclusion makes the standard deviation from the mean larger.

[ernest]

I scanned the article.

Would be nice to know what article !!!

[Robert Pope]

I believe it is: http://www.top-5000.nl/clone.htm

[ernest]

Thanks !

[Graham Banks]

I believe it is: http://www.top-5000.nl/clone.htm

Interesting to note that Rybka and Fruit don't show up as a pair to be suspicious about.

[Rebel]

I believe it is: http://www.top-5000.nl/clone.htm

Interesting to note that Rybka and Fruit don't show up as a pair to be suspicious about.

It's and indication but not more than that. The SYM tool pretty much with precision can detect a clone and even a close derivative. It can not proof an engine is clean.

[Roger Brown]

I believe it is: http://www.top-5000.nl/clone.htm

Interesting to note that Rybka and Fruit don't show up as a pair to be suspicious about.

Hello Graham,

You should be cautious when referring to other works.

The information presented doesn't attempt to validate or invalidate suspicions, so it really isn't as interesting as you think it is.

Later

The need to avoid false accusation is greater than the need to determine authors who break the rules slightly. In other words, it is better to let lesser offenders slip through than to make accusations against innocent authors.

This tool should not be used solely for determining derivatives and clones. Other methods should be used in conjunction with this tool. Ultimately, any accusation of cloning requires an examination of the code of the accused author.

[Norm Pollock]

Adam,

Would it be unreasonable to test a 2nd copy of each engine to establish consistency?

I think you need to have some assurance that each engine is strongly consistent and will produce the same move from the same position the great majority of the time if given a 2nd chance.

After running a second version through the 8000+ positions, the 2 versions of each engine can be compared. If the versions don't have at least 95% matched moves, then I would not consider the engine consistent and possible disqualify the engine from the test.

-Norm