a beat b,b beat c,c beat a question

[Uri Blass]

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

[Gerd Isenberg]

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

I think this is the usual szenario if you do two or more changes
The huge test-dilemma!?

Gerd

[mjlef]

The solution is simple, never, ever play version X+2 of any program versus version X and cause this loop of confusion. If you do this it will just give you a headache! I know it has me!

[Michael Sherwin]

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

This is what I call the 'marbles effect'. Take a handfull of marbles into a big flat parking lot (hope no one sees you) and then drop them a few times to see the large variance in the random patterns they create. Marbles in a programming sense is the randomness associated with any change to the code that causes one program to see what another does not or just simply causes programs to choose a different move now and then.

[brianr]

Firstly, I think you are already in pretty good shape if in fact you can actually test two versions of your program to the extent that you see a "significant score" difference.

More importantly, many have said testing an engine against itself may highlight differences that testing an engine against a suite of other engines would not show. These same engine differences are likely to be not as meaningful as those identified by suite testing, IMO.

[tiger]

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

The answer to this paradox is probably in your definition of "beat with a significant score".

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?


// Christophe

[bob]

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

90% of such results are caused by playing _far_ too few games. 80 game matches mean nothing when the two programs are anywhere near close to each other.

As far as the other 10%, yes I have seen that happen, but I couldn't begin to remember what the changes were... But nothing says that this is a transitive relationship...

[Bill Rogers]

I have something similiar happen in the past and learn the following rules:
1. adding a new algorythm or new information will not cause the aboration.
2. modifing an existing algorythm in rare cases might cause the problem.
3. deleting part of all of and old algorythm can definately cause the problem.
Enough of my redundant comments.
Bill

[Uri Blass]

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

The answer to this paradox is probably in your definition of "beat with a significant score".

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?


// Christophe

I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).

Uri

[tiger]

Do you have a case when the following happened:
1)version X+1 of your program beat version X with a significant score.
2)version X+2 of your program beat version X+1 with a significant score.
3)version X of your program beat version X+2 with a significant score.

If the answer is positive then what are the changes that you made from X to X+1 and what are the changes that you made from X+1 to X+2?

Uri

The answer to this paradox is probably in your definition of "beat with a significant score".

Even if you use 95% of confidence error margins, never forget that in one match out of 20 you will get a result outside the error bars.

How many games did you play in order to establish that "paradox"?


// Christophe

I did not claim that I played enough games to have significant results with this paradox and I ask other if it happened to them(In my case I admit that in most cases I simply did not play enough games even to get one significant result because unfortunately I am always too lazy to play thousands of games between 2 versions).

Uri

OK, then yes it happened to me even with large numbers of games, and I think you should not care too much about the kind of changes that have produced these results because they are not the real cause.

On the other hand it is probably theorically possible to design 3 programs A, B and C so A always beat B, B always beat C and C always beat A. But I'm not sure doing so will help you solve your paradox.


// Christophe