ICGA Rule 2 discussion

[hgm]

I must add that any attempt so far to thwart the similarity test (such as Uri's Twinfish) has come at a great cost in elo.

Well, 'great' is a relative notion. Some nowadays think they have the right to a certian victory if they are 10 Elo better.

And how many attemps have there been, exactly, 'so far'?

[arjuntemurnikar]

Well, 'great' is a relative notion. Some nowadays think they have the right to a certian victory if they are 10 Elo better.

I don't know what you are talking about.

And how many attemps have there been, exactly, 'so far'?

Not many. What's your point?

It seems to me you are arguing for the sake of arguing. As I said, there is no perfect solution and we must make practical compromises. There will always be cheaters and it is unavoidable. The best we can do is to make it as hard as possible for people to cheat.

[Ralph Stoesser]

Those rules are way too complicated. Chess programming is no science and nothing serious anymore. All the secrets are disclosed. We are talking about a little mind jogging, about programming a game of medium difficulty. It is like cooking by following a well known recipe. Rule 2 should be: Join us for free and forget about the old rules. That would be a good start.

[Adam Hair]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

I must add that any attempt so far to thwart the similarity test (such as Uri's Twinfish) has come at a great cost in elo. It will be very difficult for a participant to bypass the test in the most crudest of fashions and win the championship at the same time.

The cost in Elo is not as great as you may think. I tested Cheng4 0.36 with its PSTs, my PSTs (those are the values used in Twinfish), and a modification of my PSTs. My modified PSTs cost Cheng4 ~31 Elo at 40/30". Assuming (and that is a big assumption) that this difference holds for much longer time controls, it is not enough loss to drastically hurt the chances of winning a Swiss system tournament.

However, it is not clear yet that changing the PST values that drastically creates a false negative. The similarity percentage is definitely reduced, but not so much as to hide the relationship. I will do some more testing to determine how much different PST values do obscure relationships.

There will always be cheaters and you have to just deal with it. Such is the nature of computer programming. In an ideal world, all source code would be open and everything would be transparent, but it is not.

I urge everyone to get practical now, shun your idealism and arrange for a good WCC tournament before the end of this year. I presume that should not be a difficult goal.

[Adam Hair]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this). That might could thwart the sim test without losing too many Elo.

There is a second similarity comparison that could be done, which is compare scores for each position. While it is also not fool-proof, it probably would not be fooled by the scenario described just above. And one run of the similarity tool can produce both the move selections and the scores.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

Exactly.

[bob]

It sounds like you advocate "all derivatives are OK except if they come from IPPOLIT?"

To date, ippolit has not been proven to be a rybka derivative, so what makes it special here? My issue is that I believe ippolit is not original, based on looking at the source code which appears to come from hex rays or something similar. And we have no author to question.

But is it really ok to have fruit derivatives, crafty derivatives, stockfish derivatives, any other open-source derivatives?

No of course not. If a program is derived from another program and it violate's its license, it should not be allowed. Thus, in the case of open-source derivative programs, if no license violation has occured, and the program passes the similarity test, it should be allowed to participate.

Note: I only mentioned IPPOLIT as special case because it is "public domain" but is unoriginal and illegal derivative of rybka (unproven, but highly suspicious). Again, I must emphasize that we can debate about IPPOLIT all day and never come to any scientific conclusion, so lets not get idealistic. It is commonly accepted in the computer chess community that IPPOLIT is an illegal derivative, so lets not waste time about it and just move on. As long as it passes the similarity test, an IPPOLIT derivate should be fine. If not, it should not be allowed.

What about question #2. Ok to have 30 stockfish derivatives? Again, that is not something I want to do. I could run that on my cluster, for example.

30 stockfish derivatives? Again, if they pass the simtest, then they are through. I highly doubt though that they would.

So you want to use the simtest as THE test? Pass it, play, fail it, don't play? This does not do a thing for "originality".

[bob]

Well, 'great' is a relative notion. Some nowadays think they have the right to a certian victory if they are 10 Elo better.

I don't know what you are talking about.

And how many attemps have there been, exactly, 'so far'?

Not many. What's your point?

It seems to me you are arguing for the sake of arguing. As I said, there is no perfect solution and we must make practical compromises. There will always be cheaters and it is unavoidable. The best we can do is to make it as hard as possible for people to cheat.

I think his point was that the most recent attempt was done SOLELY to show that one can make a copy that contains 99.9+% of a program, yet passes simtest. There was no effort to try to find the OPTIMAL set of changes that let it pass the simtest without wrecking the Elo.

The problem here is that as programs get stronger, they will get more similar. At some point they will ALL play exactly the same moves, obviously.

[bob]

That sounds like a very bad idea. It is only a matter of time before people will learn how to thwart the simtest, and then you basically give them permission to do whatever they want.

One such way that comes to my mind is that if the score for the second best move lies within x cp of the best move, report the second move as the best move (I am assuming there is no technical difficulty in doing this). That might could thwart the sim test without losing too many Elo.

There is a second similarity comparison that could be done, which is compare scores for each position. While it is also not fool-proof, it probably would not be fooled by the scenario described just above. And one run of the similarity tool can produce both the move selections and the scores.

There must be time-resistent objective rules of what is allowed. The simtest could at best be a tool used to unmask cheaters that violate the objective rules.

Exactly.

Scores are not so good. One can simply initialize a random number for the base score, so that rather than score = 0 at the top of eval, it is score = initial;. Change initial each time the program is started. Scores will vary all over the place, run to run, but since scores are relative, whether you start off at +100 or -100 makes absolutely no difference to the move actually played.

[bob]

First, here is the current ICGA Rule 2, which is regarding originality:

2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.

It is my understanding that Levy rewrote them for the Yokohama tournament.

When his changes were discussed, your feedback was that there was no need for change. Good to see your mind has changed. The new rules are not much better IMHO, but it is good that it is recognised that something needed to be done about them and a step was taken.

You are mis-reading. I am not trying to CHANGE rule 2. I am trying to carefully CLARIFY rule two. To get around things like Vas' comment that "yes, if I type the code then it is original". It seems wise to try to explain it as clearly and carefully as possible, so that we don't have to deal with all the nonsensical excuses when someone gets caught. If you notice, nothing I wrote has changed the originality requirement at all. I simply included the things we had already agreed was acceptable to copy (egtb, magic, rotated, etc) as another point of clarity.

[bob]

Those rules are way too complicated. Chess programming is no science and nothing serious anymore. All the secrets are disclosed. We are talking about a little mind jogging, about programming a game of medium difficulty. It is like cooking by following a well known recipe. Rule 2 should be: Join us for free and forget about the old rules. That would be a good start.

Nobody would enter. This was tried last year in fact.