ICGA Rule 2 discussion

[bob]

First, here is the current ICGA Rule 2, which is regarding originality:

2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.

Here are some suggested expansions to discuss:

2a. One can not copy and translate an existing program and call the result original. Examples include copying a mailbox program and converting it to bit boards, but keeping the same search strategy, same evaluation components, etc, with the only difference being the basic board representation; copying a program written in language X and converting it to language Y, keeping the basic search, evaluation and such unchanged except for the new language; taking an existing program in one window and rewriting on the fly in another window, but keeping the same overall features and organization as the original. There are issues to discuss here. For example, what if you copy and translate, but then over time modify the code so that it is significantly different. How different must it be, and MOST IMPORTANTLY, how can the rule be written so that it can actually be objectively enforced?

2b. One can not copy a significant part of one program and use that in another, except for exceptions enumerated below.

2c. Parts of a program that may NOT be copied include the search, move generation, move ordering, evaluation, hashing, book move selection code, any learning code, pondering code, etc. The intent here is to exclude any code, expressed as y=f(x) where f() is the function being copied, and for any value x, it is possible to write f() such that it produces different values given the same input. Things like SEE do NOT fit in here because there are lots of options within SEE. Deal with pins or not? Exclude illegal moves or not? Detect pieces attacking behind an attacking piece. Etc.

2d. Given the same y=f(x) notation, Ii y is derived from x, where there is only one possible y for a give value of x, then copying is acceptable. Examples of this include egtb probe code. For a given position, you get a given score that will not change unless the position changes; the most significant bit of a value is constant for a specific input; A random number generator also fits here since for a starting seed, the string of PRNs will be constant;

2e. Exceptions. Certain pieces of code have been developed independently and have universally become accepted and do not fall under the copying/originality rule. These include (a) endgame database probe code where the original author has either made the code public domain, or has given a participant explicit permission to use that code and data. (b) magic move generation code, but ONLY the part that actually produces a set of squares that are attacked. Anything beyond that is considered part of a program's move generator and copying that does violate this rule; (c) S. Edwards' EPD kit for those wanting to include EPD support in their program (this does not actually play chess anyway, but is included here for completeness). (d) miscellaneous bits and pieces of code, often written in inline assembly language, to handle things like spin locks, bit counts, find LSB/MSB set, and such Note that ALL of these fall under 2d above since they all have a specific output for any specific input.

I would hardly call that all-inclusive yet, but it gives a starting point. I do NOT think a "public domain source" can be copied and modified and then used. Supposedly Ippolit was made "public domain" and I do NOT believe we want 10 copies of that in a chess tournament.

Comments and suggestions welcome. Please keep the complaining, Rybka stuff, and such elsewhere. It would be nice to put something together and pass it on to the ICGA for consideration...

[bob]

An addendum, for those interested in the similarity test as a possible filter or qualification step.

1. We know it can produce false negatives. That is, fail to recognize that a program is a 99.999% copy of another program. The twin fish experiment shows that clearly.

2. That suggests that it can also produce false negatives. That is, suggest that two programs are related when they are not.

I would suggest the following usage:

(a) run the test. If two programs are shown to be related, then a source comparison must be done before the new program is admitted. There's much less potential for error there. And note that I DID say "source". Not binary. All of this before an event. If the entrant does not want to provide source to some sort of originality verification process, no play.

(b) we still need the protest option, where someone can present credible evidence that code was improperly copied. Note that the similarity test completely fails if you just copy my move ordering, or my hash code, etc. Similarity test really depends on evaluation and then search to a lesser degree. Credible evidence again causes an investigation where source is compared as in (a) above.

[Steve Maughan]

Hi Bob,

(...)2. That suggests that it can also produce false negatives. That is, suggest that two programs are related when they are not.(...)

With respect I don't think 1 implies 2. Do we have any concrete evidence it's the case?

I asked for a similarity test for Maverick as it has a v basic evaluation. I thought a simple evaluation may possibly throw up a false negative. The similarity score showed it was unlike anything else.

To validate I challenge anyone to change a simple open source engine (e.g. Fruit or Maverick) so it is shown to be the same (>55% similarity) as a top engine (e.g. Stockfish).

Steve

[bob]

Hi Bob,

(...)2. That suggests that it can also produce false negatives. That is, suggest that two programs are related when they are not.(...)

With respect I don't think 1 implies 2. Do we have any concrete evidence it's the case?

I asked for a similarity test for Maverick as it has a v basic evaluation. I thought a simple evaluation may possibly throw up a false negative. The similarity score showed it was unlike anything else.

To validate I challenge anyone to change a simple open source engine (e.g. Fruit or Maverick) so it is shown to be the same (>55% similarity) as a top engine (e.g. Stockfish).

Steve

I didn't particularly say 1 implies 2. I said that it has been proven to show false negatives, and based on that, it seems obvious to me that a false positive is just as possible. We have only seen one false negative (verified) and no false positives. YET. We didn't know we could put a man on the moon until we did it, however. This is one of those "decidability" issues where the only way you can say with 100% certainty that no false positives exist is to test every possible pair of programs, even those not yet written, and show there are none. A task of infinite complexity. We see grammars where you can answer "yes, this string is a member of this language" but you can't say "no, this string is not a member" and vice-versa. Leads to lively discussions in theoretical comp sci courses. But it doesn't help here, because we simply do not know that no false positive will happen.

I neither want to include a derivative, nor exclude a non-derivative, just because we have a test failure.

Your challenge is interesting. But a BIG waste of time for the person trying to do it. If someone were to put up some sort of prize fund, you might get a taker and answer this question factually. Otherwise who would want to waste the time?

[arjuntemurnikar]

1. Programs can copy from any public domain, except for IPPOLIT. (since IPPOLIT is controversial, and this can be verified using the similarity tool against rybka). This rule would not apply if a program has been significantly modified and it can be objectively verified.
2. The similarity tool is the best that we have right now and it should be used. The first step would be to ask the programmer to send his source to be verified by a special panel as to its originality. If the programmer does not wish to comply with this test, the similarity tool would be used to verify its originality. If the results of this test are not acceptable to the programmer, he may protest, but then he must be able to send objective evidence to support his case. (Yes, the similarity tool is not perfect or ideal, but it is the best we have right now, so lets not get too idealistic and just deal with it.)
3. The biggest problem IMO is not the originality rules, it is the sponsorship. In this day and age, not many programmers are going to travel abroad just to participate in a tournaments. For most, computer chess is merely a hobby. I propose we switch to a fully-online format (like CCT) and we broadcast all games to the public like TCEC so that people can get involved in the tournament instead of the tournament happening in some corner of the world in some remote private function room. This was not possible in the olden days but it is not difficult now. I see no reason to cling on to legacy formats. To survive, one must evolve. This also bypasses the problem of high travel expenses and also brings entertainment to the fans (exactly what a World Championship ought to do in the first place!)
4. Bonus: If we can muster sponsorship from entities, such as Chessbase or Chessbomb for example, (maybe this will be possible in the 3rd or 4th iteration if the event becomes popular enough), then we can redistribute the sponsorship money among tournament organizers and participants and even give out prizes to the winners and runner ups. This will further increase interest in the tournament for the participant programmers.

In summary I would say:
public domain is fine, except for IPPOLIT
don't get idealistic about originality tests
switch to an online-only format because it ain't 1985 anymore

[bob]

1. Programs can copy from any public domain, except for IPPOLIT. (since IPPOLIT is controversial, and this can be verified using the similarity tool against rybka). This rule would not apply if a program has been significantly modified and it can be objectively verified.
2. The similarity tool is the best that we have right now and it should be used. The first step would be to ask the programmer to send his source to be verified by a special panel as to its originality. If the programmer does not wish to comply with this test, the similarity tool would be used to verify its originality. If the results of this test are not acceptable to the programmer, he may protest, but then he must be able to send objective evidence to support his case. (Yes, the similarity tool is not perfect or ideal, but it is the best we have right now, so lets not get too idealistic and just deal with it.)
3. The biggest problem IMO is not the originality rules, it is the sponsorship. In this day and age, not many programmers are going to travel abroad just to participate in a tournaments. For most, computer chess is merely a hobby. I propose we switch to a fully-online format (like CCT) and we broadcast all games to the public like TCEC so that people can get involved in the tournament instead of the tournament happening in some corner of the world in some remote private function room. This was not possible in the olden days but it is not difficult now. I see no reason to cling on to legacy formats. To survive, one must evolve. This also bypasses the problem of high travel expenses and also brings entertainment to the fans (exactly what a World Championship ought to do in the first place!)
4. Bonus: If we can muster sponsorship from entities, such as Chessbase or Chessbomb for example, (maybe this will be possible in the 3rd or 4th iteration if the event becomes popular enough), then we can redistribute the sponsorship money among tournament organizers and participants and even give out prizes to the winners and runner ups. This will further increase interest in the tournament for the participant programmers.

In summary I would say:
public domain is fine, except for IPPOLIT
don't get idealistic about originality tests
switch to an online-only format because it ain't 1985 anymore

It sounds like you advocate "all derivatives are OK except if they come from IPPOLIT?"

To date, ippolit has not been proven to be a rybka derivative, so what makes it special here? My issue is that I believe ippolit is not original, based on looking at the source code which appears to come from hex rays or something similar. And we have no author to question.

But is it really ok to have fruit derivatives, crafty derivatives, stockfish derivatives, any other open-source derivatives?

[arjuntemurnikar]

I might add:

5. The number of games in the tournament should be significantly increased. The tournament should last a reasonable number of days (at least 7 days) and it should be switched to a knock-out format, similar to TCEC.
6. We can have both open-hardware, and software-only competitions. Since the programmers are administrating their engines from the luxury of their own home, they can use any hardware they wish. For software-only competitions however, we should fix some hardware and have the programmers send in their programs to be run on it. They can of course make changes to their programs between matches as and when they desire. Custom opening books should be allowed, of course.

[hgm]

2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.

I would like to see a clarification of the "or the source of such code".

The exisiting rule explicitly allows "inclusion of game-playing code written by others", and thus apparently does not consider such code by definition "non-original work". It is just an orthogonal property, code written by others can be an original work (or not) of these others, and this is apparently what matters.

The rule does state I must list the name of these others in the submission details. It does not seem to require that these others actually sign in person. Is there another rule that requires that?

In any case it offers an alternative to giving the names: it is enough if I just say how I obtained the code. This does not seem to exclude code from anonymous authors. Of course common law in that case would force me to show that I can legally use that code, but a license agreement that came with the code would satisfy that requirement.

I don't know if this is all intended, or just an unforseen side effect. But I would like to see it made explicit that legal use of open-source code is in your program is not a violation of rule #2 per se, but will be treated as any other code written by a co-author that has given permission to enter it in the ICGA.

This will require that there also must be a clarification as to what should happen when two entrants want to use the same code. (Which can also happen when former collaborators fall out with each other, and continue development on their own. So a rule for this is needed anyway.) I would propose that in such a case the code can only be used by one of the participants, and that in cases where the would-be participants cannot agree about which of them will enter, they can be obliged to play a qualifyer match to decide this.

[arjuntemurnikar]

It sounds like you advocate "all derivatives are OK except if they come from IPPOLIT?"

To date, ippolit has not been proven to be a rybka derivative, so what makes it special here? My issue is that I believe ippolit is not original, based on looking at the source code which appears to come from hex rays or something similar. And we have no author to question.

But is it really ok to have fruit derivatives, crafty derivatives, stockfish derivatives, any other open-source derivatives?

No of course not. If a program is derived from another program and it violate's its license, it should not be allowed. Thus, in the case of open-source derivative programs, if no license violation has occured, and the program passes the similarity test, it should be allowed to participate.

Note: I only mentioned IPPOLIT as special case because it is "public domain" but is unoriginal and illegal derivative of rybka (unproven, but highly suspicious). Again, I must emphasize that we can debate about IPPOLIT all day and never come to any scientific conclusion, so lets not get idealistic. It is commonly accepted in the computer chess community that IPPOLIT is an illegal derivative, so lets not waste time about it and just move on. As long as it passes the similarity test, an IPPOLIT derivate should be fine. If not, it should not be allowed.

[hgm]

1. Programs can copy from any public domain, except for IPPOLIT. (since IPPOLIT is controversial, and this can be verified using the similarity tool against rybka). This rule would not apply if a program has been significantly modified and it can be objectively verified.

It is a bad idea to use explicit names when designing rules. Tomorrow a 'Robbodini' could be published that is just as illegal, or even more obviously so, than IPPOLIT, and your rule would then allow it??? The rules should specify objective criteria. Like whether the code can be legally used. Note that a public-domain or GPL license is void when someone slams it on stolen code.

3. The biggest problem IMO is not the originality rules, it is the sponsorship. In this day and age, not many programmers are going to travel abroad just to participate in a tournaments. For most, computer chess is merely a hobby.

Well, it is a hobby for me, and I would like to travel. This seems a pointless rule. Participation in CCT was not significantly better than most WCCC or Leiden, except at the very low (2400-) end. It is of much more importance where the event is held. Japan of course is not the best place for Chess. (But it is the best place for sponsors...)

I propose we switch to a fully-online format (like CCT) and we broadcast all games to the public like TCEC so that people can get involved in the tournament instead of the tournament happening in some corner of the world in some remote private function room. This was not possible in the olden days but it is not difficult now.

The last two WCCCs actually did have live broadcasts. Not as advanced as I would have liked them and could provide them, but they were there.

... then we can redistribute the sponsorship money among tournament organizers and participants and even give out prizes to the winners and runner ups.

Dream on.