CCT 12: Rule changes and upcoming faster event.

[michiguel]

......

The reason for the shorter time control tournaments is to help with gaining rating and establishing your rating deviation on FICS.

.......

Peter

That lame equation again. That is why I stopped playing on FICS as a human. The rating equation is wrong. Even the author admitted it
years ago and modified the equation to adjust for some glaring issues with it. Then he applied for a patent on the new equation. So, you have to pay to use it (that was the case 4 years ago).

Is FICS using the new equation or the same bad old one?

A patent for an equation? What on earth is that?

BTW, why should we care about these ratings to organize a tournament?

Miguel

[Peter Skinner]

BTW, why should we care about these ratings to organize a tournament?
Miguel

Purely for seeding value. Then we can get an accurate rating on the programs entered by getting actual games played with the most recent versions.

While using the CCRL/SSDF and other rating lists are fine, there currently is no way to manually set the seeds on the mamer bot on FICS.

I could do it manually like always, but most would prefer a bot running it like the ACCA on ICC.

Peter

[CThinker]

Good idea, Peter.

I have actually suggested long time ago that we move to FICS for online tournaments. I have also suggested that we use Harm's server (that is, when all the mamer quirks gets fixed). I have always argued that guests on ICC have limited ability to interact.

My experiences with FICS are mixed.

1. It took more than one year for me to get an account at FICS. I applied a number of times, but got nowhere. Finally, there was this one nice fellow, (handle: beuki), who was an admin there. Even then, I never got an account. Seven months later, he checked-up on me to see if I have gotten the account. He then had to follow-up on it. Weeks later, I finally got one. Total turn around time is more than one year.

2. There are times when FICS would go down for several days.

I also like your idea that the participants would go by their handle as their engine names. However, some of the engine names are already taken. Take the case of 'Thinker'. That account is already taken (but that has not connected since 1995, and played only 1 game). The same goes for the handle 'Fruit', which has not connected since 1999 (long before the Fruit engine was created). You see a lot of these 'abandoned' account names in FICS.

Lastly, the current Thinker (LancePerkins) account on FICS is for the Pocket PC version of Thinker. I believe FICS admins don't like it when you change the hardware or the engine. I was told that it ruins the rating and history.

Again, I like the idea of moving to FICS. Although, I still think that we should consider Harm's server. He can have full control of the accounts there.

To summarize my reasons on why we should start moving away from ICC, and hopefully Harm's server:
1. Better control of accounts and account names.
2. Friendlier environment for the tournament. The server can be configured for the tournament only. E.g., the start-up screen can say, "Welcome to the CCT. The useful commands are...", etc.
3. Full set of features are available to 'guests'.
4. The server can be potentially hosted by 'anyone'. Harm has the working source. That's way better fault tolerance than ICC or FICS.

The only downside to Harm's server right now is the bugs with mamer. So, FICS is good enough for me (still better than ICC).

[Michel]

It took more than one year for me to get an account at FICS

I really don't understand this. I got my account immediately. Did you follow the proper procedure? It is described when you do "help computer_app". I had to provide some motivation but for engine authors there should be absolutely no problems. My son also easily got an account (it has now been deactivated since he has not used it for a long time).

Here are the rules (quoting from the help file computer_app).

o Priority is given to those who can play variants.

o Priority is given to automatic computers (not need of an operator.)

o Priority is given to those who have had an account here for some time.

o Priority is given to those which are built from home grown code and
operated by the programmer.

Of course, the main criteria in deciding on which computer accounts are
allowed is the current demand.

[hgm]

Of course any feedback is welcomed and taken seriously. The changes I have made are result of direct communication with authors since the last CCT event.

Peter Skinner
CCT Tournament Director

OK, the discussion has drifted away a bit from the rule changes, but I think the current rules or the way they are formulated are still wanting.

For one, it is still not clear if authors (that are present on-line) can enter more than one engine. The current rules only say you cannot _operate_ two engines. But what if an author enters two engines (which has been allowed on the three previous instances of CCT), operates one himself, (so he will be on-line), and appoints an operator for the other?

My take on this is that this should remain a tournament for engines, not for authors. (Which doesn't have any bearing on the fact that authors should be present.) That is, if two engines are different enough that they would not be considered clones of each other (i.e. no shared code, correlation of their moves and evaluations below a certain level) when they have different authors, it should be possible for these two engines to also participate if they were written by the same author.

I don't see any use for the rule to forbid one operator oprerate two engines? It is a pointless insult to operators, to tell them they are assumed to be cheaters. Pointless, because if you are really afraid of cheating of the type where two title candidate fix their result to favor the one with the best chances, that type of cheating only needs a tell between two different operators. What if I operate strong program X and my brother operates strong program Y? I think the basic assumption should be that people are honest, rather than go the path of paranoia.

What is the meaning of "team member"? And who decides about that? If I (an author) would say: "this person you have never heard of always tests my engine for me", is that good enough? Can it be a book builder, can it be a strong chess player that (I say) points out to me the weaknesses of my evaluation?

Can an author that is recognized for program X enter a completely unknown program Y? Or does the program has to be recognized, rather than the author? What if next month I were to release an engine that is approximately as strong as, say, Fruit? Could it participate?

This rule of being recognized seems inviting problems. For one, if every event would require such a rule, new authors would be shutout forever. So when generally applied the rule would not work at all, making adopting such a rule "anti-social behavior". Furthermore, by this rule you make your tournament subsidiary to other tournaments, deferring the authority (and problems) to others.
This rule is really very anti-beginner, anti-weak engine. CCRL only tests engines above a certain strength, Paderborn was cancelled, hardly anyone participates in WCCC because it is outrageously expensive, ACCA events are only for Americans. So only Chess War seems to remain.

I think anti-clone rule #3 is too harsh. It should always be possible for authors to keep a particular work confidential. It is reasonable to bar them from entering this tournament, but seems very unreasonable to extend such a suspension indefinitely into the future. E.g. the program in dispute could be made open-source later, or the same author could write another open-source engine later, or even a private engine which he is willing to make available for scrutiny to the organizers.
If a program is suspect, the request of it being checked out should be made before the tournament, and it should be possible for the author to withdraw in that case without further consequences other than that he will not play (with that engine) in this single event.

If what you really want is to have an invitation tourney for the world top-N programs, then just publish the list of who can.

[Peter Skinner]

For one, it is still not clear if authors (that are present on-line) can enter more than one engine. The current rules only say you cannot _operate_ two engines. But what if an author enters two engines (which has been allowed on the three previous instances of CCT), operates one himself, (so he will be on-line), and appoints an operator for the other?

I have never said that one author can't enter two engines. Charles and Richard have entered both their programs in the past without an issue.

I don't have a problem with one author entering two programs as long as both programs are unique.

Why have 100 versions of Crafty when they are based on the same code structure? Charles has two programs that are indeed much different than each other with NoonianChess and Telepath.

Richard has The Baron and Crash Test Dummy. Both entered and played without issue.

If they are unique from each other, then play on. If both are of similiar strength, as in Alaric/Bright, then why use both? Are they really so different if the same strength?

I don't see any use for the rule to forbid one operator oprerate two engines? It is a pointless insult to operators, to tell them they are assumed to be cheaters. Pointless, because if you are really afraid of cheating of the type where two title candidate fix their result to favor the one with the best chances, that type of cheating only needs a tell between two different operators. What if I operate strong program X and my brother operates strong program Y? I think the basic assumption should be that people are honest, rather than go the path of paranoia.

While I believe that most "operators" would be honest and not try to manipulate a game or tournament, others don't believe so.

This is to prevent this from happening in the event that it could, but it also ensures that the operator has more than "running" knowledge of the program entered.

I have 5 computers in my home, so what if I wanted to run all 5, then my internet goes out, and 5 authors are stuck in la-la land with no person to operate?

How well do you think that would go over? One person pretty much ruining a tournament?

What is the meaning of "team member"? And who decides about that? If I (an author) would say: "this person you have never heard of always tests my engine for me", is that good enough? Can it be a book builder, can it be a strong chess player that (I say) points out to me the weaknesses of my evaluation?

If you look at the entry for Crafty, we are very transparent in who a "Team Member" is and who isn't. One of the 4 people on the registration page can tell you what is new with Crafty, our testing methods, any changes we made for the tournament... pretty much anything. We are _all_ involved in the discussion and changes. We test probably like no one else due to the clusters at UAB, and Robert's access.

Or would you rather talk to someone during the game that doesn't know if they are using the right hash size, or why the tablebases aren't working?

Can an author that is recognized for program X enter a completely unknown program Y? Or does the program has to be recognized, rather than the author? What if next month I were to release an engine that is approximately as strong as, say, Fruit? Could it participate?

This really depends on the author to be quite truthful. If the author is known, then likely I wouldn't have an issue with it.

If Joe Blow enters with something close to the strength of Toga/Fruit, wouldn't you think there would be suspicions?

This rule of being recognized seems inviting problems. For one, if every event would require such a rule, new authors would be shutout forever. So when generally applied the rule would not work at all, making adopting such a rule "anti-social behavior". Furthermore, by this rule you make your tournament subsidiary to other tournaments, deferring the authority (and problems) to others.

New authors join all the time. They would never be shut out. Some when asked have even sent me their source code to verify they are legitimate entries.

I am not doing these events to harm people. In fact quite the opposite.

I think anti-clone rule #3 is too harsh. It should always be possible for authors to keep a particular work confidential. It is reasonable to bar them from entering this tournament, but seems very unreasonable to extend such a suspension indefinitely into the future. E.g. the program in dispute could be made open-source later, or the same author could write another open-source engine later, or even a private engine which he is willing to make available for scrutiny to the organizers.

If someone is caught cheating, why would we want them back? Have I ever kicked out an engine? Suspended anyone? Nope, want to know why?

Because I ask for code with courtesy and respect. Authors have _never_ said no to me before, during, or after an event. If someone makes a claim, I investigate it. Even private engines have given me their code and the executable running in the event to check their games and what not.

I want as many engines playing in the tournament possible. I am not there to piss everyone off or make them never enter again. I try to be as fair as possible, sometimes to the point it makes others angry, but they always respect my decisions because I remain impartial.

If what you really want is to have an invitation tourney for the world top-N programs, then just publish the list of who can.

If I wanted only the strongest programs entered I could make it happen. Do I? Of course not.

During the event I watch more games of the lower rated entries than the top entries because I like the "what if's" that happen. The blunders. The mistakes. It is much more entertaining.

I like watching an engine grow. Almost like baby steps.

Not only that, the lower rated entries are almost always the actual authors sitting there talking.

To be honest the last CCT was brutally boring for me. There was no discussion in channel 64 from authors. If it wasn't for the banter with Zach, I would have nearly shot myself in the head. It really was that boring...

CCT9 was the best one in my books. The discussion was lively, so many authors were available to talk to. Vas and Anthony had some great discussions about Rybka and Zappa. The Hiarcs team was terrific with communication and "egging" people on during games. It was so much fun as a fan of computer chess.

That is all I am trying to get back to.... the fun of it all.

Peter

[bob]

May as well start another firestorm, so here goes. I am now firmly convinced that for every four programs that participate in a computer chess event, one of them is a copy/clone of something else.

And I don't see an easy way to solve that. The cheaters will not participate if you require copies of the source, since you could then compile and verify output to that which is whispered/kibitzed.

I think we are nearing the end of useful life for computer chess tournaments, since we have such a significant number of people that have never heard of the concept called "ethics" or "pride" or whatever.

As I discussed with a few via private email, the genie is out of the bottle, and with several strong open-source programs (stockfish is the latest, fruit/toga prior to that) it is simply too easy to copy rather than doing the hard work for yourself.

Seems like a pretty sad day for computer chess in general, unfortunately. Now, about the only viable I see that is workable is to simply play against programs written by people I personally know. Too many "instant wonders" are showing up. Computer Chess software development just doesn't work like that. You don't suddenly show up with a 2800+ program with no prior exposure. Although quite a few expect us to believe this nonsense.

CCT may well become "Computer Clone Tournament" at this rate.

[Peter Skinner]

May as well start another firestorm, so here goes. I am now firmly convinced that for every four programs that participate in a computer chess event, one of them is a copy/clone of something else.

And I don't see an easy way to solve that. The cheaters will not participate if you require copies of the source, since you could then compile and verify output to that which is whispered/kibitzed.

I think we are nearing the end of useful life for computer chess tournaments, since we have such a significant number of people that have never heard of the concept called "ethics" or "pride" or whatever.

As I discussed with a few via private email, the genie is out of the bottle, and with several strong open-source programs (stockfish is the latest, fruit/toga prior to that) it is simply too easy to copy rather than doing the hard work for yourself.

Seems like a pretty sad day for computer chess in general, unfortunately. Now, about the only viable I see that is workable is to simply play against programs written by people I personally know. Too many "instant wonders" are showing up. Computer Chess software development just doesn't work like that. You don't suddenly show up with a 2800+ program with no prior exposure. Although quite a few expect us to believe this nonsense.

CCT may well become "Computer Clone Tournament" at this rate.

You are quite correct in that no one just shows up with a 2800+ program. I am trying to figure out a way of getting the honest programmers in the tournament while keeping those less than honest out.

Obviously the rules I have set out are not reasonable. In the past, we had someone clone a 2000 rated program to enter. Now this was done purely to spite me and my efforts against piracy, but none the less, it happened.

This is why I want the author in attendance. It would be pretty easy to weed the copiers from the authentic pretty quick with technical questions about implementations in their programs.

If they can't answer.. they probably have no clue...

Peter

[hgm]

I have never said that one author can't enter two engines. Charles and Richard have entered both their programs in the past without an issue.

I don't have a problem with one author entering two programs as long as both programs are unique.

Why have 100 versions of Crafty when they are based on the same code structure? Charles has two programs that are indeed much different than each other with NoonianChess and Telepath.

Richard has The Baron and Crash Test Dummy. Both entered and played without issue.

If they are unique from each other, then play on. If both are of similiar strength, as in Alaric/Bright, then why use both? Are they really so different if the same strength?

OK, great! Note that I am not trying to force a rule change or anything; I just want to be helpful is preventing discussions as we had in the Spark/Bright case for WCRCC. And we can only prevent that by making the rules very unambiguous. So if authors can enter two engines, I think it would be wise to state that explicitly. And if they cannot do it when the rating is too close, or if a game between them when they are both candidates for first place will be set to a forfetary draw, please put it in writing. (E.g. you could write that for engines differing less than 250 Elo the TD decides if they will be both allowed.)

Why would people want to play two engines that are similar in strength? Who knows? There could be very valid reasons. Like when you have an old engine, and have decided to write a new one based on an entirely different principle (e.g. you had a strong single-CPU mailbox engine, and now are writing an SMP bitboard engine), and want to compare their performance under "battle conditions". You shouldn't rely on the assumption that people don't want to do it because you see no point in it.

While I believe that most "operators" would be honest and not try to manipulate a game or tournament, others don't believe so.

This is to prevent this from happening in the event that it could, but it also ensures that the operator has more than "running" knowledge of the program entered.

I have 5 computers in my home, so what if I wanted to run all 5, then my internet goes out, and 5 authors are stuck in la-la land with no person to operate?

How well do you think that would go over? One person pretty much ruining a tournament?

So you want it for risk dispersion. I guess that is a valid reason. But then being adamant on allowing only a single program without any exceptions is overdoing it. I would even argue that it is counterproductive. E.g. if Charles wanted to enter Telepath and Noonian, I am pretty sure that the probability of his internet connection failing is far smaller than that something would go wrong when he would refer one of the engines to an operator that is not familiar with it. In the worst case you would have to drop two engines, which hardly ruins a tournament. Note that spreading risks this way only serve to limit the maximum damage, and have no or negative impact on the average damage. If programs are playing through different internet connections, this doubles the probability that one connection will fail.

If you look at the entry for Crafty, we are very transparent in who a "Team Member" is and who isn't. One of the 4 people on the registration page can tell you what is new with Crafty, our testing methods, any changes we made for the tournament... pretty much anything. We are _all_ involved in the discussion and changes. We test probably like no one else due to the clusters at UAB, and Robert's access.

Or would you rather talk to someone during the game that doesn't know if they are using the right hash size, or why the tablebases aren't working?

I am not against the rule, but I want to point out that the way you formulated it is so vague as to make the rule useless. I am not expecting the Crafty team to have any difficulty providing someone knowledgeable to operate the engine. But if you are there, and 30 other engines are operated by some "team member" popping up out of nowhere... My wife brings me coffee while I am working on the engine. Does that make here a team member, and could she operate my engine?

This really depends on the author to be quite truthful. If the author is known, then likely I wouldn't have an issue with it.

If Joe Blow enters with something close to the strength of Toga/Fruit, wouldn't you think there would be suspicions?

You bet I would!. But my doubts lean in the other direction. If the author of TSCP would come with a Fruit-calibre engine, woudn't you be suspicious?

New authors join all the time. They would never be shut out. Some when asked have even sent me their source code to verify they are legitimate entries.

I am not doing these events to harm people. In fact quite the opposite.

Well, having them send you source code is not an option according to the rules you wrote down. A fresh programmer with a 2000-Elo engine could not participate, just because he is not "recognized". He would perceive that as very unfair. While it is exactly people like that that should be encouraged (I think).

If someone is caught cheating, why would we want them back? Have I ever kicked out an engine? Suspended anyone? Nope, want to know why?

Because I ask for code with courtesy and respect. Authors have _never_ said no to me before, during, or after an event. If someone makes a claim, I investigate it. Even private engines have given me their code and the executable running in the event to check their games and what not.

I want as many engines playing in the tournament possible. I am not there to piss everyone off or make them never enter again. I try to be as fair as possible, sometimes to the point it makes others angry, but they always respect my decisions because I remain impartial.

"Caught cheating" is not the same thing as not wanting others to see your code. The rules you wrote give no guarantee as to who can examine the code and who not. Will they be sent to another chess programmer? Or even a competitor? I am pretty sure you would not get to see the Rybka or HIARCS code, if you asked for it. And I think it should be within their rights to withold it from you, without being banned "for life".

[Peter Skinner]

I am not against the rule, but I want to point out that the way you formulated it is so vague as to make the rule useless. I am not expecting the Crafty team to have any difficulty providing someone knowledgeable to operate the engine. But if you are there, and 30 other engines are operated by some "team member" popping up out of nowhere... My wife brings me coffee while I am working on the engine. Does that make here a team member, and could she operate my engine?

Does she know the inner workings of the engine? Can she answer questions pertaining to the way the book was made? The interface used?

You bet I would!. But my doubts lean in the other direction. If the author of TSCP would come with a Fruit-calibre engine, woudn't you be suspicious?

You opened the door to this one, so I am going to walk right through it...

Rybka in CCT6
Rybka in CCT8

Quite a difference in strength, wouldn't you say? When I called out the program as being a clone of Fruit back then, I was slammed from every corner of the planet.

While known in 2004, Rybka was hardly a strong program. In 2006 it was and still is pretty unbeatable.

Well, having them send you source code is not an option according to the rules you wrote down. A fresh programmer with a 2000-Elo engine could not participate, just because he is not "recognized". He would perceive that as very unfair. While it is exactly people like that that should be encouraged (I think).

I agree, this rule could use some tweaking.

"Caught cheating" is not the same thing as not wanting others to see your code. The rules you wrote give no guarantee as to who can examine the code and who not. Will they be sent to another chess programmer? Or even a competitor? I am pretty sure you would not get to see the Rybka or HIARCS code, if you asked for it. And I think it should be within their rights to withold it from you, without being banned "for life".

I would probably never challenge the code of an established program like Hiarcs or Rybka. They have been in development and known for a very long time.

If Norman Schmidt were to try and enter, his code would be challenged instantly.

Do you see what I mean?

Peter