MikeB wrote: They got lucky - they submitted an untested beta version and were getting burnt. Hopefully next time, wiser decisions are made as to which version to submit.
No we didn't get lucky. I really don't see where the lucky is.
For the record the decision was taken unilaterally by tournament managers (we were not involved in this specific decision but we have been informed afterwards as everybody else). Of course the tournament is theirs and they can do what they want.
At the moment we are discussing internally if and what to reply as official answer.
I post here as my personal comment, not as SF speaker.
The "lucky" is getting a chance to fix an obvious bug, in a situation where others have not had the opportunity. Lucky is managing to avoid significant additional losses due to a bug that slipped in due to inadequate testing. How many of us have done this (inadequate testing bugs) in the past? I certainly have. Usually it is a "fix it and wait until next year."
Personally, I have never played in an event that did not allow between-round changes. And just as personally, I have never changed code between rounds either as that is a prime way to let unexpected errors slip in. We used to jokingly call that the classic beginner's mistake. Saw a few that were trying to replace programs in the middle of a round in HGM's last tournament this past saturday...
bob wrote: The "lucky" is getting a chance to fix an obvious bug
Strictly speaking we didn't get the chance to fix any bug although that was possible simply changing an UCI parameter default value (it is a time management parameter): we have asked TCEC to set a different UCI parameter value before the game, but this was not accepted, and I think tournament managers have for sure a point in not accepting it...but then unilateral reverting to a version _they_ choose is does not seem to me a natural consequence.
I have a very personal opinion on this topic, but I prefer do not disclose it because a possible official SF answer is still under discussion and if it is decided to publish it, I will stick to it, no matter what my personal opinion is.
Regarding other authors accepting to not force SF to continue with the submitted version, I can only join Mark in saying that this is a noble gesture from them.
I must admit I was surprised to read how Martin decidedly rejected the idea of just increasing this lag parameter. It seemed the natural thing to do (and in fact not setting it to a safer value was a mistake in the first place). I guess some confusion arose because of it being suggested that this was due nonfunctioning of the easy-move code, accidentally broken by introduction of the lazy-smp patch. This caused everyone to suddenly blame the lazy smp. While the easy-move bug really did not cause the time loss at all, but was just responsible for putting Stockfish in a situation where its lag-resistance would be more heavily tested. (And of course thinking long in easy-move positions, giving the opponent free ponder time on a guaranteed ponder hit would be somewhat hurtful in itself.)
If it would have been clear to everyone that the problem was that the TCEC hardware has slightly more lag than the hardware on which Stockfish was tested, and due to an oversight the recommended configuration of Stockfish had not incorporated a safety margin for this unknown factor, I cannot imagine the TCEC organizers would have ruled this way. The natural solution would have been to just increase the value of the lag parameter, and let the current version of Stockfish (with all its easy-move flaws) play out this stage.
I also don't understand why the organizers make life difficult for themselves by first claiming this really transgresses the rules, and then have everyone vote about it, while 'play-limiting' is such a vague concept that they basically could have taken it to mean anything they wanted it to mean, and acted accordingly without this circus.
bob wrote: The "lucky" is getting a chance to fix an obvious bug
Strictly speaking we didn't get the chance to fix any bug although that was possible simply changing an UCI parameter default value (it is a time management parameter): we have asked TCEC to set a different UCI parameter value before the game, but this was not accepted, and I think tournament managers have for sure a point in not accepting it...but then unilateral reverting to a version _they_ choose is does not seem to me a natural consequence.
I have a very personal opinion on this topic, but I prefer do not disclose it because a possible official SF answer is still under discussion and if it is decided to publish it, I will stick to it, no matter what my personal opinion is.
Regarding other authors accepting to not force SF to continue with the submitted version, I can only join Mark in saying that this is a noble gesture from them.
Hi Marco!
Well, i can understand Martin's opinion in this case. The Imagination they would allow to modify UCI settings during a tourney or every game, creates more problems with the rules.
Probably they want strict UCI options to basic search things. But where starts a "search" related option and where ends a "search" related option? I think the engine developers could become very creative in that moment.
So, i think the best way is how it is. No UCI option is allowed to change during a running stage.
The worse case would be if developers and teams start to send UCI option requests for every single game and opponent to tune the engine on his best versus this special opponent.
bob wrote: The "lucky" is getting a chance to fix an obvious bug
Strictly speaking we didn't get the chance to fix any bug although that was possible simply changing an UCI parameter default value (it is a time management parameter): we have asked TCEC to set a different UCI parameter value before the game, but this was not accepted, and I think tournament managers have for sure a point in not accepting it...but then unilateral reverting to a version _they_ choose is does not seem to me a natural consequence.
I have a very personal opinion on this topic, but I prefer do not disclose it because a possible official SF answer is still under discussion and if it is decided to publish it, I will stick to it, no matter what my personal opinion is.
Regarding other authors accepting to not force SF to continue with the submitted version, I can only join Mark in saying that this is a noble gesture from them.
Hi Marco!
Well, i can understand Martin's opinion in this case. The Imagination they would allow to modify UCI settings during a tourney or every game, creates more problems with the rules.
Probably they want strict UCI options to basic search things. But where starts a "search" related option and where ends a "search" related option? I think the engine developers could become very creative in that moment.
So, i think the best way is how it is. No UCI option is allowed to change during a running stage.
The worse case would be if developers and teams start to send UCI option requests for every single game and opponent to tune the engine on his best versus this special opponent.
Regards
Daniel
I have written exactly the same thing, sorry if it was not clear.
I wonder if the SF time management bug would have been discovered if not for this particular tournament's hardware. And assuming it would be eventually discovered, how long would it have taken?
Daniel Mehrmann wrote:The worse case would be if developers and teams start to send UCI option requests for every single game and opponent to tune the engine on his best versus this special opponent.
Right on! It would start too look way too much like a Chess contest in that case. Imagine that you would know who you are playing against, and could adapt your strategy (like contempt an such) accordingly... An appalling thought!
Of course when you would submit a WB engine, you could automate all this. Or would they have taken special measure to hide the opponent name from the players, by having the GUI reject the 'name' feature?
Norm Pollock wrote:I wonder if the SF time management bug would have been discovered if not for this particular tournament's hardware. And assuming it would be eventually discovered, how long would it have taken?
Indeed, that was also what I wondered about. It is not like this version of Stockfish was never tested at all under SMP conditions, and it really seems related to the TCEC conditions. (Although of course it might not have been tested at 'critical' time controls, where you are always close to being flagged. In sudden-death games you would never see this,and in classical 40 moves/something you only get one opportnity in 40 moves to forfeit.)
So it seems a bit unfair to blame the Stockfish team for inadequate testing. The main thing they are to blame for is taking a safety margin of only 10 msec where taking even 100 sec would not have hurt them in any significant way.
Norm Pollock wrote:I wonder if the SF time management bug would have been discovered if not for this particular tournament's hardware. And assuming it would be eventually discovered, how long would it have taken?
I wonder if the SF team will now fix this bug without simulating it first.
