BubbaTough wrote:
If you are calculating chance of draw and chance of win reasonably, it will automatically account for these issues. Imagine if, instead of tracking centipawns, you track % chance white win, % chance black win, and % chance draw. If you were to do that, a contempt factor that intelligently balances taking risks to avoid a draw vs. increasing chance of losing is not that hard.
Of course, the problem for most is that they try to blend win % and draw % into a single number and THEN do contempt, which causes all the problems you both are describing (in my opinion).
-Sam
The other issue is that we are applying this inside the evaluation function. Which means we only have static information to go on. Yet, by admission, our static evaluation only works well when the position is tactically quiet. I can produce enough games to accurately fit a curve to the static eval vs win/lose/draw results. But I am not sure that the endpoint scores are as reliable as those that get backed up thru the tree to the root, which is the score I would see...
Well, yes. Evaluating things statically can be tricky, particularly things like locked pawns. Of course, there are a number of issues that come up when NOT evaluating things statically as well. I don't see this as a contempt issue, as it is just as applicable for juding win chance in a traditional eval as anything draw related.
bob wrote:
I did some playing around with this a long while back, and there are some issues. Crafty used to have a "wider range" of draw scores. And in tuning on the cluster, what I found was that you can EASILY go too far and drop elo all over the place, because if you try too hard to avoid a draw, you can easily do so by losing instead.
In studying the problem a lot, I noticed one recurring theme that I did not have much luck in solving... Some positions are drawn because they are drawn. And to avoid the draw, your only choice is to accept a loss. Others are drawn only because you think the alternative is slightly worse, but there is still a lot of "play" in the position and if you are stronger than your opponent, avoiding the draw makes sense.
The problem is, recognizing the difference between the two cases is difficult, and if you just tune for max elo, you are tuning for whichever case is more common. And that is as much a function of opening books and opponents as anything else, so the common case for one opponent might be playable if you avoid the draw, while against another it might not. The ones that are particularly hard deal with locked up pawn structure where you make a pawn break to avoid the draw, but your opponent is better positioned to penetrate first and you just invited him in and gave him the game.
This can be very tricky.
I agree with you. It's understood that sometimes you will win and sometimes you will lose by using a contempt factor as it is nothing more than a calculated gamble. The whole point is that you are willing to accept an inferior position to avoid a draw because of your belief that you can overcome a small handicap and win. Your point is well taken that this gamble is not the same for different kinds of positions.
My thoughts on this is that the draw score or contempt is much more useful in the early part of the game. This is pretty obvious. If you are 200 ELO stronger you are NOT 200 ELO stronger if you start from a position with equal chances closer to the end of the game. The reason is that in the opening you have many more opportunities to assert your superiority but in the ending there are few moves left. The better program does not play better moves EVERY time it makes a move, just once in a while.
We talked about this on the go mailing list a long time ago and why the range of strengths in GO is wider than in chess. If you assign ELO ratings to go players and the average player is rated 1500 you would see players with ratings closer to 4000, not 3000 as in chess. The reason might be partly because the games are longer and this gives even a slightly stronger player more of an opportunity to assert his superiority. A chess analogy is that if you are 100 ELO stronger you have only about a 64% chance of winning the game, but if you were to play a 10 game match your chances of winning the match is MUCH higher than 64%. If the match is long enough your chances of winning approach certainty.
So for the same reason I think it's a mistake to use the same contempt factor in the ending that you would use in the middlegame. In the endgame, as you say, a draw is usually a draw but in the middlegame a draw by repetition can be giving up too soon when you are the heavy favorite to win (or conversely, missing an opportunity for a draw when you are the underdog.)
So one suggestion is to phase the contempt out with stage of game, either partially or fully.
I have worked on this idea off and on for a while. In Crafty, I tend to drag the score toward draw under certain circumstances, like opposite bishops as an example. But there is more to it. If you know that your position is pretty dire, and you have some choices, you could do (a) trade pawns but not pieces which makes the endgame that much less winnable; (b) lock pawns to make it hard for the opponent to create an endgame advantage. But deciding when to do those, as opposed to preventing them from happening, becomes a problem. I pretty difficult one. Every time I've had some "breakthrough idea" it was quickly relegated to "broken idea". Trying to assess "is this winning, or losing?" statically is a real challenge. That's why we need a q-search and extensions in fact. Yet we are applying the knowledge in the eval, so we sort of have to depend on it. And I have not yet found a way to make this happen...
I'm trying the idea of phasing out the contempt factor by stage and I'm having no success. So it may be my theory on this is incorrect.
Don wrote:
I'm trying the idea of phasing out the contempt factor by stage and I'm having no success. So it may be my theory on this is incorrect.
My advice: forget about contempt. It is a 'gamble' instead of playing _real_ chess. Contempt in computer chess is an double edged sword... It may give you a few elo points avoiding draws against much weaker opponents, but it can backfire really ugly when playing against equals or betters. I don't like the concept. Guess why Evaluator::contempt in Critter 1.2 is initialized to "false" and never changes?
Btw my unsuccessful (and 'turned-off') implementation of contempt is exactly like Houdini1.5 (with uci option 'contempt=1'). In H15 the 'contempt' is not just about scoring draws slightly negatively - it even encourages exchanging material (hoping that it outplays almost any engine in endgames).
Don wrote:
I'm trying the idea of phasing out the contempt factor by stage and I'm having no success. So it may be my theory on this is incorrect.
My advice: forget about contempt. It is a 'gamble' instead of playing _real_ chess. Contempt in computer chess is an double edged sword... It may give you a few elo points avoiding draws against much weaker opponents, but it can backfire really ugly when playing against equals or betters. I don't like the concept. Guess why Evaluator::contempt in Critter 1.2 is initialized to "false" and never changes?
Btw my unsuccessful (and 'turned-off') implementation of contempt is exactly like Houdini1.5 (with uci option 'contempt=1'). In H15 the 'contempt' is not just about scoring draws slightly negatively - it even encourages exchanging material (hoping that it outplays almost any engine in endgames).
That should be a normal enough issue. The usual rule is "when ahead, trade pieces but not pawns, when behind trade pawns but not pieces." If you think you are better than your opponent, you should trade pieces and win the endgame. As a natural occurrence, in fact.
Don wrote:
I'm trying the idea of phasing out the contempt factor by stage and I'm having no success. So it may be my theory on this is incorrect.
My advice: forget about contempt. It is a 'gamble' instead of playing _real_ chess. Contempt in computer chess is an double edged sword... It may give you a few elo points avoiding draws against much weaker opponents, but it can backfire really ugly when playing against equals or betters. I don't like the concept. Guess why Evaluator::contempt in Critter 1.2 is initialized to "false" and never changes?
Our default contempt is only 5 centipawns, very modest and it hurts by about 1 or 2 ELO in games against equal opponents, but since Komodo is a top program we assume it is playing down more than it plays equal or up. It hurts against your program, Houdini and Rybka 4 but only slightly.
However as an option it makes a difference when playing up or down. It's not just a few ELO points, it's something like 20 ELO points if you have it set wrong against a much weaker or strong opponent.
My tests suggest you need 1 cp of contempt for even 10 or 15 ELO.
Btw my unsuccessful (and 'turned-off') implementation of contempt is exactly like Houdini1.5 (with uci option 'contempt=1'). In H15 the 'contempt' is not just about scoring draws slightly negatively - it even encourages exchanging material (hoping that it outplays almost any engine in endgames).
The principle that every beginner knows is to trade pieces when winning but not pawns. So this is probably just another way of expressing contempt. I assume Houdini has a more correct rule and this is on top of that. Is that what you see?
I'm not sure that is a good rule but I think I understand the reasoning.
Don wrote:
I'm trying the idea of phasing out the contempt factor by stage and I'm having no success. So it may be my theory on this is incorrect.
My advice: forget about contempt. It is a 'gamble' instead of playing _real_ chess. Contempt in computer chess is an double edged sword... It may give you a few elo points avoiding draws against much weaker opponents, but it can backfire really ugly when playing against equals or betters. I don't like the concept. Guess why Evaluator::contempt in Critter 1.2 is initialized to "false" and never changes?
Our default contempt is only 5 centipawns, very modest and it hurts by about 1 or 2 ELO in games against equal opponents, but since Komodo is a top program we assume it is playing down more than it plays equal or up. It hurts against your program, Houdini and Rybka 4 but only slightly.
However as an option it makes a difference when playing up or down. It's not just a few ELO points, it's something like 20 ELO points if you have it set wrong against a much weaker or strong opponent.
My tests suggest you need 1 cp of contempt for even 10 or 15 ELO.
Btw my unsuccessful (and 'turned-off') implementation of contempt is exactly like Houdini1.5 (with uci option 'contempt=1'). In H15 the 'contempt' is not just about scoring draws slightly negatively - it even encourages exchanging material (hoping that it outplays almost any engine in endgames).
The principle that every beginner knows is to trade pieces when winning but not pawns. So this is probably just another way of expressing contempt. I assume Houdini has a more correct rule and this is on top of that. Is that what you see?
I'm not sure that is a good rule but I think I understand the reasoning.
I have run a pretty simple test on our cluster. Maybe 500,000 games total, 30K per "contempt value" trying numbers like -70, -50, ..., 0, ..., +70. 0 is certainly the best for me, I'll report the results when they finish. I then am going to modify my cluster test driver, which currently does not relay "rating" to the two programs when a match is started, so that it will. I will then see if tweaking the contempt against weaker opponents (negative) or stronger opponents (positive) makes any significant difference. Right now, 0 is best and going in either direction drops the Elo...
But it is a contempt factor applied to all opponents equally, no rating scale at all...
