"Contempt" in Komodo broken?

[lkaufman]

It looks to me like Bob Hyatt was right; it seems to depend on the opponent. Contempt in the 15 to 30 range helped quite a bit against SF2, but against Rybka 3 (which at this level is slightly weaker than SF2) Contempt does not seem to help at all. I'll have to try other opponents.

[bob]

It looks to me like Bob Hyatt was right; it seems to depend on the opponent. Contempt in the 15 to 30 range helped quite a bit against SF2, but against Rybka 3 (which at this level is slightly weaker than SF2) Contempt does not seem to help at all. I'll have to try other opponents.

I'm collecting data. What is REALLY pissing me off is that I have three programs that are maybe 200 Elo weaker than Crafty, and when tuning just the "weaker opponent side" I can make a change and do better against one, and worse against another, yet both have elo within 10 points of each other.

Tuning is therefore giving wrong results. Because it is the "average" value, and if different divisors give different results, you might discover that several work just as well, unless you pick optimal for a single opponent.

I'll post some results once everything runs. First I am going to show results against each program individually, changing just the divisor. Then I will post the average against all those opponents, changing the divisor.

I suppose this is perfectly logical when you think about it, as two 2400 programs are not equal, they just have the same rating. One might be better in endgames, one better in middle games. Tuning the contempt will influence the results in different ways depending on the opponent.

I am also beginning to think that perhaps the contempt should not be static for the entire game, either.

Fortunately this is not a huge Elo gain...

[Leto]

I've been testing some high/very low contempt settings in K9.2 for fun, here's what I have so far, all games with same time parameters as used in my CEGT Blitz testing and all matches are 50 games each:

K92 c500 1CPU scored vs
Critter 0.42: 49% (2% draw rate)
Rodent 0.17a: 75% (2% draw rate)
Rodent 1.7: 31% (10% draw rate)
Arasan 16: 53% (6% draw rate)

I estimate K9.2 c500 1CPU is around 2500 CEGT Blitz, almost 700 elo weaker than the default 15 contempt.

I then ran a match between K92 c-500 1CPU vs Arasan 16 to see what effect maximum negative contempt has. It scored 50% with a 60% draw rate.

I then wondered what would happen if I give the maximum negative contempt 12 cores vs maximum contempt with just 1 core, so I ran 50 games and the maximum negative contempt on 12 cores scored 72% with 44% draw rate.

I then was curious to see how strong the maximum contempt version can get when you add more cores so I gave it 12 cores and ran it against Critter 1.6 4CPU and it scored only 12.5% out of 28 games so I figure this setting is probably 300 elo weaker than Critter 1.6 4CPU so it's probably in the 2800 CEGT Blitz range but I'd need to test it against a 2800 engine and then against something weaker than that.

[mjlef]

It seems that the best "Contempt" against a 300 ELO points weaker engine is consistent with Contempt=0 or even with slightly _negative_ Contempt. What this stuff means? I suspected the time control being too short, but I played at longer 10''+0.1'' Contempt=0 compared to Contempt=30, and "0" won convincingly. Is Komodo 9.2 "Contempt" broken?

Heh Heh..... I knew there was something wrong with the Contempt parameter, but the Komodo Team refused to believe me.
I stand vindicated !

Well, that is not true. I asked for your data and you gave me nothing to p[rove you claim. I explained that we were still working on Contempt and making adjustments. And our best evidence is it helps against many weaker opponents and you can turn it off by setting it to 0. How is that "refused to believe me"? Complaining never motivates me. Providing data/evidence does. We always work with the best data we can get.

[bob]

Let's see, if these issues are very much time control dependent

What you say is actually true even from a Gamer's point of view.
I have found that for very STC/Blitz, Contempt 0 is always the best, irrespective of the Engine being used !
A Playchess pal of mine theorized that this is because at very short time controls, the Victor is the Engine that makes the least amount of mistakes (!), NOT the most aggressive/brilliant move finding Engine ! Contempt 0 facilitates this. It is only logical I think.
So, it is only in LTC games that increasing Contempt has a positive impact because the Engine has more time to find brilliant or game-winning moves.
So, obviously, it follows from this, that only LTC games really and truly show which is indeed the superior Engine, other factors like Opening book strength and Hardware being equal.
On another note, this is the main reason why I have always been against equating thousands of short time control games to a few hundred LTC games; a favorite theory of Engine developers.
In my opinion, the two are as different as chalk and cheese and anyone who thinks differently is merely deluding himself.
Would you consider ten thousand pieces of chalk equivalent to even one cheese cube ? Of course NOT !

I don't see how this can be true. Contempt simply sets a score for any position the program categorizes as a draw. Repetitions. 50 moves. Insufficient material. stalemates. recognized drawn endgames such as k + rp + wrong bishop. Etc. It doesn't say "play more drawish stuff". It does say "if you can force a repetition, this is the score you will get..."

You want to avoid draws against much weaker opponents... a draw vs a player 300 Elo below you is NOT a "0.0" score. You should have won handily.

[RJN]

Does contempt use much overhead, vs having it "off" at 0?

[lkaufman]

Let's see, if these issues are very much time control dependent

What you say is actually true even from a Gamer's point of view.
I have found that for very STC/Blitz, Contempt 0 is always the best, irrespective of the Engine being used !
A Playchess pal of mine theorized that this is because at very short time controls, the Victor is the Engine that makes the least amount of mistakes (!), NOT the most aggressive/brilliant move finding Engine ! Contempt 0 facilitates this. It is only logical I think.
So, it is only in LTC games that increasing Contempt has a positive impact because the Engine has more time to find brilliant or game-winning moves.
So, obviously, it follows from this, that only LTC games really and truly show which is indeed the superior Engine, other factors like Opening book strength and Hardware being equal.
On another note, this is the main reason why I have always been against equating thousands of short time control games to a few hundred LTC games; a favorite theory of Engine developers.
In my opinion, the two are as different as chalk and cheese and anyone who thinks differently is merely deluding himself.
Would you consider ten thousand pieces of chalk equivalent to even one cheese cube ? Of course NOT !

I don't see how this can be true. Contempt simply sets a score for any position the program categorizes as a draw. Repetitions. 50 moves. Insufficient material. stalemates. recognized drawn endgames such as k + rp + wrong bishop. Etc. It doesn't say "play more drawish stuff". It does say "if you can force a repetition, this is the score you will get..."

You want to avoid draws against much weaker opponents... a draw vs a player 300 Elo below you is NOT a "0.0" score. You should have won handily.

Bob, you are describing contempt in Crafty. In Komodo it does much more than this, mostly trying to avoid even exchanges. My conclusions so far are that against SF2 around 24 is best, against Rybka 3 zero seems best, and against Houdini 1.5 around 5 seems best. So on average against those three opponents a value of 10 looks best. Since at the level tested they averaged more than 250 elo below Komodo, this suggests a divisor of 25 instead of ten in Kai's formula. But for now I'll recommend using 20 since we do have some prior data suggesting a lower divisor. Probably the bottom line is that we should have set default value to 10 rather than 15 to optimize against opponents 200 elo lower.

[Laskos]

Bob, you are describing contempt in Crafty. In Komodo it does much more than this, mostly trying to avoid even exchanges. My conclusions so far are that against SF2 around 24 is best, against Rybka 3 zero seems best, and against Houdini 1.5 around 5 seems best. So on average against those three opponents a value of 10 looks best. Since at the level tested they averaged more than 250 elo below Komodo, this suggests a divisor of 25 instead of ten in Kai's formula. But for now I'll recommend using 20 since we do have some prior data suggesting a lower divisor. Probably the bottom line is that we should have set default value to 10 rather than 15 to optimize against opponents 200 elo lower.

I tested at 5''+0.1'' comparatively Komodo 9.2 and Komodo 9.1 to see the impact of the contempt and the empiric optimal value for it.

Komodo 9.2 "Contempt"

Code: Select all

Rank Name                          ELO   Games   Score   Draws
   1 Komodo 30                     394    4000     91%     10%
   2 Komodo 60                     390    4000     90%      9%
   3 Komodo 0                      382    4000     90%     12%
   4 Komodo -30                    355    4000     89%     14%
   5 Stockfish                    -380   16000     10%     11%
Finished match

Rank Name                          ELO   Games   Score   Draws
   1 Komodo -30                    363    4000     89%     11%
   2 Komodo 30                     362    4000     89%      9%
   3 Komodo 0                      360    4000     89%      9%
   4 Komodo 60                     350    4000     88%      8%
   5 Hakkapeliitta                -359   16000     11%      9%
Finished match

Rank Name                          ELO   Games   Score   Draws
   1 Komodo 30                     324    4000     87%     12%
   2 Komodo 0                      318    4000     86%     12%
   3 Komodo 60                     318    4000     86%     11%
   4 Komodo -30                    313    4000     86%     15%
   5 Gaviota                      -318   16000     14%     13%
Finished match

The maximum gains compared to Contempt=0 are 12, 3, 6 ELO points, on average 7 ELO points gain. The average optimal "Contempt" is 23 against engines on average 350 ELO points weaker, therefore the optimal "Contempt" is around (ELO difference) / 15.

Komodo 9.1 "Drawscore"

Code: Select all

Rank Name                          ELO   Games   Score   Draws
   1 Komodo 60                     398    4000     91%      9%
   2 Komodo 90                     392    4000     91%      9%
   3 Komodo 30                     387    4000     90%     10%
   4 Komodo 0                      382    4000     90%     11%
   5 Stockfish                    -390   16000     10%     10%
Finished match

Rank Name                          ELO   Games   Score   Draws
   1 Komodo 60                     377    4000     90%      8%
   2 Komodo 30                     374    4000     90%      8%
   3 Komodo 0                      364    4000     89%      9%
   4 Komodo 90                     359    4000     89%      7%
   5 Hakkapeliitta                -368   16000     11%      8%
Finished match

Rank Name                          ELO   Games   Score   Draws
   1 Komodo 30                     316    4000     86%     12%
   2 Komodo 60                     308    4000     85%     11%
   3 Komodo 90                     304    4000     85%     11%
   4 Komodo 0                      304    4000     85%     13%
   5 Gaviota                      -308   16000     15%     12%
Finished match

The maximum gains compared to Contempt=0 are 16, 13, 12 ELO points, on average 14 ELO points gain. The average optimal "Drawscore" is minus 45 against engines on average 350 ELO points weaker, therefore the optimal "Drawscore" is around (- ELO difference) / 8.

At least against these 3 engines, "Contempt" seems a bit more erratic compared to "Drawscore". But this on only 3 opponents, and we saw that each opponent is different, and there is a dependency on time control too.

[bob]

Let's see, if these issues are very much time control dependent

What you say is actually true even from a Gamer's point of view.
I have found that for very STC/Blitz, Contempt 0 is always the best, irrespective of the Engine being used !
A Playchess pal of mine theorized that this is because at very short time controls, the Victor is the Engine that makes the least amount of mistakes (!), NOT the most aggressive/brilliant move finding Engine ! Contempt 0 facilitates this. It is only logical I think.
So, it is only in LTC games that increasing Contempt has a positive impact because the Engine has more time to find brilliant or game-winning moves.
So, obviously, it follows from this, that only LTC games really and truly show which is indeed the superior Engine, other factors like Opening book strength and Hardware being equal.
On another note, this is the main reason why I have always been against equating thousands of short time control games to a few hundred LTC games; a favorite theory of Engine developers.
In my opinion, the two are as different as chalk and cheese and anyone who thinks differently is merely deluding himself.
Would you consider ten thousand pieces of chalk equivalent to even one cheese cube ? Of course NOT !

I don't see how this can be true. Contempt simply sets a score for any position the program categorizes as a draw. Repetitions. 50 moves. Insufficient material. stalemates. recognized drawn endgames such as k + rp + wrong bishop. Etc. It doesn't say "play more drawish stuff". It does say "if you can force a repetition, this is the score you will get..."

You want to avoid draws against much weaker opponents... a draw vs a player 300 Elo below you is NOT a "0.0" score. You should have won handily.

Bob, you are describing contempt in Crafty. In Komodo it does much more than this, mostly trying to avoid even exchanges. My conclusions so far are that against SF2 around 24 is best, against Rybka 3 zero seems best, and against Houdini 1.5 around 5 seems best. So on average against those three opponents a value of 10 looks best. Since at the level tested they averaged more than 250 elo below Komodo, this suggests a divisor of 25 instead of ten in Kai's formula. But for now I'll recommend using 20 since we do have some prior data suggesting a lower divisor. Probably the bottom line is that we should have set default value to 10 rather than 15 to optimize against opponents 200 elo lower.

I understood that. But remember, the poster here was talking about older versions of Komodo being broken also...

[mjlef]

No, it will take the same amount of time in the evaluation.