"Contempt" in Komodo broken?
Moderators: hgm, Rebel, chrisw
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: "Contempt" in Komodo broken?
It looks to me like Bob Hyatt was right; it seems to depend on the opponent. Contempt in the 15 to 30 range helped quite a bit against SF2, but against Rybka 3 (which at this level is slightly weaker than SF2) Contempt does not seem to help at all. I'll have to try other opponents.
Komodo rules!
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: "Contempt" in Komodo broken?
I'm collecting data. What is REALLY pissing me off is that I have three programs that are maybe 200 Elo weaker than Crafty, and when tuning just the "weaker opponent side" I can make a change and do better against one, and worse against another, yet both have elo within 10 points of each other.lkaufman wrote:It looks to me like Bob Hyatt was right; it seems to depend on the opponent. Contempt in the 15 to 30 range helped quite a bit against SF2, but against Rybka 3 (which at this level is slightly weaker than SF2) Contempt does not seem to help at all. I'll have to try other opponents.
Tuning is therefore giving wrong results. Because it is the "average" value, and if different divisors give different results, you might discover that several work just as well, unless you pick optimal for a single opponent.
I'll post some results once everything runs. First I am going to show results against each program individually, changing just the divisor. Then I will post the average against all those opponents, changing the divisor.
I suppose this is perfectly logical when you think about it, as two 2400 programs are not equal, they just have the same rating. One might be better in endgames, one better in middle games. Tuning the contempt will influence the results in different ways depending on the opponent.
I am also beginning to think that perhaps the contempt should not be static for the entire game, either.
Fortunately this is not a huge Elo gain...
-
- Posts: 2071
- Joined: Thu May 04, 2006 3:40 am
- Location: Dune
Re: "Contempt" in Komodo broken?
I've been testing some high/very low contempt settings in K9.2 for fun, here's what I have so far, all games with same time parameters as used in my CEGT Blitz testing and all matches are 50 games each:
K92 c500 1CPU scored vs
Critter 0.42: 49% (2% draw rate)
Rodent 0.17a: 75% (2% draw rate)
Rodent 1.7: 31% (10% draw rate)
Arasan 16: 53% (6% draw rate)
I estimate K9.2 c500 1CPU is around 2500 CEGT Blitz, almost 700 elo weaker than the default 15 contempt.
I then ran a match between K92 c-500 1CPU vs Arasan 16 to see what effect maximum negative contempt has. It scored 50% with a 60% draw rate.
I then wondered what would happen if I give the maximum negative contempt 12 cores vs maximum contempt with just 1 core, so I ran 50 games and the maximum negative contempt on 12 cores scored 72% with 44% draw rate.
I then was curious to see how strong the maximum contempt version can get when you add more cores so I gave it 12 cores and ran it against Critter 1.6 4CPU and it scored only 12.5% out of 28 games so I figure this setting is probably 300 elo weaker than Critter 1.6 4CPU so it's probably in the 2800 CEGT Blitz range but I'd need to test it against a 2800 engine and then against something weaker than that.
K92 c500 1CPU scored vs
Critter 0.42: 49% (2% draw rate)
Rodent 0.17a: 75% (2% draw rate)
Rodent 1.7: 31% (10% draw rate)
Arasan 16: 53% (6% draw rate)
I estimate K9.2 c500 1CPU is around 2500 CEGT Blitz, almost 700 elo weaker than the default 15 contempt.
I then ran a match between K92 c-500 1CPU vs Arasan 16 to see what effect maximum negative contempt has. It scored 50% with a 60% draw rate.
I then wondered what would happen if I give the maximum negative contempt 12 cores vs maximum contempt with just 1 core, so I ran 50 games and the maximum negative contempt on 12 cores scored 72% with 44% draw rate.
I then was curious to see how strong the maximum contempt version can get when you add more cores so I gave it 12 cores and ran it against Critter 1.6 4CPU and it scored only 12.5% out of 28 games so I figure this setting is probably 300 elo weaker than Critter 1.6 4CPU so it's probably in the 2800 CEGT Blitz range but I'd need to test it against a 2800 engine and then against something weaker than that.
-
- Posts: 1494
- Joined: Thu Mar 30, 2006 2:08 pm
Re: "Contempt" in Komodo broken?
Well, that is not true. I asked for your data and you gave me nothing to p[rove you claim. I explained that we were still working on Contempt and making adjustments. And our best evidence is it helps against many weaker opponents and you can turn it off by setting it to 0. How is that "refused to believe me"? Complaining never motivates me. Providing data/evidence does. We always work with the best data we can get.shrapnel wrote:Heh Heh..... I knew there was something wrong with the Contempt parameter, but the Komodo Team refused to believe me.Laskos wrote:It seems that the best "Contempt" against a 300 ELO points weaker engine is consistent with Contempt=0 or even with slightly _negative_ Contempt. What this stuff means? I suspected the time control being too short, but I played at longer 10''+0.1'' Contempt=0 compared to Contempt=30, and "0" won convincingly. Is Komodo 9.2 "Contempt" broken?
I stand vindicated !
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: "Contempt" in Komodo broken?
I don't see how this can be true. Contempt simply sets a score for any position the program categorizes as a draw. Repetitions. 50 moves. Insufficient material. stalemates. recognized drawn endgames such as k + rp + wrong bishop. Etc. It doesn't say "play more drawish stuff". It does say "if you can force a repetition, this is the score you will get..."shrapnel wrote:What you say is actually true even from a Gamer's point of view.Laskos wrote:Let's see, if these issues are very much time control dependent
I have found that for very STC/Blitz, Contempt 0 is always the best, irrespective of the Engine being used !
A Playchess pal of mine theorized that this is because at very short time controls, the Victor is the Engine that makes the least amount of mistakes (!), NOT the most aggressive/brilliant move finding Engine ! Contempt 0 facilitates this. It is only logical I think.
So, it is only in LTC games that increasing Contempt has a positive impact because the Engine has more time to find brilliant or game-winning moves.
So, obviously, it follows from this, that only LTC games really and truly show which is indeed the superior Engine, other factors like Opening book strength and Hardware being equal.
On another note, this is the main reason why I have always been against equating thousands of short time control games to a few hundred LTC games; a favorite theory of Engine developers.
In my opinion, the two are as different as chalk and cheese and anyone who thinks differently is merely deluding himself.
Would you consider ten thousand pieces of chalk equivalent to even one cheese cube ? Of course NOT !
You want to avoid draws against much weaker opponents... a draw vs a player 300 Elo below you is NOT a "0.0" score. You should have won handily.
-
- Posts: 303
- Joined: Fri Jun 21, 2013 5:18 am
- Location: Orion Spiral Arm
Re: "Contempt" in Komodo broken?
Does contempt use much overhead, vs having it "off" at 0?
i7-5930K @4.5GHz, H100i Hydro Cooler, 64GB DDR4 Corsair Dominator Platinum @3000MHz, ASUS X99 Deluxe mboard, 1TB EVO 850 SSD
-
- Posts: 5960
- Joined: Sun Jan 10, 2010 6:15 am
- Location: Maryland USA
Re: "Contempt" in Komodo broken?
Bob, you are describing contempt in Crafty. In Komodo it does much more than this, mostly trying to avoid even exchanges. My conclusions so far are that against SF2 around 24 is best, against Rybka 3 zero seems best, and against Houdini 1.5 around 5 seems best. So on average against those three opponents a value of 10 looks best. Since at the level tested they averaged more than 250 elo below Komodo, this suggests a divisor of 25 instead of ten in Kai's formula. But for now I'll recommend using 20 since we do have some prior data suggesting a lower divisor. Probably the bottom line is that we should have set default value to 10 rather than 15 to optimize against opponents 200 elo lower.bob wrote:I don't see how this can be true. Contempt simply sets a score for any position the program categorizes as a draw. Repetitions. 50 moves. Insufficient material. stalemates. recognized drawn endgames such as k + rp + wrong bishop. Etc. It doesn't say "play more drawish stuff". It does say "if you can force a repetition, this is the score you will get..."shrapnel wrote:What you say is actually true even from a Gamer's point of view.Laskos wrote:Let's see, if these issues are very much time control dependent
I have found that for very STC/Blitz, Contempt 0 is always the best, irrespective of the Engine being used !
A Playchess pal of mine theorized that this is because at very short time controls, the Victor is the Engine that makes the least amount of mistakes (!), NOT the most aggressive/brilliant move finding Engine ! Contempt 0 facilitates this. It is only logical I think.
So, it is only in LTC games that increasing Contempt has a positive impact because the Engine has more time to find brilliant or game-winning moves.
So, obviously, it follows from this, that only LTC games really and truly show which is indeed the superior Engine, other factors like Opening book strength and Hardware being equal.
On another note, this is the main reason why I have always been against equating thousands of short time control games to a few hundred LTC games; a favorite theory of Engine developers.
In my opinion, the two are as different as chalk and cheese and anyone who thinks differently is merely deluding himself.
Would you consider ten thousand pieces of chalk equivalent to even one cheese cube ? Of course NOT !
You want to avoid draws against much weaker opponents... a draw vs a player 300 Elo below you is NOT a "0.0" score. You should have won handily.
Komodo rules!
-
- Posts: 10948
- Joined: Wed Jul 26, 2006 10:21 pm
- Full name: Kai Laskos
Re: "Contempt" in Komodo broken?
I tested at 5''+0.1'' comparatively Komodo 9.2 and Komodo 9.1 to see the impact of the contempt and the empiric optimal value for it.lkaufman wrote: Bob, you are describing contempt in Crafty. In Komodo it does much more than this, mostly trying to avoid even exchanges. My conclusions so far are that against SF2 around 24 is best, against Rybka 3 zero seems best, and against Houdini 1.5 around 5 seems best. So on average against those three opponents a value of 10 looks best. Since at the level tested they averaged more than 250 elo below Komodo, this suggests a divisor of 25 instead of ten in Kai's formula. But for now I'll recommend using 20 since we do have some prior data suggesting a lower divisor. Probably the bottom line is that we should have set default value to 10 rather than 15 to optimize against opponents 200 elo lower.
Komodo 9.2 "Contempt"
Code: Select all
Rank Name ELO Games Score Draws
1 Komodo 30 394 4000 91% 10%
2 Komodo 60 390 4000 90% 9%
3 Komodo 0 382 4000 90% 12%
4 Komodo -30 355 4000 89% 14%
5 Stockfish -380 16000 10% 11%
Finished match
Rank Name ELO Games Score Draws
1 Komodo -30 363 4000 89% 11%
2 Komodo 30 362 4000 89% 9%
3 Komodo 0 360 4000 89% 9%
4 Komodo 60 350 4000 88% 8%
5 Hakkapeliitta -359 16000 11% 9%
Finished match
Rank Name ELO Games Score Draws
1 Komodo 30 324 4000 87% 12%
2 Komodo 0 318 4000 86% 12%
3 Komodo 60 318 4000 86% 11%
4 Komodo -30 313 4000 86% 15%
5 Gaviota -318 16000 14% 13%
Finished match
Komodo 9.1 "Drawscore"
Code: Select all
Rank Name ELO Games Score Draws
1 Komodo 60 398 4000 91% 9%
2 Komodo 90 392 4000 91% 9%
3 Komodo 30 387 4000 90% 10%
4 Komodo 0 382 4000 90% 11%
5 Stockfish -390 16000 10% 10%
Finished match
Rank Name ELO Games Score Draws
1 Komodo 60 377 4000 90% 8%
2 Komodo 30 374 4000 90% 8%
3 Komodo 0 364 4000 89% 9%
4 Komodo 90 359 4000 89% 7%
5 Hakkapeliitta -368 16000 11% 8%
Finished match
Rank Name ELO Games Score Draws
1 Komodo 30 316 4000 86% 12%
2 Komodo 60 308 4000 85% 11%
3 Komodo 90 304 4000 85% 11%
4 Komodo 0 304 4000 85% 13%
5 Gaviota -308 16000 15% 12%
Finished match
At least against these 3 engines, "Contempt" seems a bit more erratic compared to "Drawscore". But this on only 3 opponents, and we saw that each opponent is different, and there is a dependency on time control too.
-
- Posts: 20943
- Joined: Mon Feb 27, 2006 7:30 pm
- Location: Birmingham, AL
Re: "Contempt" in Komodo broken?
I understood that. But remember, the poster here was talking about older versions of Komodo being broken also...lkaufman wrote:Bob, you are describing contempt in Crafty. In Komodo it does much more than this, mostly trying to avoid even exchanges. My conclusions so far are that against SF2 around 24 is best, against Rybka 3 zero seems best, and against Houdini 1.5 around 5 seems best. So on average against those three opponents a value of 10 looks best. Since at the level tested they averaged more than 250 elo below Komodo, this suggests a divisor of 25 instead of ten in Kai's formula. But for now I'll recommend using 20 since we do have some prior data suggesting a lower divisor. Probably the bottom line is that we should have set default value to 10 rather than 15 to optimize against opponents 200 elo lower.bob wrote:I don't see how this can be true. Contempt simply sets a score for any position the program categorizes as a draw. Repetitions. 50 moves. Insufficient material. stalemates. recognized drawn endgames such as k + rp + wrong bishop. Etc. It doesn't say "play more drawish stuff". It does say "if you can force a repetition, this is the score you will get..."shrapnel wrote:What you say is actually true even from a Gamer's point of view.Laskos wrote:Let's see, if these issues are very much time control dependent
I have found that for very STC/Blitz, Contempt 0 is always the best, irrespective of the Engine being used !
A Playchess pal of mine theorized that this is because at very short time controls, the Victor is the Engine that makes the least amount of mistakes (!), NOT the most aggressive/brilliant move finding Engine ! Contempt 0 facilitates this. It is only logical I think.
So, it is only in LTC games that increasing Contempt has a positive impact because the Engine has more time to find brilliant or game-winning moves.
So, obviously, it follows from this, that only LTC games really and truly show which is indeed the superior Engine, other factors like Opening book strength and Hardware being equal.
On another note, this is the main reason why I have always been against equating thousands of short time control games to a few hundred LTC games; a favorite theory of Engine developers.
In my opinion, the two are as different as chalk and cheese and anyone who thinks differently is merely deluding himself.
Would you consider ten thousand pieces of chalk equivalent to even one cheese cube ? Of course NOT !
You want to avoid draws against much weaker opponents... a draw vs a player 300 Elo below you is NOT a "0.0" score. You should have won handily.
-
- Posts: 1494
- Joined: Thu Mar 30, 2006 2:08 pm
Re: "Contempt" in Komodo broken?
No, it will take the same amount of time in the evaluation.