Huge SF 11 test with 7 different Contempts

[pohl4711]

Huge experimental test-tournament (31500 games (!)) of Stockfish 11 with 7 different Contempts (-40, -24, -15, 0, +15, +24 (=default of SF 11), +40).
Thinking-time: 1'+1'', singlethread, 256 Hash, no ponder, no endgame-bases for engines (5 Syzygy for cutechess-cli). 5 human moves openings.

Code: Select all

     Program                 Elo    +    -   Games   Score   Av.Op.  Draws

   1 Stockfish 11 C=0      : 3558    4    4  9000    50.7 %   3553   79.3 %
   2 Stockfish 11 C=+15    : 3558    4    4  9000    50.6 %   3554   77.8 %
   3 Stockfish 11 C=-24    : 3555    4    4  9000    50.1 %   3554   82.7 %
   4 Stockfish 11 C=-15    : 3554    4    4  9000    50.0 %   3554   81.7 %
   5 Stockfish 11 C=+24    : 3554    4    4  9000    50.0 %   3554   76.2 %
   6 Stockfish 11 C=+40    : 3551    4    4  9000    49.5 %   3555   73.8 %
   7 Stockfish 11 C=-40    : 3549    4    4  9000    49.2 %   3555   84.5 %

Conclusions: Only the +40 and -40 Contempt results are somewhat weaker. All other Contempts are inside errorbar at the same level of strength.

https://www.sp-cc.de/experiments.htm

Games: https://www.sp-cc.de/files/sf11_contempt_experiment.zip

[Ovyron]

C+24 is still my favorite, not too passive, but not too insane.

[Hai]

But -15, -24 and -40 are much better against LC0.

[Nordlandia]

But -15, -24 and -40 are much better against LC0.

Conduct test with negative contempt.

[Jouni]

Isn't contempt used to win against weaker engines? Some 200-500 ELO weaker! Here we found bigger gains:

[Jouni]

Addition: Data that shows dependance of elo difference between SFdev of october 2018 and older versions of Stockfish depending on contempt value. Upper and lower bounds represent value with maximum error.

[CMCanavessi]

Try the same but using another engine as a rival, and we'll see if things change a bit or not. SF vs SF seems a bit... weird.

[mwyoung]

But -15, -24 and -40 are much better against LC0.

What contempt do you recommend? Right now the best Stockfish in no match for the best Lc0 T61. In my testing.

[Hai]

But -15, -24 and -40 are much better against LC0.

What contempt do you recommend? Right now the best Stockfish in no match for the best Lc0 T61. In my testing.

I haven't tested it enough but I think the best contempt for Stockfish vs LC0 is somewhere between -30 and -100.
Even if LC0 wins the match, the elo difference between Stockfish 11 and LC0 will be much much smaller.

It's good to see that T61 is so strong.
That means LC0 Sergio 30x384 and 40x512 will be much stronger after the new trained games.
Do we have some new of these two nets?
I'm only interested in analysis and long time control games.

[mwyoung]

But -15, -24 and -40 are much better against LC0.

What contempt do you recommend? Right now the best Stockfish in no match for the best Lc0 T61. In my testing.

I haven't tested it enough but I think the best contempt for Stockfish vs LC0 is somewhere between -30 and -100.
Even if LC0 wins the match, the elo difference between Stockfish 11 and LC0 will be much much smaller.

It's good to see that T61 is so strong.
That means LC0 Sergio 30x384 and 40x512 will be much stronger after the new trained games.
Do we have some new of these two nets?
I'm only interested in analysis and long time control games.

Yes, I have ALL the sergio nets and have been testing them. I will try -30 in my next broadcast match. Currently playing T61 vs Stockfish. Current score of this match...

DESKTOP-CORSAIR, Rapid 0min+30sec 0


1 Lc0 v0.23.2+git.c8d9095 +26 +3/=23/-1 53.70% 14.5/27
2 Stockfish 020220 64 POPCNT -26 +1/=23/-3 46.30% 12.5/27

Live score