Stockfish Handicap Matches

[lkaufman]

Finished the elo 2900 pool.

Stockfish gauntlet, knight-odds, tc=40/10

Code: Select all

   # ENGINE          : RATING    POINTS  PLAYED    (%)
   1 cheng4_4.39     : 3273.3     128.0     200   64.0%
   2 Bobcat_8        : 3250.9     122.0     200   61.0%
   3 Stockfish_11    : 3172.5     269.5     600   44.9%
   4 Crafty_25.6     : 3103.3      80.5     200   40.3%

tc=40/20

Code: Select all

   # ENGINE          : RATING    POINTS  PLAYED    (%)
   1 cheng4_4.39     : 3315.9     137.5     200   68.8%
   2 Bobcat_8        : 3294.0     132.0     200   66.0%
   3 Stockfish_11    : 3177.8     274.5     600   45.8%
   4 Crafty_25.6     : 3012.3      56.0     200   28.0%

tc=40/40

Code: Select all

   # ENGINE          : RATING    POINTS  PLAYED    (%)
   1 cheng4_4.39     : 3318.8     146.0     200   73.0%
   2 Bobcat_8        : 3295.0     140.5     200   70.3%
   3 Stockfish_11    : 3144.5     242.0     600   40.3%
   4 Crafty_25.6     : 3041.7      71.5     200   35.8%

Next, 2800 pool.

So results improved steadily with more time as expected for cheng and bobcat, but not for crafty (between 40/10 and 40/20 regression); I wonder why? Two questions: How were the positions used chosen from the ChrisW set? I'm finding that taking them from the middle (pruning equal number from each end) is the fairest and closest simulation to real knight odds.

We can’t go cherry picking positions according to subjective criteria. And this concept of “real knight odds” is about as subjective as it gets, and it isn’t reached by asking an engine to evaluate at the root and using that as the definition. Imagine defining “real chess odds” by asking an engine to search from the root and give the answer. 42?
There are no “real knight odds”, all there is are positions without the knight and see how the results work out from *many* tests. We can try to use “natural” positions without either side having an apparent head start, eg remove the outliers.
Nor are we trying to determine what knights odds are in some numerical sense, we trying to determine how modern engines do against strong oldies with various handicaps, the first handicap being minus a knight.

Also, did Stockfish use default Contempt, or 0, or max (100)? It would do best with 100 I'm sure.

It’s better to just use defaults, too much parameter fiddling around just confuses everything.

Anyway, I prepared suites of 25, 100, 250 and 1000 epds. They are each a randomly selected subset of about 1200 epds taken from, I forget, it says in the github readme, roughly 370 to 420 I think. Probably that selection is actually in line with your desires, actually.

Posit from me: the most sensible course would be to use those sets only for a while, we’ll soon see if the 25 suite gives very different results from the 1000 suite, and then we can start worrying if small subsets and the positions in general are too noisy. For example, we don’t know right now if the anomalous(?) results of Crafty are down to unlucky position selection.

First result using your database. We took your 5000 knight odds set, which you had already pruned to 3870 positions, and removed 1435 positions from each end, producing a list of 1000 positions exactly in the center of your list, and put it in our tester. I hope you will agree that this is fair and unbiased. The score range was -4.30 to -4.11. quite narrow, and just by chance the worst score was the same score I got from the root position at 10 seconds for both positions. For the first test, I just had Komodo 14 play against itself at the very fast time control of 10 seconds + 0.1" increment, and the result was that White won one game, one draw, and Black won 998, so 1139 elo advantage. I'm sure that at a more normal time control the result would have been even more lopsided, probably just 100%. But the results of the tests between unrelated engines aren't showing a knight handicap to be worth a thousand elo. I suppose it's just a lot harder to give a handicap to someone who knows everything you know than it is to someone with very different skills.

You should be using the 25, 100, 250 or 1000 knight odds databases, depending of how many games in the gauntlet, posted to github last night. Then everybody is using the same base data.

Personally I am not interested in what an engine that would cost me a hundred euros to use does, so, again, because of free widespread access, it’s more interesting to stay with Stockfish (or LC0). Unsurprising that your program trounces itself when given knight odds.

You should, btw, know better than to ascribe 1000 Elo to a 99% result, let alone extrapolate from it. Elo scale is not able, nor meant, to deal with tail results of that nature.

I made the book from your data before I knew you were going to post subsets yourself. Regarding Elo, I know that the elo estimate for 99.85% is subject to a large margin of error, for multiple reasons, but the point was to show that your set of positions is completely winning for Black, as it should be, and that it is not easy to explain why 2750 rated engines only break even from these positions vs SF.
My main goal with this is to find an engine that will perform just as well taking knight odds as would a strong human player of the same Elo, so that we could reasonably predict results of engine vs GM handicap matches by simulation.

trying hard to decode this.
Failed.

You probably haven't followed all the Komodo vs. master/GM handicap matches, so to make a long story short, we know that at Rapid time controls (let's say 15' + 10" to standardize), Komodo (on 32 core machine) does well giving knight odds to players below about 2300 FIDE, poorly vs. players above that. If I say that Komodo performs about 2300 FIDE giving knight odds at this time control, I won't be far off from the truth. So I'd like to find an engine that would be equal with a 2300 FIDE player in standard chess at 15' + 10" which would also be even with Komodo at this TC at knight odds. It's obvious that the normal engines are hundreds of elos away from this, they need to be something like 2750 CCRL rapid, and even those ratings are unrealistically low compared to human FIDE ratings at Rapid. I'm looking for some engine below 2300 CCRL that can hold its own with Komodo (or Stockfish, doesn't matter that much) at knight odds. The closest I've come is the weakened Stockfish levels, but it looks like even they need to be much stronger than 2300 CCRL to have a chance at knight odds, although I don't really know what SF levels would be rated on that list at Rapid. I can determine this, but it will take a lot of time.

[lkaufman]

Stockfish gauntlet, knight-odds, ccrl elo pool <2800

tc=40/40 only.

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw 
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  
   1 Benjamin                       26      43     200   53.8%   19.5%
   2 ProDeo                         17      45     200   52.5%   14.0%
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  
   5 Ruffian_2                    -135      47     200   31.5%   16.0%   

Most of these engines don't have an exactly named copy in CCRL 40/15 (Benjamin and ProDeo have version numbers, two others aren't identical), but it looks like roughly 2750 on that list is the break-even point for SF 11 at that TC. I'll be curious to see if Komodo 14 can score as well against the same opponents. It would score much better with high Contempt, but so would Stockfish, so I guess it's fair enough to compare.

Have it running, takes about 2 hours.

Regarding elo's:

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw  CCRL
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  3537
   1 Benjamin                       26      43     200   53.8%   19.5%  2646
   2 ProDeo 2.2                     17      45     200   52.5%   14.0%  2770
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%  2783
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  2684
   5 Ruffian_2                    -135      47     200   31.5%   16.0%  2608

Interesting is that Benjamin while listed 124 elo less than ProDeo scores better. I think that much of this kind of testing has to do with the killer instinct of an engine. And Benjamin is the gambit version of ProDeo. Style decides?

I think I've figured out a big part of the mystery of why both Stockfish and Komodo can give knight odds to such strong engines in your tests. When I first read that you were testing at 40/10, I assumed that the 10 meant ten minutes, since we are constantly referring to CCRL tests at 40/15 meaning 15 minutes. But when you starting giving results at 40/40 and even 40/80, I realized that you must mean 40/x SECONDS, not minutes, since you are unlikely to have enough hardware to play so many games so quickly (correct me if I'm wrong here)! So even these 40/40 results are (if I'm right now) bullet games, not rapid games. At bullet chess even the top human GMs would have trouble beating Stockfish or Komodo at knight odds. So the question is, what level engine can Komodo and Stockfish give knight odds to in rapid chess (15' + 10" being the standard now)? Obviously, it will be a weaker engine than these 2700+ engines, but how much weaker? I do have one data point: at 3' + 2" (blitz, roughly midway between bullet and rapid) I got a +94 elo result for Komodo 14 vs. Arasan 14 64 bit, about 2640 on CCRL 40/15 list (est. based on versions just before and after), which is a 2734 performance. But I used Contempt 150, which helps a lot at knight odds; I'll have to redo the test without Contempt (or with the default of just 4). My best guess is that with default Contempt, Komodo 14 and SF need opponents in the mid 2600s in blitz, and in the mid 2500s in Rapid (15' + 10"). I can run Komodo vs. Arasan 14 overnight at Rapid with default Contempt; if I'm correct Komodo will lose but not too badly. I run 63 games at once, so the thousand games won't finish in 8 or 9 hours, but maybe in 15 hours or so.

[Laskos]

Stockfish gauntlet, knight-odds, ccrl elo pool <2800

tc=40/40 only.

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw 
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  
   1 Benjamin                       26      43     200   53.8%   19.5%
   2 ProDeo                         17      45     200   52.5%   14.0%
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  
   5 Ruffian_2                    -135      47     200   31.5%   16.0%   

Most of these engines don't have an exactly named copy in CCRL 40/15 (Benjamin and ProDeo have version numbers, two others aren't identical), but it looks like roughly 2750 on that list is the break-even point for SF 11 at that TC. I'll be curious to see if Komodo 14 can score as well against the same opponents. It would score much better with high Contempt, but so would Stockfish, so I guess it's fair enough to compare.

Have it running, takes about 2 hours.

Regarding elo's:

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw  CCRL
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  3537
   1 Benjamin                       26      43     200   53.8%   19.5%  2646
   2 ProDeo 2.2                     17      45     200   52.5%   14.0%  2770
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%  2783
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  2684
   5 Ruffian_2                    -135      47     200   31.5%   16.0%  2608

Interesting is that Benjamin while listed 124 elo less than ProDeo scores better. I think that much of this kind of testing has to do with the killer instinct of an engine. And Benjamin is the gambit version of ProDeo. Style decides?

I think I've figured out a big part of the mystery of why both Stockfish and Komodo can give knight odds to such strong engines in your tests. When I first read that you were testing at 40/10, I assumed that the 10 meant ten minutes, since we are constantly referring to CCRL tests at 40/15 meaning 15 minutes. But when you starting giving results at 40/40 and even 40/80, I realized that you must mean 40/x SECONDS, not minutes, since you are unlikely to have enough hardware to play so many games so quickly (correct me if I'm wrong here)! So even these 40/40 results are (if I'm right now) bullet games, not rapid games. At bullet chess even the top human GMs would have trouble beating Stockfish or Komodo at knight odds. So the question is, what level engine can Komodo and Stockfish give knight odds to in rapid chess (15' + 10" being the standard now)? Obviously, it will be a weaker engine than these 2700+ engines, but how much weaker? I do have one data point: at 3' + 2" (blitz, roughly midway between bullet and rapid) I got a +94 elo result for Komodo 14 vs. Arasan 14 64 bit, about 2640 on CCRL 40/15 list (est. based on versions just before and after), which is a 2734 performance. But I used Contempt 150, which helps a lot at knight odds; I'll have to redo the test without Contempt (or with the default of just 4). My best guess is that with default Contempt, Komodo 14 and SF need opponents in the mid 2600s in blitz, and in the mid 2500s in Rapid (15' + 10"). I can run Komodo vs. Arasan 14 overnight at Rapid with default Contempt; if I'm correct Komodo will lose but not too badly. I run 63 games at once, so the thousand games won't finish in 8 or 9 hours, but maybe in 15 hours or so.

We hyper-analyzed these issues several years ago. IIRC, at bullet engine-engine Knight odds are some 600 (logistic) engine Elo points, at rapid 45min + 15s some 1200 logistic engine ELO points, and by continuation at tournament TC maybe 1400 engine Elo points (the last one was never really tested). The handicap is heavily TC dependent, and with humans it could be even more dependent (humans are weak at bullet and blitz). I don't think that even a perfect engine can give Knight odds to Carlsen at tournament TC. Currently SF and Komodo in whatever configuration are no stronger than 2100 FIDE Elo points Knight odds at tournament TC against a human.

[Laskos]

Stockfish gauntlet, knight-odds, ccrl elo pool <2800

tc=40/40 only.

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw 
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  
   1 Benjamin                       26      43     200   53.8%   19.5%
   2 ProDeo                         17      45     200   52.5%   14.0%
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  
   5 Ruffian_2                    -135      47     200   31.5%   16.0%   

Most of these engines don't have an exactly named copy in CCRL 40/15 (Benjamin and ProDeo have version numbers, two others aren't identical), but it looks like roughly 2750 on that list is the break-even point for SF 11 at that TC. I'll be curious to see if Komodo 14 can score as well against the same opponents. It would score much better with high Contempt, but so would Stockfish, so I guess it's fair enough to compare.

Have it running, takes about 2 hours.

Regarding elo's:

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw  CCRL
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  3537
   1 Benjamin                       26      43     200   53.8%   19.5%  2646
   2 ProDeo 2.2                     17      45     200   52.5%   14.0%  2770
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%  2783
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  2684
   5 Ruffian_2                    -135      47     200   31.5%   16.0%  2608

Interesting is that Benjamin while listed 124 elo less than ProDeo scores better. I think that much of this kind of testing has to do with the killer instinct of an engine. And Benjamin is the gambit version of ProDeo. Style decides?

I think I've figured out a big part of the mystery of why both Stockfish and Komodo can give knight odds to such strong engines in your tests. When I first read that you were testing at 40/10, I assumed that the 10 meant ten minutes, since we are constantly referring to CCRL tests at 40/15 meaning 15 minutes. But when you starting giving results at 40/40 and even 40/80, I realized that you must mean 40/x SECONDS, not minutes, since you are unlikely to have enough hardware to play so many games so quickly (correct me if I'm wrong here)! So even these 40/40 results are (if I'm right now) bullet games, not rapid games. At bullet chess even the top human GMs would have trouble beating Stockfish or Komodo at knight odds. So the question is, what level engine can Komodo and Stockfish give knight odds to in rapid chess (15' + 10" being the standard now)? Obviously, it will be a weaker engine than these 2700+ engines, but how much weaker? I do have one data point: at 3' + 2" (blitz, roughly midway between bullet and rapid) I got a +94 elo result for Komodo 14 vs. Arasan 14 64 bit, about 2640 on CCRL 40/15 list (est. based on versions just before and after), which is a 2734 performance. But I used Contempt 150, which helps a lot at knight odds; I'll have to redo the test without Contempt (or with the default of just 4). My best guess is that with default Contempt, Komodo 14 and SF need opponents in the mid 2600s in blitz, and in the mid 2500s in Rapid (15' + 10"). I can run Komodo vs. Arasan 14 overnight at Rapid with default Contempt; if I'm correct Komodo will lose but not too badly. I run 63 games at once, so the thousand games won't finish in 8 or 9 hours, but maybe in 15 hours or so.

We hyper-analyzed these issues several years ago. IIRC, at bullet engine-engine Knight odds are some 600 (logistic) engine Elo points, at rapid 45min + 15s some 1200 logistic engine ELO points, and by continuation at tournament TC maybe 1400 engine Elo points (the last one was never really tested). The handicap is heavily TC dependent, and with humans it could be even more dependent (humans are weak at bullet and blitz). I don't think that even a perfect engine can give Knight odds to Carlsen at tournament TC. Currently SF and Komodo in whatever configuration are no stronger than 2100 FIDE Elo points Knight odds at tournament TC against a human.

My estimates for good "human" sparring at tournament TC is Lc0

11248 at 1000 nodes --- 2750 FIDE
11248 at 100 nodes --- 2450 FIDE
11248 at 10 nodes --- 2100 FIDE.

Again, tournament TC. At Blitz 5min + 3s add some 300 Elo points to those to get FIDE Blitz rating. All this is rough estimation, but it can be improved by testing.

[lkaufman]

Stockfish gauntlet, knight-odds, ccrl elo pool <2800

tc=40/40 only.

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw 
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  
   1 Benjamin                       26      43     200   53.8%   19.5%
   2 ProDeo                         17      45     200   52.5%   14.0%
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  
   5 Ruffian_2                    -135      47     200   31.5%   16.0%   

Most of these engines don't have an exactly named copy in CCRL 40/15 (Benjamin and ProDeo have version numbers, two others aren't identical), but it looks like roughly 2750 on that list is the break-even point for SF 11 at that TC. I'll be curious to see if Komodo 14 can score as well against the same opponents. It would score much better with high Contempt, but so would Stockfish, so I guess it's fair enough to compare.

Have it running, takes about 2 hours.

Regarding elo's:

Code: Select all

Rank Name                          Elo     +/-   Games   Score    Draw  CCRL
   0 Stockfish_11                   52      20    1000   57.4%   16.2%  3537
   1 Benjamin                       26      43     200   53.8%   19.5%  2646
   2 ProDeo 2.2                     17      45     200   52.5%   14.0%  2770
   3 Fruit_2.3                     -54      45     200   42.3%   16.5%  2783
   4 Fruit_2.1                    -123      47     200   33.0%   15.0%  2684
   5 Ruffian_2                    -135      47     200   31.5%   16.0%  2608

Interesting is that Benjamin while listed 124 elo less than ProDeo scores better. I think that much of this kind of testing has to do with the killer instinct of an engine. And Benjamin is the gambit version of ProDeo. Style decides?

I think I've figured out a big part of the mystery of why both Stockfish and Komodo can give knight odds to such strong engines in your tests. When I first read that you were testing at 40/10, I assumed that the 10 meant ten minutes, since we are constantly referring to CCRL tests at 40/15 meaning 15 minutes. But when you starting giving results at 40/40 and even 40/80, I realized that you must mean 40/x SECONDS, not minutes, since you are unlikely to have enough hardware to play so many games so quickly (correct me if I'm wrong here)! So even these 40/40 results are (if I'm right now) bullet games, not rapid games. At bullet chess even the top human GMs would have trouble beating Stockfish or Komodo at knight odds. So the question is, what level engine can Komodo and Stockfish give knight odds to in rapid chess (15' + 10" being the standard now)? Obviously, it will be a weaker engine than these 2700+ engines, but how much weaker? I do have one data point: at 3' + 2" (blitz, roughly midway between bullet and rapid) I got a +94 elo result for Komodo 14 vs. Arasan 14 64 bit, about 2640 on CCRL 40/15 list (est. based on versions just before and after), which is a 2734 performance. But I used Contempt 150, which helps a lot at knight odds; I'll have to redo the test without Contempt (or with the default of just 4). My best guess is that with default Contempt, Komodo 14 and SF need opponents in the mid 2600s in blitz, and in the mid 2500s in Rapid (15' + 10"). I can run Komodo vs. Arasan 14 overnight at Rapid with default Contempt; if I'm correct Komodo will lose but not too badly. I run 63 games at once, so the thousand games won't finish in 8 or 9 hours, but maybe in 15 hours or so.

We hyper-analyzed these issues several years ago. IIRC, at bullet engine-engine Knight odds are some 600 (logistic) engine Elo points, at rapid 45min + 15s some 1200 logistic engine ELO points, and by continuation at tournament TC maybe 1400 engine Elo points (the last one was never really tested). The handicap is heavily TC dependent, and with humans it could be even more dependent (humans are weak at bullet and blitz). I don't think that even a perfect engine can give Knight odds to Carlsen at tournament TC. Currently SF and Komodo in whatever configuration are no stronger than 2100 FIDE Elo points Knight odds at tournament TC against a human.

My estimates for good "human" sparring at tournament TC is Lc0

11248 at 1000 nodes --- 2750 FIDE
11248 at 100 nodes --- 2450 FIDE
11248 at 10 nodes --- 2100 FIDE.

Again, tournament TC. At Blitz 5min + 3s add some 300 Elo points to those to get FIDE Blitz rating. All this is rough estimation, but it can be improved by testing.

Yes, we reached those conclusions a few years ago based on self-testing of the same engine at different time controls, IIRC. Whether it applies to games involving humans or to games with NN or NNUE engines that didn't exist then is unknown. Your estimate of 2100 FIDE Elo for Knight odds at TC vs human is pretty consistent with my estimate of 2300 for same at 15' + 10". In this new world of online-only chess it seems that 5 hour games may be mostly a thing of the past, and that Rapid has become the main form of chess, like it or not. Lc0 is probably a better substitute for a human than a conventional program for these tests; note that the networks starting with 70xxx are now the best for such purposes as they reverted to the no-resign training that made 11248 so good at playing lost positions, and they now play quite well down a piece or more, and they are stronger and play more sensibly than the old network. For testing against CPU engines there is the problem that you can't make proper use of a machine with many CPUs and just one GPU; you could use the Lc0 cpu version I suppose. Perhaps the best "human" for such tests might be an NNUE like Stockfish NNUE, but it's a bit early to say yet.
My overnight test of Komodo 14 giving knight odds to Arasan 14 at 15' + 10" on one thread is down 8.6 elo after 686 games, implying an elo of about 2570 CCRL Rapid at knight odds. This would probably mean something like 2670 on 32 Threads (each doubling is worth much less giving knight odds than in normal chess, so maybe 20 elo per doubling), which in turn means something like 2900 FIDE Rapid based on some estimates I made of the likely human equivalence in Rapid of CCRL rapid ratings, and this was without even setting Contempt. So it appears that using conventional engines to predict the rating needed to beat Komodo at knight odds in Rapid overstates the reality by something like 600 elo! Exactly why this is so is a bit of a mystery, even allowing for the fact that the engines don't fully appreciate the circumstances and aren't optimized for winning when up a piece. Perhaps this won't be the case if we substitute an NN (or NNUE) engine for the human.

[Rebel]

From the SF11 - Crafty knight-odds match.

[pgn][Event "?"]
[Site "?"]
[Date "2020.06.25"]
[Round "10"]
[White "Stockfish_11"]
[Black "Crafty_25.6"]
[Result "1-0"]
[FEN "rnbqkbnr/1p1p1ppp/p3p3/2p5/3PP3/2P5/PP3PPP/R1BQKBNR w KQkq - 0 1"]
[GameDuration "00:01:34"]
[GameEndTime "2020-06-25T08:17:11.715 West-Europa (zomertijd)"]
[GameStartTime "2020-06-25T08:15:37.398 West-Europa (zomertijd)"]
[PlyCount "90"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "40/40"]

1. Bd3 {-4.20/20 2.6s} cxd4 {-3.64/16 0.61s} 2. cxd4 {-4.00/17 0.38s}
d5 {-3.61/17 0.93s} 3. e5 {-4.12/20 0.58s} Qa5+ {-3.71/18 1.6s}
4. Kf1 {-4.03/20 0.37s} Nc6 {-3.57/16 0.94s} 5. Nf3 {-3.90/20 2.9s}
Bd7 {-3.73/16 0.63s} 6. a3 {-3.88/20 1.3s} Qb6 {-3.82/16 1.4s}
7. b4 {-3.92/18 0.58s} Rc8 {-3.79/16 0.93s} 8. g3 {-3.78/19 1.5s}
Nh6 {-3.57/16 1.2s} 9. Kg2 {-3.76/22 2.7s} Be7 {-3.52/17 1.6s}
10. Bxh6 {-3.36/16 0.27s} gxh6 {-3.61/17 0.62s} 11. Rc1 {-3.78/22 3.7s}
O-O {-3.50/17 1.2s} 12. Qd2 {-3.62/19 0.50s} Kg7 {-3.64/16 0.59s}
13. Qf4 {-3.94/20 0.61s} f5 {-3.65/16 0.95s} 14. Rhg1 {-3.83/20 1.8s}
h5 {-3.68/16 1.6s} 15. Rgd1 {-4.23/20 1.4s} Na7 {-3.79/14 1.8s}
16. Ng1 {-4.37/18 0.48s} Nb5 {-4.05/17 1.3s} 17. Bxb5 {-4.22/21 0.66s}
Bxb5 {-4.16/17 0.57s} 18. Qd2 {-3.99/18 0.28s} Rc4 {-4.18/16 0.58s}
19. Nh3 {-4.01/19 0.72s} Rg8 {-4.16/15 0.59s} 20. Nf4 {-5.14/21 4.1s}
h4 {-4.18/15 0.61s} 21. Kh1 {-5.02/17 0.45s} hxg3 {-4.33/16 1.5s}
22. hxg3 {-5.10/18 0.47s} Kh8 {-4.40/17 1.0s} 23. Kg2 {-5.21/18 0.48s}
Bg5 {-4.78/17 0.62s} 24. Qe3 {-4.71/18 0.22s} Ba4 {-4.70/17 0.62s}
25. Rh1 {-4.98/19 0.74s} Qc6 {-4.67/17 1.3s} 26. Qd2 {-6.08/22 3.2s}
Bc2 {-5.03/17 1.3s} 27. Rhe1 {-6.68/22 1.8s} Qa4 {-5.36/17 1.2s}
28. Qe3 {-6.09/17 0.14s} a5 {-5.89/16 0.62s} 29. Rh1 {-5.90/18 0.61s}
axb4 {-6.27/15 0.77s} 30. axb4 {-6.47/17 0.22s} Rxb4 {-6.49/17 1.2s}
31. Rxc2 {-7.71/22 1.5s} Qxc2 {-7.51/18 0.64s} 32. Qd3 {-7.69/16 0.082s}
Qc6 {-7.79/17 1.0s} 33. Nh5 {-7.84/21 0.89s} Rg6 {-7.78/16 1.0s}
34. Nf6 {-7.65/17 0.19s} Bxf6 {-8.00/18 1.1s} 35. exf6 {-7.64/17 0.22s}
Rxf6 {-8.00/16 0.69s} 36. Qe3 {-7.73/17 0.20s} Qc2 {-8.57/19 0.93s}
37. Rh6 {-8.50/19 0.54s} Rxh6 {-9.61/19 0.85s} 38. Qxh6 {-8.40/16 0.10s}
Qe4+ {-9.77/18 1.0s} 39. Kh2 {-9.14/15 0.33s} Kg8 {-10.62/19 0.72s}
40. Qg5+ {-9.39/17 0.20s} Kf7 {-9.84/10 0.007s} 41. Qh5+ {-9.67/22 2.3s}
Kf8 {-10.77/20 1.5s} 42. Qh6+ {-10.32/32 3.0s} Ke8 {-10.77/19 0.63s}
43. Qh5+ {-15.47/25 2.4s} Kd8 {-11.86/20 1.0s} 44. Qg5+ {-55.36/29 2.0s}
Kd7 {-12.00/21 0.65s} 45. Qg7+ {-62.66/34 1.8s}
Kc6 {-11.99/21 0.67s, White wins by adjudication} 1-0[/pgn]
White wins by adjudication} 1-0

Wrong adjudication by cute-chess, never seen that before.

And another one.

[pgn][Event "?"]
[Site "?"]
[Date "2020.06.25"]
[Round "19"]
[White "Stockfish_11"]
[Black "Crafty_25.6"]
[Result "1-0"]
[FEN "rnbqkbnr/1p1p1ppp/p3p3/1Bp5/4P3/5N2/PPPP1PPP/R1BQK2R w KQkq - 0 1"]
[GameDuration "00:02:20"]
[GameEndTime "2020-06-25T08:24:17.955 West-Europa (zomertijd)"]
[GameStartTime "2020-06-25T08:21:57.258 West-Europa (zomertijd)"]
[PlyCount "136"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "40/40"]

1. Be2 {-4.39/20 2.7s} Nc6 {-3.52/17 0.87s} 2. c3 {-4.46/18 1.00s}
Nf6 {-3.78/18 0.68s} 3. e5 {-4.41/18 0.39s} Nd5 {-3.94/18 0.62s}
4. O-O {-4.41/17 0.47s} Be7 {-3.96/17 0.65s} 5. d4 {-4.51/16 0.49s}
cxd4 {-3.93/14 0.64s} 6. cxd4 {-4.55/17 0.68s} d6 {-4.08/15 1.1s}
7. Bc4 {-4.25/18 1.1s} O-O {-4.04/16 0.86s} 8. Qe2 {-4.14/17 0.52s}
b5 {-4.07/16 0.95s} 9. Bd3 {-4.82/20 3.5s} Bd7 {-4.23/16 0.66s}
10. Re1 {-4.64/20 2.2s} Rc8 {-4.33/16 0.94s} 11. Be4 {-4.67/18 1.1s}
Qb6 {-4.28/16 1.2s} 12. Rd1 {-4.79/16 0.37s} h6 {-4.28/14 0.65s}
13. Bb1 {-4.14/17 0.43s} f6 {-4.34/17 1.8s} 14. exf6 {-4.78/20 1.6s}
Nxf6 {-4.42/16 2.0s} 15. Be3 {-4.24/18 0.55s} Nd5 {-4.21/17 1.9s}
16. Qd3 {-4.70/19 0.98s} Rf5 {-4.28/16 0.64s} 17. a3 {-4.58/17 0.49s}
Na5 {-4.21/17 1.4s} 18. Bd2 {-4.58/20 1.2s} Nc4 {-4.36/15 0.96s}
19. Bc1 {-4.37/16 0.24s} Qa7 {-4.37/16 1.5s} 20. Ra2 {-4.25/21 4.4s}
Kf8 {-4.10/17 3.5s} 21. Re1 {-4.30/19 1.6s} Na5 {-4.08/16 2.3s}
22. h3 {-4.24/17 0.42s} Qa8 {-4.17/15 1.7s} 23. Qd1 {-3.31/16 0.45s}
Rf7 {-4.11/15 0.68s} 24. Qd3 {-3.89/21 3.0s} Nf6 {-4.07/15 0.44s}
25. b4 {-3.07/18 0.57s} Nc4 {-4.08/14 0.66s} 26. Rae2 {-2.86/20 0.35s}
Bd8 {-3.93/15 1.4s} 27. Nh4 {-4.29/21 3.1s} Bb6 {-4.13/14 0.53s}
28. Kh2 {-4.24/18 0.41s} Qb8 {-4.30/14 2.1s} 29. Bf4 {-3.63/19 1.0s}
Qa7 {-4.15/13 0.97s} 30. Ng6+ {-3.65/16 0.34s} Kg8 {-4.24/14 0.45s}
31. Rd1 {-3.93/21 1.0s} a5 {-4.28/13 0.34s} 32. Bc1 {-3.78/17 0.61s}
axb4 {-4.76/17 0.89s} 33. axb4 {-4.22/15 0.14s} Qa4 {-4.76/16 0.34s}
34. Bc2 {-4.99/18 0.50s} Qxb4 {-4.98/14 0.38s} 35. Qg3 {-4.36/18 0.67s}
Nd5 {-5.11/14 0.59s} 36. Qh4 {-6.32/18 0.60s} Na5 {-5.76/13 0.93s}
37. Be3 {-6.34/16 0.36s} Nc6 {-5.63/14 0.81s} 38. Red2 {-6.15/16 0.21s}
Qa5 {-5.54/12 0.42s} 39. Kh1 {-6.05/14 0.13s} b4 {-5.74/12 0.23s}
40. Qg3 {-6.03/13 0.097s} Nxd4 {-5.94/10 0.056s} 41. Bxd4 {-5.71/21 0.97s}
Bxd4 {-5.35/15 0.65s} 42. Bb3 {-5.66/19 0.73s} Rc3 {-5.27/17 0.64s}
43. Qxd6 {-7.36/22 2.3s} Qb6 {-5.42/18 0.64s} 44. Qxb6 {-6.82/20 0.92s}
Bxb6 {-5.42/16 0.64s} 45. Rb1 {-6.76/20 0.89s} Bc7 {-5.99/15 0.69s}
46. h4 {-6.84/21 1.7s} Bb5 {-6.45/16 0.87s} 47. h5 {-7.25/22 3.7s}
Bd3 {-6.32/17 0.89s} 48. Ra1 {-6.84/18 0.60s} Rd7 {-6.22/16 0.98s}
49. Ba4 {-8.23/22 7.8s} Rd6 {-6.50/16 0.65s} 50. Ne5 {-8.10/19 0.92s}
Nf4 {-6.73/16 0.68s} 51. Nxd3 {-7.10/17 0.52s} Rdxd3 {-6.77/18 0.81s}
52. Rc2 {-6.79/21 2.5s} Rxc2 {-7.11/20 0.69s} 53. Bxc2 {-7.88/20 0.99s}
Rc3 {-7.11/20 0.69s} 54. Be4 {-8.06/23 0.97s} Kf7 {-7.22/20 0.96s}
55. Rb1 {-7.23/22 1.8s} Nxh5 {-7.49/17 0.84s} 56. Bf3 {-7.78/25 2.7s}
Nf4 {-8.05/19 0.97s} 57. Bd1 {-7.71/20 0.16s} Nd3 {-8.59/16 0.71s}
58. Kg1 {-7.78/20 0.34s} Bb6 {-8.77/19 1.1s} 59. Kf1 {-7.83/19 0.38s}
Bxf2 {-8.83/18 0.72s} 60. Ke2 {-7.70/21 1.2s} Bc5 {-9.11/17 0.74s}
61. Kd2 {-8.24/23 2.5s} Nf4 {-9.11/21 0.79s} 62. g4 {-7.94/16 0.25s}
Kf6 {-9.46/16 0.80s} 63. Bc2 {-8.13/19 0.49s} Kg5 {-9.77/16 0.83s}
64. Bd1 {-8.85/19 0.72s} e5 {-10.11/20 0.84s} 65. Ba4 {-10.49/19 1.3s}
e4 {-10.12/17 0.95s} 66. Kd1 {-12.99/18 0.40s} Kxg4 {-11.79/16 0.87s}
67. Bd7+ {-57.00/19 0.55s} Kf3 {-14.01/19 0.91s} 68. Bc6 {-12.54/13 0.048s}
b3 {-18.65/19 0.95s, White wins by adjudication} 1-0
[/pgn]
And there are many more.

[chrisw]

From the SF11 - Crafty knight-odds match.

[pgn][Event "?"]
[Site "?"]
[Date "2020.06.25"]
[Round "10"]
[White "Stockfish_11"]
[Black "Crafty_25.6"]
[Result "1-0"]
[FEN "rnbqkbnr/1p1p1ppp/p3p3/2p5/3PP3/2P5/PP3PPP/R1BQKBNR w KQkq - 0 1"]
[GameDuration "00:01:34"]
[GameEndTime "2020-06-25T08:17:11.715 West-Europa (zomertijd)"]
[GameStartTime "2020-06-25T08:15:37.398 West-Europa (zomertijd)"]
[PlyCount "90"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "40/40"]

1. Bd3 {-4.20/20 2.6s} cxd4 {-3.64/16 0.61s} 2. cxd4 {-4.00/17 0.38s}
d5 {-3.61/17 0.93s} 3. e5 {-4.12/20 0.58s} Qa5+ {-3.71/18 1.6s}
4. Kf1 {-4.03/20 0.37s} Nc6 {-3.57/16 0.94s} 5. Nf3 {-3.90/20 2.9s}
Bd7 {-3.73/16 0.63s} 6. a3 {-3.88/20 1.3s} Qb6 {-3.82/16 1.4s}
7. b4 {-3.92/18 0.58s} Rc8 {-3.79/16 0.93s} 8. g3 {-3.78/19 1.5s}
Nh6 {-3.57/16 1.2s} 9. Kg2 {-3.76/22 2.7s} Be7 {-3.52/17 1.6s}
10. Bxh6 {-3.36/16 0.27s} gxh6 {-3.61/17 0.62s} 11. Rc1 {-3.78/22 3.7s}
O-O {-3.50/17 1.2s} 12. Qd2 {-3.62/19 0.50s} Kg7 {-3.64/16 0.59s}
13. Qf4 {-3.94/20 0.61s} f5 {-3.65/16 0.95s} 14. Rhg1 {-3.83/20 1.8s}
h5 {-3.68/16 1.6s} 15. Rgd1 {-4.23/20 1.4s} Na7 {-3.79/14 1.8s}
16. Ng1 {-4.37/18 0.48s} Nb5 {-4.05/17 1.3s} 17. Bxb5 {-4.22/21 0.66s}
Bxb5 {-4.16/17 0.57s} 18. Qd2 {-3.99/18 0.28s} Rc4 {-4.18/16 0.58s}
19. Nh3 {-4.01/19 0.72s} Rg8 {-4.16/15 0.59s} 20. Nf4 {-5.14/21 4.1s}
h4 {-4.18/15 0.61s} 21. Kh1 {-5.02/17 0.45s} hxg3 {-4.33/16 1.5s}
22. hxg3 {-5.10/18 0.47s} Kh8 {-4.40/17 1.0s} 23. Kg2 {-5.21/18 0.48s}
Bg5 {-4.78/17 0.62s} 24. Qe3 {-4.71/18 0.22s} Ba4 {-4.70/17 0.62s}
25. Rh1 {-4.98/19 0.74s} Qc6 {-4.67/17 1.3s} 26. Qd2 {-6.08/22 3.2s}
Bc2 {-5.03/17 1.3s} 27. Rhe1 {-6.68/22 1.8s} Qa4 {-5.36/17 1.2s}
28. Qe3 {-6.09/17 0.14s} a5 {-5.89/16 0.62s} 29. Rh1 {-5.90/18 0.61s}
axb4 {-6.27/15 0.77s} 30. axb4 {-6.47/17 0.22s} Rxb4 {-6.49/17 1.2s}
31. Rxc2 {-7.71/22 1.5s} Qxc2 {-7.51/18 0.64s} 32. Qd3 {-7.69/16 0.082s}
Qc6 {-7.79/17 1.0s} 33. Nh5 {-7.84/21 0.89s} Rg6 {-7.78/16 1.0s}
34. Nf6 {-7.65/17 0.19s} Bxf6 {-8.00/18 1.1s} 35. exf6 {-7.64/17 0.22s}
Rxf6 {-8.00/16 0.69s} 36. Qe3 {-7.73/17 0.20s} Qc2 {-8.57/19 0.93s}
37. Rh6 {-8.50/19 0.54s} Rxh6 {-9.61/19 0.85s} 38. Qxh6 {-8.40/16 0.10s}
Qe4+ {-9.77/18 1.0s} 39. Kh2 {-9.14/15 0.33s} Kg8 {-10.62/19 0.72s}
40. Qg5+ {-9.39/17 0.20s} Kf7 {-9.84/10 0.007s} 41. Qh5+ {-9.67/22 2.3s}
Kf8 {-10.77/20 1.5s} 42. Qh6+ {-10.32/32 3.0s} Ke8 {-10.77/19 0.63s}
43. Qh5+ {-15.47/25 2.4s} Kd8 {-11.86/20 1.0s} 44. Qg5+ {-55.36/29 2.0s}
Kd7 {-12.00/21 0.65s} 45. Qg7+ {-62.66/34 1.8s}
Kc6 {-11.99/21 0.67s, White wins by adjudication} 1-0[/pgn]
White wins by adjudication} 1-0

Wrong adjudication by cute-chess, never seen that before.

And another one.

[pgn][Event "?"]
[Site "?"]
[Date "2020.06.25"]
[Round "19"]
[White "Stockfish_11"]
[Black "Crafty_25.6"]
[Result "1-0"]
[FEN "rnbqkbnr/1p1p1ppp/p3p3/1Bp5/4P3/5N2/PPPP1PPP/R1BQK2R w KQkq - 0 1"]
[GameDuration "00:02:20"]
[GameEndTime "2020-06-25T08:24:17.955 West-Europa (zomertijd)"]
[GameStartTime "2020-06-25T08:21:57.258 West-Europa (zomertijd)"]
[PlyCount "136"]
[SetUp "1"]
[Termination "adjudication"]
[TimeControl "40/40"]

1. Be2 {-4.39/20 2.7s} Nc6 {-3.52/17 0.87s} 2. c3 {-4.46/18 1.00s}
Nf6 {-3.78/18 0.68s} 3. e5 {-4.41/18 0.39s} Nd5 {-3.94/18 0.62s}
4. O-O {-4.41/17 0.47s} Be7 {-3.96/17 0.65s} 5. d4 {-4.51/16 0.49s}
cxd4 {-3.93/14 0.64s} 6. cxd4 {-4.55/17 0.68s} d6 {-4.08/15 1.1s}
7. Bc4 {-4.25/18 1.1s} O-O {-4.04/16 0.86s} 8. Qe2 {-4.14/17 0.52s}
b5 {-4.07/16 0.95s} 9. Bd3 {-4.82/20 3.5s} Bd7 {-4.23/16 0.66s}
10. Re1 {-4.64/20 2.2s} Rc8 {-4.33/16 0.94s} 11. Be4 {-4.67/18 1.1s}
Qb6 {-4.28/16 1.2s} 12. Rd1 {-4.79/16 0.37s} h6 {-4.28/14 0.65s}
13. Bb1 {-4.14/17 0.43s} f6 {-4.34/17 1.8s} 14. exf6 {-4.78/20 1.6s}
Nxf6 {-4.42/16 2.0s} 15. Be3 {-4.24/18 0.55s} Nd5 {-4.21/17 1.9s}
16. Qd3 {-4.70/19 0.98s} Rf5 {-4.28/16 0.64s} 17. a3 {-4.58/17 0.49s}
Na5 {-4.21/17 1.4s} 18. Bd2 {-4.58/20 1.2s} Nc4 {-4.36/15 0.96s}
19. Bc1 {-4.37/16 0.24s} Qa7 {-4.37/16 1.5s} 20. Ra2 {-4.25/21 4.4s}
Kf8 {-4.10/17 3.5s} 21. Re1 {-4.30/19 1.6s} Na5 {-4.08/16 2.3s}
22. h3 {-4.24/17 0.42s} Qa8 {-4.17/15 1.7s} 23. Qd1 {-3.31/16 0.45s}
Rf7 {-4.11/15 0.68s} 24. Qd3 {-3.89/21 3.0s} Nf6 {-4.07/15 0.44s}
25. b4 {-3.07/18 0.57s} Nc4 {-4.08/14 0.66s} 26. Rae2 {-2.86/20 0.35s}
Bd8 {-3.93/15 1.4s} 27. Nh4 {-4.29/21 3.1s} Bb6 {-4.13/14 0.53s}
28. Kh2 {-4.24/18 0.41s} Qb8 {-4.30/14 2.1s} 29. Bf4 {-3.63/19 1.0s}
Qa7 {-4.15/13 0.97s} 30. Ng6+ {-3.65/16 0.34s} Kg8 {-4.24/14 0.45s}
31. Rd1 {-3.93/21 1.0s} a5 {-4.28/13 0.34s} 32. Bc1 {-3.78/17 0.61s}
axb4 {-4.76/17 0.89s} 33. axb4 {-4.22/15 0.14s} Qa4 {-4.76/16 0.34s}
34. Bc2 {-4.99/18 0.50s} Qxb4 {-4.98/14 0.38s} 35. Qg3 {-4.36/18 0.67s}
Nd5 {-5.11/14 0.59s} 36. Qh4 {-6.32/18 0.60s} Na5 {-5.76/13 0.93s}
37. Be3 {-6.34/16 0.36s} Nc6 {-5.63/14 0.81s} 38. Red2 {-6.15/16 0.21s}
Qa5 {-5.54/12 0.42s} 39. Kh1 {-6.05/14 0.13s} b4 {-5.74/12 0.23s}
40. Qg3 {-6.03/13 0.097s} Nxd4 {-5.94/10 0.056s} 41. Bxd4 {-5.71/21 0.97s}
Bxd4 {-5.35/15 0.65s} 42. Bb3 {-5.66/19 0.73s} Rc3 {-5.27/17 0.64s}
43. Qxd6 {-7.36/22 2.3s} Qb6 {-5.42/18 0.64s} 44. Qxb6 {-6.82/20 0.92s}
Bxb6 {-5.42/16 0.64s} 45. Rb1 {-6.76/20 0.89s} Bc7 {-5.99/15 0.69s}
46. h4 {-6.84/21 1.7s} Bb5 {-6.45/16 0.87s} 47. h5 {-7.25/22 3.7s}
Bd3 {-6.32/17 0.89s} 48. Ra1 {-6.84/18 0.60s} Rd7 {-6.22/16 0.98s}
49. Ba4 {-8.23/22 7.8s} Rd6 {-6.50/16 0.65s} 50. Ne5 {-8.10/19 0.92s}
Nf4 {-6.73/16 0.68s} 51. Nxd3 {-7.10/17 0.52s} Rdxd3 {-6.77/18 0.81s}
52. Rc2 {-6.79/21 2.5s} Rxc2 {-7.11/20 0.69s} 53. Bxc2 {-7.88/20 0.99s}
Rc3 {-7.11/20 0.69s} 54. Be4 {-8.06/23 0.97s} Kf7 {-7.22/20 0.96s}
55. Rb1 {-7.23/22 1.8s} Nxh5 {-7.49/17 0.84s} 56. Bf3 {-7.78/25 2.7s}
Nf4 {-8.05/19 0.97s} 57. Bd1 {-7.71/20 0.16s} Nd3 {-8.59/16 0.71s}
58. Kg1 {-7.78/20 0.34s} Bb6 {-8.77/19 1.1s} 59. Kf1 {-7.83/19 0.38s}
Bxf2 {-8.83/18 0.72s} 60. Ke2 {-7.70/21 1.2s} Bc5 {-9.11/17 0.74s}
61. Kd2 {-8.24/23 2.5s} Nf4 {-9.11/21 0.79s} 62. g4 {-7.94/16 0.25s}
Kf6 {-9.46/16 0.80s} 63. Bc2 {-8.13/19 0.49s} Kg5 {-9.77/16 0.83s}
64. Bd1 {-8.85/19 0.72s} e5 {-10.11/20 0.84s} 65. Ba4 {-10.49/19 1.3s}
e4 {-10.12/17 0.95s} 66. Kd1 {-12.99/18 0.40s} Kxg4 {-11.79/16 0.87s}
67. Bd7+ {-57.00/19 0.55s} Kf3 {-14.01/19 0.91s} 68. Bc6 {-12.54/13 0.048s}
b3 {-18.65/19 0.95s, White wins by adjudication} 1-0
[/pgn]
And there are many more.

Crafty sends out weirdo texts in the middle of search, often right in the middle of a PV stream. Common one is “Trojan check” or something. If a UI is not expecting random non UCI output mixed in with the UCI stream?

[Rebel]

Interesting is that Benjamin while listed 124 elo less than ProDeo scores better. I think that much of this kind of testing has to do with the killer instinct of an engine. And Benjamin is the gambit version of ProDeo. Style decides?

I think I've figured out a big part of the mystery of why both Stockfish and Komodo can give knight odds to such strong engines in your tests. When I first read that you were testing at 40/10, I assumed that the 10 meant ten minutes, since we are constantly referring to CCRL tests at 40/15 meaning 15 minutes. But when you starting giving results at 40/40 and even 40/80, I realized that you must mean 40/x SECONDS, not minutes, since you are unlikely to have enough hardware to play so many games so quickly (correct me if I'm wrong here)! So even these 40/40 results are (if I'm right now) bullet games, not rapid games. At bullet chess even the top human GMs would have trouble beating Stockfish or Komodo at knight odds. So the question is, what level engine can Komodo and Stockfish give knight odds to in rapid chess (15' + 10" being the standard now)? Obviously, it will be a weaker engine than these 2700+ engines, but how much weaker? I do have one data point: at 3' + 2" (blitz, roughly midway between bullet and rapid) I got a +94 elo result for Komodo 14 vs. Arasan 14 64 bit, about 2640 on CCRL 40/15 list (est. based on versions just before and after), which is a 2734 performance. But I used Contempt 150, which helps a lot at knight odds; I'll have to redo the test without Contempt (or with the default of just 4). My best guess is that with default Contempt, Komodo 14 and SF need opponents in the mid 2600s in blitz, and in the mid 2500s in Rapid (15' + 10"). I can run Komodo vs. Arasan 14 overnight at Rapid with default Contempt; if I'm correct Komodo will lose but not too badly. I run 63 games at once, so the thousand games won't finish in 8 or 9 hours, but maybe in 15 hours or so.

Sorry for the time control confusion, I used the cute-chess format.

It won't be easy to find engines fit for the purpose you want it to be, a short test:

Code: Select all

Komodo 14  1 sec  2 sec  4 sec  8 sec  CCRL
Fruit 2.1  29.3%  30.5%  41.0%  47.5%  2684
Benjamin   46.0%  56.5%                2646

While Benjamin is CCRL rated lower it only needs 2 seconds average to beat Komodo.

[lkaufman]

Interesting is that Benjamin while listed 124 elo less than ProDeo scores better. I think that much of this kind of testing has to do with the killer instinct of an engine. And Benjamin is the gambit version of ProDeo. Style decides?

I think I've figured out a big part of the mystery of why both Stockfish and Komodo can give knight odds to such strong engines in your tests. When I first read that you were testing at 40/10, I assumed that the 10 meant ten minutes, since we are constantly referring to CCRL tests at 40/15 meaning 15 minutes. But when you starting giving results at 40/40 and even 40/80, I realized that you must mean 40/x SECONDS, not minutes, since you are unlikely to have enough hardware to play so many games so quickly (correct me if I'm wrong here)! So even these 40/40 results are (if I'm right now) bullet games, not rapid games. At bullet chess even the top human GMs would have trouble beating Stockfish or Komodo at knight odds. So the question is, what level engine can Komodo and Stockfish give knight odds to in rapid chess (15' + 10" being the standard now)? Obviously, it will be a weaker engine than these 2700+ engines, but how much weaker? I do have one data point: at 3' + 2" (blitz, roughly midway between bullet and rapid) I got a +94 elo result for Komodo 14 vs. Arasan 14 64 bit, about 2640 on CCRL 40/15 list (est. based on versions just before and after), which is a 2734 performance. But I used Contempt 150, which helps a lot at knight odds; I'll have to redo the test without Contempt (or with the default of just 4). My best guess is that with default Contempt, Komodo 14 and SF need opponents in the mid 2600s in blitz, and in the mid 2500s in Rapid (15' + 10"). I can run Komodo vs. Arasan 14 overnight at Rapid with default Contempt; if I'm correct Komodo will lose but not too badly. I run 63 games at once, so the thousand games won't finish in 8 or 9 hours, but maybe in 15 hours or so.

Sorry for the time control confusion, I used the cute-chess format.

It won't be easy to find engines fit for the purpose you want it to be, a short test:

Code: Select all

Komodo 14  1 sec  2 sec  4 sec  8 sec  CCRL
Fruit 2.1  29.3%  30.5%  41.0%  47.5%  2684
Benjamin   46.0%  56.5%                2646

While Benjamin is CCRL rated lower it only needs 2 seconds average to beat Komodo.

Very good, it looks like Benjamin is much closer to what I'm looking for than most (if not all) other engines. Based on CCRL ratings and my estimates for converting to FIDE rapid, it should be roughly a fair match at standard chess, Rapid (15' + 10") time control, with Magnus Carlsen on one core on a modern I7. By extrapolating to about 25" per move it should score somewhere in the 80 to 90% range at knight odds vs Komodo. That's probably still less than Carlsen would score, but it's not ridiculous. So, some questions about Benjamin.
1. Is it currently available? If so, how?
2. Is there a Linux version? I use Windows myself, but our Komodo tester uses Linux. I can test either way, but much faster on our tester.
3. Does it have a way to reduce the level of play (moderately), other than just giving it less time? If not, shortening the time should work fine.
4. Any insights into why it is so much better at this than Fruit? You mentioned gambit-style play, but that's not normally the sensible way to exploit an extra piece. More like the way to play when down a piece!
5. Final question: When you say "1 sec" (for example) above, is that movetime = 1 second, or 40 moves in 40 seconds so average one second? It's not a huge difference, but obviously quality of play is higher in the second case.
The idea here is that when and if we find a way to make Komodo play much better down a piece than it does now, we need a way to prove this without going to the trouble and expense of a GM match without some idea that we might do well. While it is unrealistic to expect to beat an active GM at knight odds in a Rapid match, if we specify Armageddon knight odds, meaning that draws count as wins for White (which is logical, he's down a piece), then it becomes a realistic goal.

[Rebel]

Wrote a util that checks if the result tag is right in cute-chess, from the 200 SF11-Crafty games the util reported 52 errors, won games by Crafty reported by cute-chess as a win for Stockfish ??

Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -62.66/34 1.8s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -12.54/13 0.048s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -64.15/20 0.50s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -17.17/17 0.29s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -11.32/16 0.12s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -9.49/14 0.096s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M26/27 0.69s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -16.27/21 4.0s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -14.00/20 0.64s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -9.35/12 0.11s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -9.00/21 1.9s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -56.28/26 4.6s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -10.17/16 0.33s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -9.45/16 0.25s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -16.37/22 0.97s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M20/32 0.11s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -15.38/19 1.3s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -20.86/15 0.094s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M12/30 0.053s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -68.32/23 1.7s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M46/31 0.82s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -12.16/14 0.029s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M16/37 0.27s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -65.97/27 0.30s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -56.09/1 0s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M18/23 0.032s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -26.43/17 0.51s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M18/49 2.2s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -22.86/15 0.12s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -68.94/29 0.26s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -11.33/17 0.42s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -9.55/18 0.47s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -9.41/22 0.091s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M20/27 0.43s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -70.33/1 0s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M22/48 1.5s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M6/245 0.10s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -13.62/16 0.17s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M4/245 0.008s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -57.16/17 0.13s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -65.53/26 1.2s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M26/38 1.5s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M28/42 0.63s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M20/33 0.13s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M16/50 1.1s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M16/34 0.19s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -63.28/24 0.64s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M16/40 0.32s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M26/35 0.30s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M14/31 0.048s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -11.89/20 1.2s
Stockfish_11 - Crafty_25.6 [1-0] Stockfish_11 last move -M54/21 0.26s

Definitely a cute-chess bug.

Found out that adding the option "whitepov" fixes the problem and Crafty scored as one would expect.

Code: Select all

   # ENGINE          : RATING    POINTS  PLAYED    (%)
   1 Crafty_25.6     : 3034.6     119.5     200   59.8%
   2 Stockfish_11    : 2965.4      80.5     200   40.3%