Benjamin 1.0 Gauntlet for CCRL 40/15

[Graham Banks]

PGN - http://kirill-kryukov.com/chess/discuss ... p?id=46933

[Graham Banks]

Code: Select all

CCRL 40/15 Rating List - Custom engine selection
1169618 games played by 2696 programs, run by 23 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 15 minutes on an Intel i7-4770k.
Computed on August 15, 2020 with Bayeselo based on 1'169'618 games
Tested by CCRL team, 2005-2020, http://ccrl.chessdom.com/ccrl/4040/

Rank                 Engine                   Elo   +    -   Score  AvOp  Games
1 ProDeo 2.2                              2723  +17  -17  48.3%   +9.7  1193
  ProDeo 2.8                              2706  +16  -16  50.9%   -7.6  1372
  ProDeo 1.86                             2701  +30  -30  49.2%   +3.2   370
  ProDeo 1.81                             2699  +32  -32  48.6%   +6.8   316
  Benjamin 1.0                            2696  +27  -27  51.2%   -5.7   464
  ProDeo 2.0                              2682  +21  -21  48.0%  +14.2   718
  ProDeo 1.85                             2674  +28  -28  46.7%  +21.7   423
  ProDeo 1.87                             2663  +25  -25  46.8%  +21.0   522
  ProDeo 1.83c                            2649  +30  -30  52.3%  -13.8   377
  ProDeo 1.74                             2638  +24  -24  51.5%   -9.1   587
  ProDeo 1.7                              2629  +33  -33  51.1%   -7.9   308
  ProDeo 1.2                              2619  +18  -18  50.0%   -3.2  1128
  ProDeo 1.6                              2618  +20  -20  49.4%   +2.2   858
  ProDeo 1.1                              2587  +30  -31  47.1%  +21.6   361
  ProDeo 1.1 Silver                       2574  +32  -32  49.0%   +5.2   335
  ProDeo 1.2 Mx4                          2557  +35  -35  46.8%  +22.8   277

[lkaufman]

Code: Select all

CCRL 40/15 Rating List - Custom engine selection
1169618 games played by 2696 programs, run by 23 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 15 minutes on an Intel i7-4770k.
Computed on August 15, 2020 with Bayeselo based on 1'169'618 games
Tested by CCRL team, 2005-2020, http://ccrl.chessdom.com/ccrl/4040/

Rank                 Engine                   Elo   +    -   Score  AvOp  Games
1 ProDeo 2.2                              2723  +17  -17  48.3%   +9.7  1193
  ProDeo 2.8                              2706  +16  -16  50.9%   -7.6  1372
  ProDeo 1.86                             2701  +30  -30  49.2%   +3.2   370
  ProDeo 1.81                             2699  +32  -32  48.6%   +6.8   316
  Benjamin 1.0                            2696  +27  -27  51.2%   -5.7   464
  ProDeo 2.0                              2682  +21  -21  48.0%  +14.2   718
  ProDeo 1.85                             2674  +28  -28  46.7%  +21.7   423
  ProDeo 1.87                             2663  +25  -25  46.8%  +21.0   522
  ProDeo 1.83c                            2649  +30  -30  52.3%  -13.8   377
  ProDeo 1.74                             2638  +24  -24  51.5%   -9.1   587
  ProDeo 1.7                              2629  +33  -33  51.1%   -7.9   308
  ProDeo 1.2                              2619  +18  -18  50.0%   -3.2  1128
  ProDeo 1.6                              2618  +20  -20  49.4%   +2.2   858
  ProDeo 1.1                              2587  +30  -31  47.1%  +21.6   361
  ProDeo 1.1 Silver                       2574  +32  -32  49.0%   +5.2   335
  ProDeo 1.2 Mx4                          2557  +35  -35  46.8%  +22.8   277

Since you have so much experience testing so many engines on the CCRL 40/15 list, I wanted to ask your opinion regarding the human meaning of a 2700 rating like Benjamin is getting (roughly) above on that list. Since this is well within the range of human FIDE ratings, both classical and rapid, it should in theory be possible to say that it is an estimate of the rating it would earn in a human FIDE tournament, playing on your reference hardware, at some time control. Based on whatever data you have, would you say it is an estimate for the rating it would get vs. humans at standard (2 hours + 30" inc or 40/2 hours), at something like 40/40 min., at something like 40/15 min., or at some other tc, or would you say that some constant would need to be added or subtracted? Clearly, whatever rating it would earn at standard tc, it would earn a higher one at 15' + 10" (roughly same as your 40/15 tc), so this should be clarified as much as possible. I've investigated this question somewhat myself but the evidence is rather contradictory, so I would appreciate your input.

[Graham Banks]

Code: Select all

CCRL 40/15 Rating List - Custom engine selection
1169618 games played by 2696 programs, run by 23 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 15 minutes on an Intel i7-4770k.
Computed on August 15, 2020 with Bayeselo based on 1'169'618 games
Tested by CCRL team, 2005-2020, http://ccrl.chessdom.com/ccrl/4040/

Rank                 Engine                   Elo   +    -   Score  AvOp  Games
1 ProDeo 2.2                              2723  +17  -17  48.3%   +9.7  1193
  ProDeo 2.8                              2706  +16  -16  50.9%   -7.6  1372
  ProDeo 1.86                             2701  +30  -30  49.2%   +3.2   370
  ProDeo 1.81                             2699  +32  -32  48.6%   +6.8   316
  Benjamin 1.0                            2696  +27  -27  51.2%   -5.7   464
  ProDeo 2.0                              2682  +21  -21  48.0%  +14.2   718
  ProDeo 1.85                             2674  +28  -28  46.7%  +21.7   423
  ProDeo 1.87                             2663  +25  -25  46.8%  +21.0   522
  ProDeo 1.83c                            2649  +30  -30  52.3%  -13.8   377
  ProDeo 1.74                             2638  +24  -24  51.5%   -9.1   587
  ProDeo 1.7                              2629  +33  -33  51.1%   -7.9   308
  ProDeo 1.2                              2619  +18  -18  50.0%   -3.2  1128
  ProDeo 1.6                              2618  +20  -20  49.4%   +2.2   858
  ProDeo 1.1                              2587  +30  -31  47.1%  +21.6   361
  ProDeo 1.1 Silver                       2574  +32  -32  49.0%   +5.2   335
  ProDeo 1.2 Mx4                          2557  +35  -35  46.8%  +22.8   277

Since you have so much experience testing so many engines on the CCRL 40/15 list, I wanted to ask your opinion regarding the human meaning of a 2700 rating like Benjamin is getting (roughly) above on that list. Since this is well within the range of human FIDE ratings, both classical and rapid, it should in theory be possible to say that it is an estimate of the rating it would earn in a human FIDE tournament, playing on your reference hardware, at some time control. Based on whatever data you have, would you say it is an estimate for the rating it would get vs. humans at standard (2 hours + 30" inc or 40/2 hours), at something like 40/40 min., at something like 40/15 min., or at some other tc, or would you say that some constant would need to be added or subtracted? Clearly, whatever rating it would earn at standard tc, it would earn a higher one at 15' + 10" (roughly same as your 40/15 tc), so this should be clarified as much as possible. I've investigated this question somewhat myself but the evidence is rather contradictory, so I would appreciate your input.

I've never really thought about it in detail.
I'll have a think about it and come back to you on it.

[Rebel]

Code: Select all

CCRL 40/15 Rating List - Custom engine selection
1169618 games played by 2696 programs, run by 23 testers
Ponder off, General books (up to 12 moves), 3-4-5 piece EGTB
Time control: Equivalent to 40 moves in 15 minutes on an Intel i7-4770k.
Computed on August 15, 2020 with Bayeselo based on 1'169'618 games
Tested by CCRL team, 2005-2020, http://ccrl.chessdom.com/ccrl/4040/

Rank                 Engine                   Elo   +    -   Score  AvOp  Games
1 ProDeo 2.2                              2723  +17  -17  48.3%   +9.7  1193
  ProDeo 2.8                              2706  +16  -16  50.9%   -7.6  1372
  ProDeo 1.86                             2701  +30  -30  49.2%   +3.2   370
  ProDeo 1.81                             2699  +32  -32  48.6%   +6.8   316
  Benjamin 1.0                            2696  +27  -27  51.2%   -5.7   464
  ProDeo 2.0                              2682  +21  -21  48.0%  +14.2   718
  ProDeo 1.85                             2674  +28  -28  46.7%  +21.7   423
  ProDeo 1.87                             2663  +25  -25  46.8%  +21.0   522
  ProDeo 1.83c                            2649  +30  -30  52.3%  -13.8   377
  ProDeo 1.74                             2638  +24  -24  51.5%   -9.1   587
  ProDeo 1.7                              2629  +33  -33  51.1%   -7.9   308
  ProDeo 1.2                              2619  +18  -18  50.0%   -3.2  1128
  ProDeo 1.6                              2618  +20  -20  49.4%   +2.2   858
  ProDeo 1.1                              2587  +30  -31  47.1%  +21.6   361
  ProDeo 1.1 Silver                       2574  +32  -32  49.0%   +5.2   335
  ProDeo 1.2 Mx4                          2557  +35  -35  46.8%  +22.8   277

I estimated -28 elo on my website, you -27, for once you got it right

Or me...