CCCC Rapid Rumble results simulator

[JJJ]

lc0 memorizes patterns and was trained first with random move so probably each starting position even if of course she plays more and more the same openings with more training. But she can plays every positions thanks to that ( patterns and all opening trained ).

I call CCCC fair condition for Leela because she has enough power this time and the ratio for her seems good enough to compete.

And for your theory, you need 3 tests to be sure.
1 test with book for both engine
1 test without book for both engine
1 test with no book for Leela and a book for engine

And then you can compare if there is a big difference in elo. It might, but we can't know for sure until tested right ?

[JJJ]

Lc0 is now on part with Shredder for #4 spot and I think it will take this spot. After all, many test have shown Lc0 being the true number 4 with fair condition and in this CCCC it is also the case.

So lc0 for the #4 spot ! Your simulator is really innacurate. CCRL elo matters way more than the rating in during the tournament, except if the tournament reach his ends.

Sure, the rating in these particular conditions could be estimated as say
0.5*x*TP + (1-0.5*x)*CCRL

x is the fraction of games played out 46
TP is tournament performance
CCRL is the CCRL rating

This way the rating is always more determined by CCRL rating, from say 3 times more importance in the middle (23/46) of the tournament, to equal contribution of TP and CCRL at the end. The rating is then applied to remaining games, and sure starting with the current points in tournament.
This is just an example. I am countryside on vacation right now, so I can do no simulations, just write here mostly on my phone.
Lc0 estimates are very hard, the conditions matter a lot, CCRL is irrelevant, but as you observed, CCCC seem to be a fair tournament for Lc0 (pity it doesn't scale much beyond 2 GPUs), and in my fair conditions Lc0 is the fourth engine (at worst fifth, behind the last Fire), which is close to what we observe here. TCEC conditions were awful for Lc0.

Seems a nice idea.

[Uri Blass]

lc0 memorizes patterns and was trained first with random move so probably each starting position even if of course she plays more and more the same openings with more training. But she can plays every positions thanks to that ( patterns and all opening trained ).

I call CCCC fair condition for Leela because she has enough power this time and the ratio for her seems good enough to compete.

And for your theory, you need 3 tests to be sure.
1 test with book for both engine
1 test without book for both engine
1 test with no book for Leela and a book for engine

And then you can compare if there is a big difference in elo. It might, but we can't know for sure until tested right ?

Good conditions for Leela may be unfair for other programs.
It is not clear what is the meaning of fair when there is no equal hardware.

[Nay Lin Tun]

lc0 memorizes patterns and was trained first with random move so probably each starting position even if of course she plays more and more the same openings with more training. But she can plays every positions thanks to that ( patterns and all opening trained ).

I call CCCC fair condition for Leela because she has enough power this time and the ratio for her seems good enough to compete.

And for your theory, you need 3 tests to be sure.
1 test with book for both engine
1 test without book for both engine
1 test with no book for Leela and a book for engine

And then you can compare if there is a big difference in elo. It might, but we can't know for sure until tested right ?

Good conditions for Leela may be unfair for other programs.
It is not clear what is the meaning of fair when there is no equal hardware.

All competitions doent need to be fair!
In top gaming competitions, the conditions are not always fair but people still enjoy (one player will randomly sprawns in bad location and the other one get good location, random weapons etc)
Also, in Formula one competition, all driver uses different cars. For fairness, " All drivers must use exactly the same car, must have exactly the same body weight, exactly the same wind effects and ground condition etc." whereas those are theroetically and practically impossible.

In general, "when unbiased general public see the conditions are fair, it can be assumed as fair" .

The same in Courts, when most judges/juries see it is fair, the outcome is fair"

[chrisw]

290 rounds so far

"broken" - Milos
"LOL" - Laskos
"really inaccurate" - JJJ
"ridiculous" - Milos

Code: Select all

Engine Tournament Init   1st  2nd  3rd  4th  5th  6th  7th ....
Houdini     3452  3400   0.51 0.31 0.16 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Stockfish   3451  3439   0.34 0.40 0.23 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Komodo      3436  3404   0.14 0.28 0.51 0.05 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Shredder    3346  3287   0.00 0.00 0.04 0.34 0.29 0.20 0.09 0.03 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Lc0         3346  3300   0.00 0.00 0.03 0.32 0.28 0.22 0.10 0.04 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Fire        3348  3326   0.00 0.00 0.02 0.22 0.28 0.28 0.13 0.05 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Booot       3314  3276   0.00 0.00 0.00 0.04 0.09 0.18 0.35 0.22 0.09 0.02 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Ethereal    3304  3283   0.00 0.00 0.00 0.01 0.04 0.09 0.22 0.37 0.18 0.06 0.02 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Andscacs    3268  3244   0.00 0.00 0.00 0.00 0.01 0.03 0.08 0.21 0.41 0.16 0.06 0.03 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Fritz       3231  3200   0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.04 0.12 0.26 0.23 0.15 0.09 0.05 0.03 0.02 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Xiphos      3218  3179   0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.03 0.10 0.25 0.23 0.16 0.10 0.06 0.04 0.02 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Texel       3204  3144   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.03 0.10 0.16 0.19 0.17 0.13 0.09 0.06 0.03 0.02 0.01 0.00 0.00 0.00 0.00 0.00
Pedone      3194  3090   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.07 0.12 0.16 0.18 0.15 0.12 0.08 0.05 0.03 0.01 0.00 0.00 0.00 0.00 0.00
Gull        3200  3184   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.04 0.07 0.11 0.14 0.16 0.16 0.12 0.09 0.05 0.02 0.01 0.00 0.00 0.00 0.00
Vajolet     3177  3101   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.03 0.06 0.10 0.15 0.17 0.16 0.13 0.09 0.05 0.03 0.01 0.00 0.00 0.00 0.00
Fizbo       3185  3259   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.02 0.04 0.07 0.10 0.14 0.18 0.16 0.13 0.08 0.04 0.02 0.00 0.00 0.00
Laser       3167  3226   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.03 0.05 0.07 0.11 0.15 0.18 0.18 0.12 0.07 0.03 0.00 0.00 0.00
Arasan      3152  3123   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.02 0.04 0.06 0.10 0.14 0.19 0.19 0.14 0.08 0.04 0.00 0.00 0.00
Nemorino    3123  3099   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.01 0.02 0.04 0.07 0.12 0.19 0.25 0.28 0.01 0.00 0.00
Wasp        3112  3041   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.01 0.03 0.06 0.11 0.21 0.27 0.30 0.01 0.00 0.00
Ivanhoe     3116  3115   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.02 0.03 0.06 0.11 0.19 0.26 0.30 0.01 0.00 0.00
Senpai      3028  3112   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.76 0.17 0.04
Nirvana     2998  3186   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.15 0.53 0.32
Crafty      2961  3013   0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.29 0.64

[JJJ]

Lc0 is now 4th and will probably stay at this rank. Told you he was favorite for it. And the top 3 will stay in top 3. Stockfish still favorite for spot 1. Fire will probably catch up Shredder anytime soon as well and finished 5 and shredder will be 6. After that I can't say

[JJJ]

It is easy to predict even with a short tournament , because each engine is ranked with a good margin over the other, except Komodo / Houdini. So Stockfish is number 1 with a good margin, Lc0 number 4 with a good margin, then Fire, then Shredder, so the probability to have the number in the good order is pretty high. Higher than your simulator who needs to take in account the elo much more than the starting ranking of a tournament.

[jp]

In general, "when unbiased general public see the conditions are fair, it can be assumed as fair" .
The same in Courts, when most judges/juries see it is fair, the outcome is fair"

Yeah, the public/jury has to be unbiased or the outcome won't be fair. So jury selection, etc...
The public/jury needs to know the facts too.
I know from the other thread that lc0 CCCC hardware is 60 times more expensive than hardware for the other engines. I don't think if they knew the public would think that is fair conditions.

[Laskos]

In general, "when unbiased general public see the conditions are fair, it can be assumed as fair" .
The same in Courts, when most judges/juries see it is fair, the outcome is fair"

Yeah, the public/jury has to be unbiased or the outcome won't be fair. So jury selection, etc...
The public/jury needs to know the facts too.
I know from the other thread that lc0 CCCC hardware is 60 times more expensive than hardware for the other engines. I don't think if they knew the public would think that is fair conditions.

I am not sure about these prices, but the conditions seem fine for what will appear in just few several months. NPS Lc0 shows are not that impressive, and would be equivalent to 4 or so 1080Ti, if Lc0 scaled well above 2 GPUs (which it does not). In 2-3 months we will see the following, equivalent to CCCC conditions:

CPU: AMD Threadripper 2990X (32 cores at 3.4-4.0 GHz) ---- $1700
GPU: 2 of Nvidia Turing RTX 2080Ti --- 2x $1200 ---- $2400

RTX 2080Ti will probably have the full fp16 support, so it will be much faster than 1080Ti for Lc0, and 2 of these will probably be even more than what CCCC shows for Lc0 (it doesn't scale well to 4 GPUs).

So, I think pretty fair price-wise conditions in the frontier hardware appearing very soon. Also, the tournament performance is similar to that shown in pretty fair price-wise conditions at much smaller scale, say GTX 1060 versus 4 core i7.

[Werewolf]

RTX 2080Ti will probably have the full fp16 support, so it will be much faster than 1080Ti, and 2 of these will probably be even more than what CCCC shows for Lc0 (it doesn't scale well to 4 GPUs).

2080 Ti will be much faster than 1080 Ti for the reasons you give.
But I don't think 2x 2080 Ti will be faster than 2 x Titan V (or V100) according to the specs.

The Titan V will still be about 10% faster.