Milos wrote: ↑Fri Nov 23, 2018 11:46 pm
Your self play TC was 1min per game or similar, his games were 30sec per move. Due to draw rate you might have higher error margins with 20 super fast self-play games than with 4 30sec/move games.
In his case if nets were really of equal strength draw probability could be easily 80%. So probability of 3 wins out of 4 games for one engine in case of engines of equal strength would be like 0.1%.
So 4 games could be indeed more than sufficient to prove with almost 100% certainty that engine A is stronger than engine B.
Ofc one would need to have some knowledge of statistics which doesn't seem to be your case...
I can show a sample of CCRL 40/40 games (easily 30s/move) between equal-rated engines (2900) with a sequence of 4 1-0 for either one.
Self-play may have even higher draw rate plus probably much higher for top engines, but that doesn't change the fact that 4 games is not enough whether you like it or not. As for your "100% certainty", no comment
mar wrote: ↑Sat Nov 24, 2018 1:06 amI can show a sample of CCRL 40/40 games (easily 30s/move) between equal-rated engines (2900) with a sequence of 4 1-0 for either one.
Self-play may have even higher draw rate plus probably much higher for top engines, but that doesn't change the fact that 4 games is not enough whether you like it or not. As for your "100% certainty", no comment
CCRL 40/40 with 2900Elo engines is the level of SF selfplay from testing framework. You clearly don't understand the impact of draw rate on error margins.
Showing me an example of 4 games in a row between equal engines that have 70% draw rate would be enough, you can pick any TC and any engine strength you like.
But I'm pretty certain you can't simply because it implies at least 4000 LTC games of high-quality opponents .
custom openings that may exaggerate Elo differences ( due to the unbalance nature of the openings )..
Hmm, I read this forum post just before posting this. According to his test with decent hardware ( that would closely reflect performance in Tcec or CCCC) the estimate is -200 elo below latest SF, whereas best 11248 is known to be below -100 elo.(speed ratio 1:1000). And also, in your slow GPU or very short time control, you are testing mostly the strength of policy heads because the value net (MCTS dont have a good chance to correct the mistakes done by policy head) https://ibb.co/eq90FV
20Mnpmove close to TCEC performance for SFdev???
Didn't know TCEC used TC of 10''+0.1''.
Among testers, his hardware setup is most similar to those TCEC/CCCC. I think the average speeds of Lco and SF in last CCCC was around 40knps vs 80 MNps(1:2000), that would add another -50 elo gap between Lco and SF.
What can I say, you clearly lack basic comprehension skills...
carldaman wrote: ↑Sat Nov 24, 2018 1:46 am
If self-play is totally broken and meaningless, why don't they test against engines of established known strength?
the self play testing is useful because it shows, at a minimum, that something is happened with the training. Contrast to situation where self play got stuck.
mar wrote: ↑Sat Nov 24, 2018 1:06 amI can show a sample of CCRL 40/40 games (easily 30s/move) between equal-rated engines (2900) with a sequence of 4 1-0 for either one.
Self-play may have even higher draw rate plus probably much higher for top engines, but that doesn't change the fact that 4 games is not enough whether you like it or not. As for your "100% certainty", no comment
This is interesting...
But I tried the NN 11149 against several NNs with elo> 6000 and every time the 11149 wins
Is it possible that always getting the same result is due to the fact of using a few games?
I have no high knowledge of statistics, but I have doubts
Btw, Kings Crusher said in twitch chat that, Stockfish newer version +50 elo is also practically self play elo cos all AB engines have more or less similar search.
And one tester said, although SF 9 vs Lco in his tests is +50 from SF 9, SF10 vs Lc0 is +70.
Practically SF +50 elo is also bogus cos SF is playing against another 100 AB engines. If there were 50 AB engines and 50 NN engines, I thought it would scale to +25 or +30 only.
.
