The real elo of lczero is in its name

[Daniel Shawul]

zero! Sorry i couldn't resist. Here a 100 match game of leela vs scorpio in progress



What would have been interesting for me is to match leela against MCTS version of scorpio with the minmax back up operator to evaluate how much the NN eval + averaging backup of lczero fair against a hand-crafted evaluation. I think they should be of equal strength, scorpio-mcts also trashes TSCP and could be 2400 elo i think. If we don't go further than that with lczero it means the lczero effort is one big evaluation tuning done from scratch.

When will the reality sink in for many who blindly belive you can overcome tactical weakness with just a neural network ? 5-million games played and still counting.

I am more and more inching towards labeling the AlphaZero result an elaborate hoax and believe me i am resisting every day. This is not coming from the test setup they used as some would claim but from this very glaring tactical weakness of MCTS that they claim is no problem. You would think Stockfish would get atleast one win from 100 games against alphazero. Why are they not playing AlphaZero on chess servers like they did for Go ? It sounds to me like a very very selective reporting of results. I am not one to bash other people's work but just following the facts until proven otherwise.

I think lczero should just continue using the same exact methods AlphaZero used so that we can ask google why we can't reproduce their results if that turned out to be the case.

Daniel

[CMCanavessi]

What do you think is the elo of Scorpio using the settings you just said? Because if it's 2400 then leela woud smoke it... We can try the same match with that config, if you're up to the challenge

[jhellis3]

This should age well .

[noobpwnftw]

I was suggesting the reverse: it might be better if they use NN evaluator over an alpha-beta search. Nevertheless, I think you have a point, and I couldn't resist to be a "conspiracy theorist" on the contradicting results we have so far.

[Daniel Shawul]

What do you think is the elo of Scorpio using the settings you just said? Because if it's 2400 then leela woud smoke it... We can try the same match with that config, if you're up to the challenge

Here is a tournament i made some time ago http://talkchess.com/forum/viewtopic.ph ... 56&t=66886

Code: Select all

 1.                       scorpio-mcts-ab16    375     25     25   1209    85.1%   -26    17.1% 
 2.                                 scorpio    357     23     23   1310    83.9%   -23    19.3% 
 3.                       scorpio-mcts-ab12    285     21     21   1478    77.9%   -25    16.8% 
 4.                       scorpio-mcts-ab10    187     21     21   1309    68.1%   -11    20.6% 
 5.                        scorpio-mcts-ab9    168     21     21   1308    66.3%   -10    19.6% 
 6.                        scorpio-mcts-ab8    131     20     20   1308    62.6%    -7    18.5% 
 7.                        scorpio-mcts-ab7     78     20     20   1308    57.2%    -3    18.3% 
 8.                        scorpio-mcts-ab6     27     20     20   1308    52.3%     0    20.1% 
 9.                        scorpio-mcts-ab5    -50     20     20   1308    44.3%     5    18.1% 
10.                        scorpio-mcts-ab4   -126     21     21   1308    36.9%    10    16.3% 
11.                        scorpio-mcts-ab3   -177     21     21   1308    32.3%    14    14.2% 
12.                        scorpio-mcts-min   -236     22     22   1310    26.1%    18    18.5% 
13.                        scorpio-mcts-ab2   -258     22     22   1308    24.7%    19    13.8% 
14.                        scorpio-mcts-ab1   -365     25     25   1310    16.6%    27    11.0% 
15.                         scorpio-mcts-ab   -389     25     25   1310    14.6%    29    11.7% 

You are matching it against #2 (default scorpio with alpha-beta). I am suggesting you match it against #12, which is purely MCTS but uses minmax backup operator. From the results of your tournament, it seems scorpio-mcts-min should be able to match it, don't you think ? Once you are done with that maybe you can go up the rank with the mcts+alpha-beta(ab) versions -- the #1 is actually a hybrid mcts-ab version though i belive this might be due to luck. The #1 mcts+ab is the one that will participate in TCEC anyway.

[AdminX]

What do you think is the elo of Scorpio using the settings you just said? Because if it's 2400 then leela woud smoke it... We can try the same match with that config, if you're up to the challenge

Here is a tournament i made some time ago http://talkchess.com/forum/viewtopic.ph ... 56&t=66886

Code: Select all

 1.                       scorpio-mcts-ab16    375     25     25   1209    85.1%   -26    17.1% 
 2.                                 scorpio    357     23     23   1310    83.9%   -23    19.3% 
 3.                       scorpio-mcts-ab12    285     21     21   1478    77.9%   -25    16.8% 
 4.                       scorpio-mcts-ab10    187     21     21   1309    68.1%   -11    20.6% 
 5.                        scorpio-mcts-ab9    168     21     21   1308    66.3%   -10    19.6% 
 6.                        scorpio-mcts-ab8    131     20     20   1308    62.6%    -7    18.5% 
 7.                        scorpio-mcts-ab7     78     20     20   1308    57.2%    -3    18.3% 
 8.                        scorpio-mcts-ab6     27     20     20   1308    52.3%     0    20.1% 
 9.                        scorpio-mcts-ab5    -50     20     20   1308    44.3%     5    18.1% 
10.                        scorpio-mcts-ab4   -126     21     21   1308    36.9%    10    16.3% 
11.                        scorpio-mcts-ab3   -177     21     21   1308    32.3%    14    14.2% 
12.                        scorpio-mcts-min   -236     22     22   1310    26.1%    18    18.5% 
13.                        scorpio-mcts-ab2   -258     22     22   1308    24.7%    19    13.8% 
14.                        scorpio-mcts-ab1   -365     25     25   1310    16.6%    27    11.0% 
15.                         scorpio-mcts-ab   -389     25     25   1310    14.6%    29    11.7% 

You are matching it against #2 (default scorpio with alpha-beta). I am suggesting you match it against #12, which is purely MCTS but uses minmax backup operator. From the results of your tournament, it seems scorpio-mcts-min should be able to match it, don't you think ? Once you are done with that maybe you can go up the rank with the mcts+alpha-beta(ab) versions -- the #1 is actually a hybrid mcts-ab version though i belive this might be due to luck. The #1 mcts+ab is the one that will participate in TCEC anyway.

I am suggesting you stream this match on Twitch!

[CMCanavessi]

I'm not at home right now but when I get there I might abort the current match and start a new one with those settings.

[George Tsavdaris]

I am more and more inching towards labeling the AlphaZero result an elaborate hoax and believe me i am resisting every day. This is not coming from the test setup they used as some would claim but from this very glaring tactical weakness of MCTS that they claim is no problem. You would think Stockfish would get atleast one win from 100 games against alphazero. Why are they not playing AlphaZero on chess servers like they did for Go ? It sounds to me like a very very selective reporting of results. I am not one to bash other people's work but just following the facts until proven otherwise.

Well Deepmind is a very serious researching company that produces very serious papers and in big scientific magazines, so accusing them of fabricating their results(or cherry-picking which is more polite) they publish on their papers is serious.
Deepmind changed the world of GO and engine-GO and revolutionized the Go engines advancing them by thousands of ELO!!
I wouldn't be so quick saying they've tricked us about their Chess related results.

I think lczero should just continue using the same exact methods AlphaZero used so that we can ask google why we can't reproduce their results if that turned out to be the case.

Well we didn't even try yet. Deepmind played 44 million games and LC0 is still on 5 million. Not to mention the net which still is a lot smaller than what Deepmind used. It will increase in the future of course.
So be a little patient. Even though i share about the same thoughts with you, i don't believe they have fooled us so i still have faith.

[CMCanavessi]

What do you think is the elo of Scorpio using the settings you just said? Because if it's 2400 then leela woud smoke it... We can try the same match with that config, if you're up to the challenge

Here is a tournament i made some time ago http://talkchess.com/forum/viewtopic.ph ... 56&t=66886

Code: Select all

 1.                       scorpio-mcts-ab16    375     25     25   1209    85.1%   -26    17.1% 
 2.                                 scorpio    357     23     23   1310    83.9%   -23    19.3% 
 3.                       scorpio-mcts-ab12    285     21     21   1478    77.9%   -25    16.8% 
 4.                       scorpio-mcts-ab10    187     21     21   1309    68.1%   -11    20.6% 
 5.                        scorpio-mcts-ab9    168     21     21   1308    66.3%   -10    19.6% 
 6.                        scorpio-mcts-ab8    131     20     20   1308    62.6%    -7    18.5% 
 7.                        scorpio-mcts-ab7     78     20     20   1308    57.2%    -3    18.3% 
 8.                        scorpio-mcts-ab6     27     20     20   1308    52.3%     0    20.1% 
 9.                        scorpio-mcts-ab5    -50     20     20   1308    44.3%     5    18.1% 
10.                        scorpio-mcts-ab4   -126     21     21   1308    36.9%    10    16.3% 
11.                        scorpio-mcts-ab3   -177     21     21   1308    32.3%    14    14.2% 
12.                        scorpio-mcts-min   -236     22     22   1310    26.1%    18    18.5% 
13.                        scorpio-mcts-ab2   -258     22     22   1308    24.7%    19    13.8% 
14.                        scorpio-mcts-ab1   -365     25     25   1310    16.6%    27    11.0% 
15.                         scorpio-mcts-ab   -389     25     25   1310    14.6%    29    11.7% 

You are matching it against #2 (default scorpio with alpha-beta). I am suggesting you match it against #12, which is purely MCTS but uses minmax backup operator. From the results of your tournament, it seems scorpio-mcts-min should be able to match it, don't you think ? Once you are done with that maybe you can go up the rank with the mcts+alpha-beta(ab) versions -- the #1 is actually a hybrid mcts-ab version though i belive this might be due to luck. The #1 mcts+ab is the one that will participate in TCEC anyway.

Daniel, this is Scorpio's config window in Arena:



Tell me exactly what I should put on each parameter and I'll just set it to your liking. There's also the .ini file if it needs tweaking.

[FWCC]

I have to agree,in matches in Arena LCZero loses 100% of the time.
I am beginning to wonder if progress is being made.She has yet to reach 5000 on the graph on the site.