LCZero: Progress and Scaling. Relation to CCRL Elo

[nabildanial]

Reverting to an earlier version just because something went wrong is not smart.

You're actually right - letting random people commit random stuff and changing random constants without testing is not smat but rather incompetent.
Problems have been pointed out by Ronald, Alvaro and other people who actually wrote chess engines and who know the hell what they're talking about,
but you fanboys keep repeating the same nonsense over and over again, well good luck with that. You'll bury a very intersting project.

The PUCT change pull request was made by Gian-Carlo Pascutto (gcp), the one who made the original Leela Zero, that Leela Chess Zero is forked from. Is gcp a random person? Plus the PUCT change was tested a lot of times to prove its worth and it was proven that it gives more Elo. Also most of the devving discussions take place on Discord, in which "Ronald, Alvaro and other people who actually wrote chess engines" know nothing of and didn't take part in even a single discussion, what problems do they know? It also has been pointed out over and over again that training and devving (bugfixing and making improvements) are two separate things and computing power is a limiting factor to the latter. Training is not being bottlenecked at all as long as people are still running the training clients but for devving, they need dedicated machines to be able to make further improvements and experiments (thus the gofundme campaign). This is not the same as your common chess engine development where you have all your computing power towards devving.

[Milos]

Reverting to an earlier version just because something went wrong is not smart.

You're actually right - letting random people commit random stuff and changing random constants without testing is not smat but rather incompetent.
Problems have been pointed out by Ronald, Alvaro and other people who actually wrote chess engines and who know the hell what they're talking about,
but you fanboys keep repeating the same nonsense over and over again, well good luck with that. You'll bury a very intersting project.

The PUCT change pull request was made by Gian-Carlo Pascutto (gcp), the one who made the original Leela Zero, that Leela Chess Zero is forked from. Is gcp a random person? Plus the PUCT change was tested a lot of times to prove its worth and it was proven that it gives more Elo. Also most of the devving discussions take place on Discord, in which "Ronald, Alvaro and other people who actually wrote chess engines" know nothing of and didn't take part in even a single discussion, what problems do they know? It also has been pointed out over and over again that training and devving (bugfixing and making improvements) are two separate things and computing power is a limiting factor to the latter. Training is not being bottlenecked at all as long as people are still running the training clients but for devving, they need dedicated machines to be able to make further improvements and experiments (thus the gofundme campaign). This is not the same as your common chess engine development where you have all your computing power towards devving.

Stop BSing for god sake, you obviously know nothing either about chess development or ML. Martin is 100% right. Of those "great" developers there are exactly 2 that know anything gcp and Garry. The rest are fanboys and script kiddies who never in their life wrote any serious code let alone chess engine.

[Joost Buijs]

You're actually right - letting random people commit random stuff and changing random constants without testing is not smat but rather incompetent.
Problems have been pointed out by Ronald, Alvaro and other people who actually wrote chess engines and who know the hell what they're talking about,
but you fanboys keep repeating the same nonsense over and over again, well good luck with that. You'll bury a very intersting project.

The PUCT change pull request was made by Gian-Carlo Pascutto (gcp), the one who made the original Leela Zero, that Leela Chess Zero is forked from. Is gcp a random person? Plus the PUCT change was tested a lot of times to prove its worth and it was proven that it gives more Elo. Also most of the devving discussions take place on Discord, in which "Ronald, Alvaro and other people who actually wrote chess engines" know nothing of and didn't take part in even a single discussion, what problems do they know? It also has been pointed out over and over again that training and devving (bugfixing and making improvements) are two separate things and computing power is a limiting factor to the latter. Training is not being bottlenecked at all as long as people are still running the training clients but for devving, they need dedicated machines to be able to make further improvements and experiments (thus the gofundme campaign). This is not the same as your common chess engine development where you have all your computing power towards devving.

Stop BSing for god sake, you obviously know nothing either about chess development or ML. Martin is 100% right. Of those "great" developers there are exactly 2 that know anything gcp and Garry. The rest are fanboys and script kiddies who never in their life wrote any serious code let alone chess engine.

Exactly! You couldn't have said it better.

I've been following the discussion on Discord for some time, it is all about making random changes to code and parameters without properly testing anything. It is a pity because it really is an interesting project, when it continues like this I'm afraid that a lot of people c.q. hardware contributors will lose interest.

[jkiliani]

The PUCT change pull request was made by Gian-Carlo Pascutto (gcp), the one who made the original Leela Zero, that Leela Chess Zero is forked from. Is gcp a random person? Plus the PUCT change was tested a lot of times to prove its worth and it was proven that it gives more Elo. Also most of the devving discussions take place on Discord, in which "Ronald, Alvaro and other people who actually wrote chess engines" know nothing of and didn't take part in even a single discussion, what problems do they know? It also has been pointed out over and over again that training and devving (bugfixing and making improvements) are two separate things and computing power is a limiting factor to the latter. Training is not being bottlenecked at all as long as people are still running the training clients but for devving, they need dedicated machines to be able to make further improvements and experiments (thus the gofundme campaign). This is not the same as your common chess engine development where you have all your computing power towards devving.

Stop BSing for god sake, you obviously know nothing either about chess development or ML. Martin is 100% right. Of those "great" developers there are exactly 2 that know anything gcp and Garry. The rest are fanboys and script kiddies who never in their life wrote any serious code let alone chess engine.

Exactly! You couldn't have said it better.

I've been following the discussion on Discord for some time, it is all about making random changes to code and parameters without properly testing anything. It is a pity because it really is an interesting project, when it continues like this I'm afraid that a lot of people c.q. hardware contributors will lose interest.

A number of people here seem to have some misconceptions what this project actually entails. If we had some perfectly bug-free code and parameters given to us in the first place, that would be great: We could simply set up the self-play pipeline just like Google, and then not change anything until we reach Stockfish. Unfortunately we don't have that luxury, since there are only the diverse Deepmind papers to follow, which are not all that detailed in many places, and Leela Zero, which is optimised for Go and may not be the best template for a chess engine either. So the devs made their best guesses when starting Leela Chess, and then changed things once it became apparent that some parts of the code or pipeline did not work well. When you see something going wrong in such a setup, you're frequently forced to act on less than perfect information since otherwise, useless training games keep piling up which will rightly cause the hardware contributors to doubt you're using the project resources well. In some cases those changes backfired, but acquiring knowhow consists in large part of the mistakes you made, not just the successes. This project is a pioneer effort, and if we just give up when it doesn't turn out to be smooth sailing all the way, we could have saved the effort to start it in the first place. gcp stresses frequently that the goal of LZ (Go) is not to build the strongest Go engine possible, but to generate open-source knowledge about reinforcement learning.

When you develop something like Stockfish, or I presume most traditional engines, the process is more straightforward in that you make a code change, test whether that change causes a strength increase or regression, and merge it if there is a statistically significant increase only. Since Leela's strength depends on both the code and the neural net, attributing the effects of code changes is much harder, especially in cases of parameter changes that affect the reinforcement learning long-term. gcp is rather conservative in how he runs Leela Zero, preferring to change as little as possible to the engine except for cases where something is proven to be broken, and LZ also uses strict gating which makes regression highly unlikely.

While the devs here in this forum may in many cases have good ideas about some particular changes, I dare anyone to claim he knows the optimum parameter set for a deep learning chess engine. Even Deepmind will only have educated guesses which they don't publish, from their presumably many failed experiments before the ones they described in their articles. Whether Leela succeeds long term or not is impossible to say at this point, but I think it's a safe guess that success will take quite a while and many setbacks. Thankfully most project contributors understand that and run the self-play client anyway.

[noobpwnftw]

The PUCT change pull request was made by Gian-Carlo Pascutto (gcp), the one who made the original Leela Zero, that Leela Chess Zero is forked from. Is gcp a random person? Plus the PUCT change was tested a lot of times to prove its worth and it was proven that it gives more Elo. Also most of the devving discussions take place on Discord, in which "Ronald, Alvaro and other people who actually wrote chess engines" know nothing of and didn't take part in even a single discussion, what problems do they know? It also has been pointed out over and over again that training and devving (bugfixing and making improvements) are two separate things and computing power is a limiting factor to the latter. Training is not being bottlenecked at all as long as people are still running the training clients but for devving, they need dedicated machines to be able to make further improvements and experiments (thus the gofundme campaign). This is not the same as your common chess engine development where you have all your computing power towards devving.

Stop BSing for god sake, you obviously know nothing either about chess development or ML. Martin is 100% right. Of those "great" developers there are exactly 2 that know anything gcp and Garry. The rest are fanboys and script kiddies who never in their life wrote any serious code let alone chess engine.

Exactly! You couldn't have said it better.

I've been following the discussion on Discord for some time, it is all about making random changes to code and parameters without properly testing anything. It is a pity because it really is an interesting project, when it continues like this I'm afraid that a lot of people c.q. hardware contributors will lose interest.

Guess what, why this happens is probably because of those who have a clue about how it may work probably wouldn't bother to argue with those self-complacent experts in the future way of developing chess engines, unfortunately this looks more and more like a fairy tale even to them, now some of those even turned on DeepMind for not telling the caveats. Funny when people say something a while ago about their papers being unscientific, they got labeled as conspiracy theorists.

[mirek]

I plan to play 200 games to get a rough idea of how strong the current engine + net is, I'll post the results here.

I have to agree with Albert - after 100 games perf is -60 elo compared to ID 24x so fanboys can cry now, they should revert to 0.7 ASAP!

Reverting to an earlier version just because something went wrong is not smart. The correct procedure is to figure out what exactly changed, and only react to (and possibly revert) that. Rather high on the list of suspects is the change to PUCT, from 0.85 to 0.6, and (temporarily) the introduction of FPU reduction. Since there are also issues with the 50-move plane and training on individual positions happens too often, we don't know for sure if PUCT is really responsible, and will investigate it along with other possible causes for the regression.

Greetings, any update on the possibility of rollback? Now that the network is steadily recovering is the rollback still being considered, or is there already some final decision on that issue? Assuming the new nets will soon reach the old nets peak, is there any advantage in doing rollback instead of just continuing?

[jkiliani]

Greetings, any update on the possibility of rollback? Now that the network is steadily recovering is the rollback still being considered, or is there already some final decision on that issue? Assuming the new nets will soon reach the old nets peak, is there any advantage in doing rollback instead of just continuing?

A rollback seems no longer a given now that it looks like peak strength has been regained (from the Id 312 CCLS gauntlet and several other metrics). It may still happen, but likely only in the form of a partial rollback (e.g. old net but train it on new data). Also, more and more people favor simply continuing, since the potential benefit of a rollback is looking increasingly doubtful. Any long-term effects of the bugs and regression should be fixed once we move to a 256x20 network at the latest.

[mirek]

Thanks for the update.

[shrapnel]

I wish Google would simply Commercialize AlphaZero (directly or indirectly) and put everyone out of their misery.
Gamers and Testers would get their Dream Engine and Komodo Team, Houdart and Stockfish Team would reverse engineer the living daylights out of it and make their own versions.
At the very least it would put an end to the Era of alpha-beta Engines and we could all move on.

[Milos]

I wish Google would simply Commercialize AlphaZero (directly or indirectly) and put everyone out of their misery.
Gamers and Testers would get their Dream Engine and Komodo Team, Houdart and Stockfish Team would reverse engineer the living daylights out of it and make their own versions.
At the very least it would put an end to the Era of alpha-beta Engines and we could all move on.

And I wish there was a world peace and everyone was rich and happy, and probability for that event to happen is slightly higher than for what you wrote.
But don't worry, dreaming is always nice .