LCZero: Progress and Scaling. Relation to CCRL Elo

[jkiliani]

Where is LCzero being trained? Are people buying GPU cards to train it at home?

It's currently being trained by a server located in Finland, although the main dev supervising the network training is from Netherlands. A few other people are also training networks for experimental purposes, but most contributors simply let their client run self-play games to send to the server. It's really not necessary for multiple people to train networks, since the speed of self-play game generation is the bottleneck to improvement.

[duncan]

Where is LCzero being trained? Are people buying GPU cards to train it at home?

It's currently being trained by a server located in Finland, although the main dev supervising the network training is from Netherlands. A few other people are also training networks for experimental purposes, but most contributors simply let their client run self-play games to send to the server. It's really not necessary for multiple people to train networks, since the speed of self-play game generation is the bottleneck to improvement.

do you know which people wrote the code for it ?

[jkiliani]

Where is LCzero being trained? Are people buying GPU cards to train it at home?

It's currently being trained by a server located in Finland, although the main dev supervising the network training is from Netherlands. A few other people are also training networks for experimental purposes, but most contributors simply let their client run self-play games to send to the server. It's really not necessary for multiple people to train networks, since the speed of self-play game generation is the bottleneck to improvement.

do you know which people wrote the code for it ?

Majority of the codebase is from Leela Zero, which means Gian-Carlo Pascutto, along with several other major Leela Zero contributors you can find at https://github.com/gcp/leela-zero/blob/master/AUTHORS. The initial portation to chess was done by Benedict Diamond, but it was Gary Linscott who got it working, along with Folkert Huizinga. Since then, a number of additional contributors have added to the codebase (me included).

[Robert Flesher]

Very early days, but current result for LCZero 127 on NVidia 1060 vs Colossus 2008b @ 15 sec / move below. Colossus is single Intel Broadwell core @ 4.2 GHz.

4 wins
2 losses
1 draw

for LCZero

Wow, on a good GPU and longer time control, LC0 rocks. It scales completely differently from standard engines, give it strong hardware and LTC, and it soares.

Stop saying things like this, I am trying very hard over here not to upgrade my GPU! You have no idea how tempted I am to spend money on this right now. I'm like a drug addict going through withdrawals.

Pretty same here, need at least an Nvidia 1060, but maybe even higher. Bad times .

I folded Kai and just could take it anymore. Went and bought a GTX 1060, only to find out I needed a SATA 15-Pin Male to Dual 4-Pin Molex Female Y Splitter which I thought I already had, if I do it's saying find me. So I ordered a new one did not feel like driving back to Microcenter to get it. It just kept calling me and calling me. This stuff has me addicted.

+1 I see a Nvidia Titan V coming home with me really soon.

[Albert Silver]

Very early days, but current result for LCZero 127 on NVidia 1060 vs Colossus 2008b @ 15 sec / move below. Colossus is single Intel Broadwell core @ 4.2 GHz.

4 wins
2 losses
1 draw

for LCZero

Wow, on a good GPU and longer time control, LC0 rocks. It scales completely differently from standard engines, give it strong hardware and LTC, and it soares.

Stop saying things like this, I am trying very hard over here not to upgrade my GPU! You have no idea how tempted I am to spend money on this right now. I'm like a drug addict going through withdrawals.

Pretty same here, need at least an Nvidia 1060, but maybe even higher. Bad times .

I folded Kai and just could take it anymore. Went and bought a GTX 1060, only to find out I needed a SATA 15-Pin Male to Dual 4-Pin Molex Female Y Splitter which I thought I already had, if I do it's saying find me. So I ordered a new one did not feel like driving back to Microcenter to get it. It just kept calling me and calling me. This stuff has me addicted.

+1 I see a Nvidia Titan V coming home with me really soon.

My testing here, still early, is showing just how enormous a difference the GPU level makes. I don't mean the CPU vs GPU results, which are known, but from a good GPU to very good. To do this, I am running identical match (and openings) between Leela NN202 on my desktop and laptop. The CPU is almost identical (Nebula scores 8900KNPS vs 9050KNPS) while the GPU has Leela preforming about 1950 NPS on laptop (GTX980M 8GB) and 2270 on desktop (GTX1060 6GB). The scaling is not even close to comparable. On Desktop, Leela clobbered Nebula 72-28 (+166 Elo), but on laptop they are very close so far. Will report once it is done.

[Robert Flesher]

Very early days, but current result for LCZero 127 on NVidia 1060 vs Colossus 2008b @ 15 sec / move below. Colossus is single Intel Broadwell core @ 4.2 GHz.

4 wins
2 losses
1 draw

for LCZero

Wow, on a good GPU and longer time control, LC0 rocks. It scales completely differently from standard engines, give it strong hardware and LTC, and it soares.

Stop saying things like this, I am trying very hard over here not to upgrade my GPU! You have no idea how tempted I am to spend money on this right now. I'm like a drug addict going through withdrawals.

Pretty same here, need at least an Nvidia 1060, but maybe even higher. Bad times .

I folded Kai and just could take it anymore. Went and bought a GTX 1060, only to find out I needed a SATA 15-Pin Male to Dual 4-Pin Molex Female Y Splitter which I thought I already had, if I do it's saying find me. So I ordered a new one did not feel like driving back to Microcenter to get it. It just kept calling me and calling me. This stuff has me addicted.

+1 I see a Nvidia Titan V coming home with me really soon.

My testing here, still early, is showing just how enormous a difference the GPU level makes. I don't mean the CPU vs GPU results, which are known, but from a good GPU to very good. To do this, I am running identical match (and openings) between Leela NN202 on my desktop and laptop. The CPU is almost identical (Nebula scores 8900KNPS vs 9050KNPS) while the GPU has Leela preforming about 1950 NPS on laptop (GTX980M 8GB) and 2270 on desktop (GTX1060 6GB). The scaling is not even close to comparable. On Desktop, Leela clobbered Nebula 72-28 (+166 Elo), but on laptop they are very close so far. Will report once it is done.

Good to know! I am sure the wife won't see it that clearly

[Leo]

Where is LCzero being trained? Are people buying GPU cards to train it at home?

It's currently being trained by a server located in Finland, although the main dev supervising the network training is from Netherlands. A few other people are also training networks for experimental purposes, but most contributors simply let their client run self-play games to send to the server. It's really not necessary for multiple people to train networks, since the speed of self-play game generation is the bottleneck to improvement.

do you know which people wrote the code for it ?

Majority of the codebase is from Leela Zero, which means Gian-Carlo Pascutto, along with several other major Leela Zero contributors you can find at https://github.com/gcp/leela-zero/blob/master/AUTHORS. The initial portation to chess was done by Benedict Diamond, but it was Gary Linscott who got it working, along with Folkert Huizinga. Since then, a number of additional contributors have added to the codebase (me included).

Thanks for the info. Do you know the size of the learning file? It must be getting huge with millions of games.

[Leo]

Does anyone know what "200,000 playouts" means?

[Leo]

It looks like LCzero is at 2500 Elo. I think its just a matter of time before it becomes monster zero. Millions of games are being fed into the growing creature.

[Laskos]

In the meantime Leela breaks the 5400 elo from random play wall after 10 millions games played !

Will the progress be faster Kai with a bigger net ? What do you think ? And plz, don't shoot me big military guys.

I think there is a trade-off between the size of the net and the number of games (and playouts) one needs for learning per time. Larger nets are slower. I guess, until it starts to saturate, it's better to keep smaller nets, for faster games. A0 team had huge hardware for learning phase with an initial very big size of the net, and they had 40+ million games having very slow playouts, so the path is a bit different from LC0. But the final point I guess will be similar.
I also think that A0 team had many tries before finding the fastest way to reach that level using an adequate hardware, tries not mentioned in the paper.