Stockfish: what are recent game sources for NNUE?

[phhnguyen]

Just a curious question: what are the recent/new game sources for training new NNUEs of SF?

I knew SF code has a specific branch (tools) to create training games from self-play, but that branch has not been updated in a few years, and it can't run with recent SF code/NNUE. I knew the Lc0 team created a large database (for NNUE training) a few years ago. Thus, I don’t know what/where the new game sources for NNUE training are for recent SF versions.

(Many people suggested using SF Discord to ask about SF, but I have some bad experiences with it since some of my questions scrolled up and disappeared quickly, and some other questions got no answer for a very long time.)

[jdart]

lc0 data can be obtained from here: https://storage.lczero.org/files/training_data/.

In addition, Linmiao Xu aka linrock has posted some datasets in bullet format here: https://huggingface.co/linrock. These go up to 2024 and don't have the latest lc0 data. I have had no luck converting the latest lc0 data to bullet: I think the lc0 data format has changed and the converter (https://github.com/linrock/lc0-data-converter) has not been updated to read it properly.

[jshriver]

As of few months ago here are the sites that were given to me from their discord server.

https://huggingface.co/datasets/officia ... /tree/main
https://huggingface.co/linrock/datasets
https://huggingface.co/datasets/vondele ... /tree/main

[Euro_Pop]

I have a question about Stockfish,especially the new Stockfish18.
I have an old computer(a 15 year old Dell 64bit )and have loaded all the recent Stockfish,but i cannot load the new AVX only the so called very slow 64 bit that works on all computers.That is fine with me and yes,it works.But in regards to this "slowness"is that really have any importance in the long run if I am not playing fast time controls,no blitz ect,just say games like 40/2 or 20/4 and perhaps playing against other chess programs?I would think that giving the stockfish program extra and ample time to "think",that this "slowness"as the stockfish website states with the 64bit version(for older systems),that it would more than compensate for any slowness of move analysis?

I am no software expert or computer expert,but this "slowness"as it is supposed to be accoring to the stockfish website boils down to what? A severe disadvantage playing against a faster system online?I would wager not,that is,if i give Stocksfish 18 ample extra time to think out its moves.No,I am not interested in using infinite analysis...that would take too long for game results.So any help here?

[Peter Berger]

But in regards to this "slowness"is that really have any importance in the long run if I am not playing fast time controls,no blitz ect,just say games like 40/2 or 20/4 and perhaps playing against other chess programs?I would think that giving the stockfish program extra and ample time to "think",that this "slowness"as the stockfish website states with the 64bit version(for older systems),that it would more than compensate for any slowness of move analysis?

Your assumption is correct. Although this forum also has "Technical discussions" in its description you'll probably get much better answers for this kind of question in the "General topics" section though.

Why do you care?

[phhnguyen]

lc0 data can be obtained from here: https://storage.lczero.org/files/training_data/.

In addition, Linmiao Xu aka linrock has posted some datasets in bullet format here: https://huggingface.co/linrock. These go up to 2024 and don't have the latest lc0 data. I have had no luck converting the latest lc0 data to bullet: I think the lc0 data format has changed and the converter (https://github.com/linrock/lc0-data-converter) has not been updated to read it properly.

As of few months ago here are the sites that were given to me from their discord server.

https://huggingface.co/datasets/officia ... /tree/main
https://huggingface.co/linrock/datasets
https://huggingface.co/datasets/vondele ... /tree/main

Thank you all for the information. I got two surprises:
* The Lc0 team has been continuously creating game databases to train NNUEs (I thought it was created only once)
* The SF team has been continuously creating their NNUEs from Lc0's game databases. E.g., the latest SF version 18 used Lc0's game databases

Now I am curious:
* Why doesn't the SF team create training games itself? In particular, they have all the tools and code for those tasks, and the recent SF is much stronger than the one when they started with NNUE. In Discord, one explained that the quality of SF games is not as good as that of Lc0 games. Is that correct?
* Is it fair for matching between SF and Lc0 when SF NNUE is trained from Lc0 games? I remembered that Crafty required that all its clones do not match it in official tournaments. I knew there was no such requirement from the Lc0 team, but I was still curious about fairness.

[Eelco de Groot]

Thank you all for the information. I got two surprises:
* The Lc0 team has been continuously creating game databases to train NNUEs (I thought it was created only once)
* The SF team has been continuously creating their NNUEs from Lc0's game databases. E.g., the latest SF version 18 used Lc0's game databases

Now I am curious:
* Why doesn't the SF team create training games itself? In particular, they have all the tools and code for those tasks, and the recent SF is much stronger than the one when they started with NNUE. In Discord, one explained that the quality of SF games is not as good as that of Lc0 games. Is that correct?

I do not know about this stuff, so others, much better up to date will probably answer, but part of the story I would say is that Lc0 is just positionally better than Stockfish, that is the main reason it can compete? And a large benefit of NNUE is better opening handling IMO so the positional input there is important. You can not really beat Stockfish with tactics and tactics do not rely on the evaluation function at least not that much, apart from balancing material against positional characteristics. Anyway, Stockfish with NNUE now plays positionally like Lc0 so using their own game databases does not really change very much, its style will be governed by the NNUE now.

Small additional benefit, Lc0 will not suffer from Zugzwang blindness, not using Null move (NMP).

For your other question, this was raised by Rebel Ed Schroder, I have not followed that discussion at all, sorry guys, I do not think he changed his stance but he was more or less thrown out of Discord for raising it so that will point you about how others there (in Discord) think about it. It is not just Stockfish who is doing it by the way.

* Is it fair for matching between SF and Lc0 when SF NNUE is trained from Lc0 games? I remembered that Crafty required that all its clones do not match it in official tournaments. I knew there was no such requirement from the Lc0 team, but I was still curious about fairness.

[phhnguyen]

Thank you all for the information. I got two surprises:
* The Lc0 team has been continuously creating game databases to train NNUEs (I thought it was created only once)
* The SF team has been continuously creating their NNUEs from Lc0's game databases. E.g., the latest SF version 18 used Lc0's game databases

Now I am curious:
* Why doesn't the SF team create training games itself? In particular, they have all the tools and code for those tasks, and the recent SF is much stronger than the one when they started with NNUE. In Discord, one explained that the quality of SF games is not as good as that of Lc0 games. Is that correct?

I do not know about this stuff, so others, much better up to date will probably answer, but part of the story I would say is that Lc0 is just positionally better than Stockfish, that is the main reason it can compete? And a large benefit of NNUE is better opening handling IMO so the positional input there is important. You can not really beat Stockfish with tactics and tactics do not rely on the evaluation function at least not that much, apart from balancing material against positional characteristics. Anyway, Stockfish with NNUE now plays positionally like Lc0 so using their own game databases does not really change very much, its style will be governed by the NNUE now.

Small additional benefit, Lc0 will not suffer from Zugzwang blindness, not using Null move (NMP).

For your other question, this was raised by Rebel Ed Schroder, I have not followed that discussion at all, sorry guys, I do not think he changed his stance but he was more or less thrown out of Discord for raising it so that will point you about how others there (in Discord) think about it. It is not just Stockfish who is doing it by the way.

* Is it fair for matching between SF and Lc0 when SF NNUE is trained from Lc0 games? I remembered that Crafty required that all its clones do not match it in official tournaments. I knew there was no such requirement from the Lc0 team, but I was still curious about fairness.

Thanks for the interesting information!

I can't find any post in which Rebel raised that issue. However, I found a post a few years ago in which they said Lc0 games contributed +20 Elo and how good they trained SF's NNUE with both SF and Lc0 games. In that post, Vernon Crawford was the first person to raise the issue (of using Lc0 games for SF's NNUE in a tournament).