Games for Training NNUE?

[Steve Maughan]

I assumed I'd create my own set of games for training a NNUE. However, it seems that'll take quite a long time. I assume I'll need about 2 million games (which will give ~150 million positions), I can dedicate a laptop with six cores and use Cutechess-cli to play game-in-30-seconds (i.e., one game per core per minute). The result is 8,640 games per day. If I'm not mistaken this will take 231 days to complete! That's a little longer than I had planned. And that's also before any processing and evaluation of the positions.

What have others done? Shorten the time control? More computing power? Use a known databases of games? Suck it up?

— Steve

[chrisw]

I assumed I'd create my own set of games for training a NNUE. However, it seems that'll take quite a long time. I assume I'll need about 2 million games (which will give ~150 million positions), I can dedicate a laptop with six cores and use Cutechess-cli to play game-in-30-seconds (i.e., one game per core per minute). The result is 8,640 games per day. If I'm not mistaken this will take 231 days to complete! That's a little longer than I had planned. And that's also before any processing and evaluation of the positions.

What have others done? Shorten the time control? More computing power? Use a known databases of games? Suck it up?

— Steve

Depth 9, 120 cores, gives about 2-3 Billion positions per week, I think, was how I started off.

20ms per position = 50 pos/second = 50 x 100 p/s = 3600 x 5000 p/h = 15mn/h = 400 m/d = 2.8 bn/week

Which should get you going. As you can see the critical factor is the cores. Realistically, to compete with the top engines below SF, you probably need 2 x 64 core PCs and then you can work with 1024 accumulator NNUEs.

With say 12 x, you’re looking at a few hundred million positions in reasonable time, and that means probably working with 256 accumulator. Positions need filtering (read about it).

Alternatively, if PC cores are lacking and you’re willing to handle the wrath of some purists, there’s 20bn plus positions downloadable if you hunt around.

My suggestion would be basically DIY, it’s more fun and satisfying, make your own positions (d9 recommended), a few hundred million, and try training a simple 256xNx1 nnue. There’s a lot to do, and you get no results until all is done.
Nnue training is almost the simplest part. Dense fully connected net, beginners coding problem.
Converting your positions into suitable form to say hello to the trainer is a major issue. You have to process very large numbers of positions and this will be the major time bottleneck. Done well (fast) and net training takes 1-2 days. Done as normal programming problem and you’re looking at 1-2 weeks or months. Use a mechanical HD and you can multiply those times a bit more.

Then you need the inference engine. Again simple enough (read up on how the NNUE is efficiently updatable and do it), all in C++
Then you’ll find C is too slow and start the relative torture of doing inference in SIMD instructions.

By then you may have something that works. NNs with bugs (either in training or inference) still work moderately well, so determining whether your NN “works” fully is a little problematic. Testing against engines with similar size nets should give parallel results if you got it all right.

[Henk]

If you use a database your hard disk may crash. That was my experience. Maybe better to recalculate.

[Steve Maughan]

Thanks Chris — that's all lot to take in but it all sounds logical.

I definitely want to do it from scratch — I don't see any point of doing otherwise.

And so the journey begins.

— Steve

[phhnguyen]

Depth 9, 120 cores, gives about 2-3 Billion positions per week, I think, was how I started off.

Just curious, what CPU/computer are you using? How much does it cost? Thanks.

Look like it is an expensive game to me!

[AndrewGrant]

I assumed I'd create my own set of games for training a NNUE. However, it seems that'll take quite a long time. I assume I'll need about 2 million games (which will give ~150 million positions), I can dedicate a laptop with six cores and use Cutechess-cli to play game-in-30-seconds (i.e., one game per core per minute). The result is 8,640 games per day. If I'm not mistaken this will take 231 days to complete! That's a little longer than I had planned. And that's also before any processing and evaluation of the positions.

What have others done? Shorten the time control? More computing power? Use a known databases of games? Suck it up?

— Steve

30 seconds per game is an insane amount of time. You can get away with a small number of fixed nodes per move. My strongest nets are with 40k nodes per move. 80k nodes actually lost elo.

[Rebel]

Thanks Chris — that's all lot to take in but it all sounds logical.

I definitely want to do it from scratch — I don't see any point of doing otherwise.

And so the journey begins.

— Steve

It depends, I guess how good your eval is. I started with Benjamin, fixed depth 6. Later tried depth=7 and it gave no improvement. And for every HCE engine it will be different. But by all means use fixed depth.

Good luck.

[chrisw]

Depth 9, 120 cores, gives about 2-3 Billion positions per week, I think, was how I started off.

Just curious, what CPU/computer are you using? How much does it cost? Thanks.

Look like it is an expensive game to me!

The largest is an AMD 64x

[lithander]

So you take completely random positions (but legal ones? any other filters?) and then you search them to depth 9 with an existing (probably HCE in the first iteration) evaluation. Under the assumption that the eval after a depth 9 search is more accurate than a static eval looking just at the position you then train your NN on this improved evaluation data and the evaluation to gets better? Then you repeat the process?

Is that the standard that everyone uses or have alternatives been tried?

E.g for my non-NN eval the tuner finds appropriate weights from a set of PGNs of selfplay games. From each game several positions are taken and scored under the assumption, that a game that white wins eventually contains only positions that are winning for white. Seems to work because mislabeled positions tend to cancel each other out and a small imbalance towards the ground truth remains. I wonder if it would also work for training NNUE type evals.

[Steve Maughan]

30 seconds per game is an insane amount of time. You can get away with a small number of fixed nodes per move. My strongest nets are with 40k nodes per move. 80k nodes actually lost elo.

Fascinating! It makes sense because you'll undoubtedly have a lot of crazy positions in that training set.

— Steve