Has Stockfish NNUE reached it's limit now?

[Nay Lin Tun]

In fact even 1 node per move can tell you whether newer nets are improving or not. (You can see newer nets are better than older nets even in 1 node per move)



Similarly, if Stockfish 12 is better than Stockfish 10, it is likely to be better in both 1 second per game and 1 hour per game.

[Dann Corbit]

If you look at the learning curves for the NN programs, they start learning very rapidly and gradually tail off until it is nearly level.
It may simply be the normal behavior.

[Dann Corbit]

Look at the learning graphs on page 4:
https://arxiv.org/pdf/1712.01815.pdf

[brianr]

Look at the learning graphs on page 4:
https://arxiv.org/pdf/1712.01815.pdf

Yes, it is the normal behavior and most NN learning graphs look like that.

However, that is for the particular net parameters of that training run.
There are many hyper-parameters that change from one net to another.
A somewhat extreme case is the totally different net architecture and input between AZ (Leela) type nets and NNUE nets.
The latter was a jump of about 200 Elo.
Even within the same net architecture well tuned parameters can improve a net by 1,000 Elo (from admittedly poor initial choices). But, nobody really knows without trying things and there are still many experiments to do.

SF may have reached something of a plateau with the current NNUE format, but something else will likely come along if enough people remain interested and keep experimenting. For instance, perhaps a somewhat larger net with input that is not currently provided.

Finally, the nets are roughly like the eval in A/B engines, and beyond the nets there are search improvements that will also continue to be found, especially with newer net architectures; opening books, tablebases, time management, and contempt might well all also continue to improve.

Of course, the next 100 Elo will likely always be increasingly more difficult to find.

[Jouni]

Suddenly +10 in NCM testing for latest SF version. Really unexpected and weird !

[Nay Lin Tun]

Or maybe go back to classical eval?

Probably yes, but it will be a long long way to go now ( Probably 3 years + working).

In the last Sufi 19, it was NNUE that just stepped in time to save Stockfish team. ( Lc0 vs SF Classical ended up 59-41 in bonus match).

And in cccc, Classical lost to Lc0 in 2 pawns handicap.

[pgn][Event "Odds Ladder (3|2)"]
[Site "?"]
[Date "2020.12.14"]
[Round "1"]
[White "Lc0"]
[Black "Stockfish Classic"]
[Result "1-0"]
[ECO "A00"]
[GameDuration "00:06:15"]
[GameEndTime "2020-12-14T18:18:15.145 PST"]
[GameStartTime "2020-12-14T18:11:59.558 PST"]
[Opening "Anti-Borg (Desprez) Opening"]
[PlyCount "79"]
[TimeControl "180+2"]

1. h4 Nf6 2. h5 Nxh5 3. a4 Nf6 4. a5 Nc6 5. Nf3 Nxa5 6. Ng1 Nc6 7. Nf3 Ng8 8.
Ng1 Nb8 9. e4 g6 10. Nf3 d6 11. d4 Bg7 12. Bc4 h6 13. Nc3 e6 14. Be3 Nf6 15. Qe2
Ng4 16. Bf4 Nc6 17. O-O-O Qe7 18. Bg3 O-O 19. Bh4 Qe8 20. Nb5 Qd7 21. e5 d5 22.
Nh2 Nxh2 23. Rxh2 dxc4 24. Bf6 Nxe5 25. dxe5 Qxb5 26. Qe3 c3 27. bxc3 Qa6 28.
Rd3 g5 29. Rh5 Qa1+ 30. Kd2 Qf1 31. Rxg5 hxg5 32. Qxg5 Qxf2+ 33. Kc1 Qg1+ 34.
Rd1 Qe3+ 35. Qxe3 Rd8 36. Rxd8+ Kh7 37. Qe4+ Kh6 38. Qh4+ Kg6 39. Qg5+ Kh7 40.
Qxg7# 1-0 [/pgn]

[Ozymandias]

Suddenly +10 in NCM testing for latest SF version. Really unexpected and weird !

Not just the one patch, 16 points in three consecutive ones to get a new high (+5 over previous record). Still, that's 5 Elo points since October 5th.

[supersharp77]

Or maybe go back to classical eval?

It is a possibilty....ongoing RR test shows SugaR NN scoring about thesame as Crystal NNUE and Corechess NNUE....it is possible Stockfish has taken a "wrong turn" but still way ahead of LC0 on CPU (LC0 is closing in)....so that's a plus!

[kinderchocolate]

What's product roadmap of Stockfish NNUE development? What's supposed to be done?

[kinderchocolate]

No progress in any list for some weeks. May be next step is a bigger net!?

Sergio Vieri was the only person on the Stockfish team who really knew how to train nets, and when he suddenly stopped training nobody else could really step in and replicate his efforts. So they've had to start from scratch.

Seriously, nobody else is doing it? The Discoard channel is always busy...