In fact even 1 node per move can tell you whether newer nets are improving or not. (You can see newer nets are better than older nets even in 1 node per move)
Similarly, if Stockfish 12 is better than Stockfish 10, it is likely to be better in both 1 second per game and 1 hour per game.
Has Stockfish NNUE reached it's limit now?
Moderators: hgm, Rebel, chrisw
-
- Posts: 708
- Joined: Mon Jan 16, 2012 6:34 am
-
- Posts: 12541
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Has Stockfish NNUE reached it's limit now?
If you look at the learning curves for the NN programs, they start learning very rapidly and gradually tail off until it is nearly level.
It may simply be the normal behavior.
It may simply be the normal behavior.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 12541
- Joined: Wed Mar 08, 2006 8:57 pm
- Location: Redmond, WA USA
Re: Has Stockfish NNUE reached it's limit now?
Look at the learning graphs on page 4:
https://arxiv.org/pdf/1712.01815.pdf
https://arxiv.org/pdf/1712.01815.pdf
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
-
- Posts: 536
- Joined: Thu Mar 09, 2006 3:01 pm
Re: Has Stockfish NNUE reached it's limit now?
Yes, it is the normal behavior and most NN learning graphs look like that.Dann Corbit wrote: ↑Mon Dec 14, 2020 8:59 pm Look at the learning graphs on page 4:
https://arxiv.org/pdf/1712.01815.pdf
However, that is for the particular net parameters of that training run.
There are many hyper-parameters that change from one net to another.
A somewhat extreme case is the totally different net architecture and input between AZ (Leela) type nets and NNUE nets.
The latter was a jump of about 200 Elo.
Even within the same net architecture well tuned parameters can improve a net by 1,000 Elo (from admittedly poor initial choices). But, nobody really knows without trying things and there are still many experiments to do.
SF may have reached something of a plateau with the current NNUE format, but something else will likely come along if enough people remain interested and keep experimenting. For instance, perhaps a somewhat larger net with input that is not currently provided.
Finally, the nets are roughly like the eval in A/B engines, and beyond the nets there are search improvements that will also continue to be found, especially with newer net architectures; opening books, tablebases, time management, and contempt might well all also continue to improve.
Of course, the next 100 Elo will likely always be increasingly more difficult to find.
-
- Posts: 3291
- Joined: Wed Mar 08, 2006 8:15 pm
Re: Has Stockfish NNUE reached it's limit now?
Suddenly +10 in NCM testing for latest SF version. Really unexpected and weird !
Jouni
-
- Posts: 708
- Joined: Mon Jan 16, 2012 6:34 am
Re: Has Stockfish NNUE reached it's limit now?
Probably yes, but it will be a long long way to go now ( Probably 3 years + working).
In the last Sufi 19, it was NNUE that just stepped in time to save Stockfish team. ( Lc0 vs SF Classical ended up 59-41 in bonus match).
And in cccc, Classical lost to Lc0 in 2 pawns handicap.
[pgn][Event "Odds Ladder (3|2)"]
[Site "?"]
[Date "2020.12.14"]
[Round "1"]
[White "Lc0"]
[Black "Stockfish Classic"]
[Result "1-0"]
[ECO "A00"]
[GameDuration "00:06:15"]
[GameEndTime "2020-12-14T18:18:15.145 PST"]
[GameStartTime "2020-12-14T18:11:59.558 PST"]
[Opening "Anti-Borg (Desprez) Opening"]
[PlyCount "79"]
[TimeControl "180+2"]
1. h4 Nf6 2. h5 Nxh5 3. a4 Nf6 4. a5 Nc6 5. Nf3 Nxa5 6. Ng1 Nc6 7. Nf3 Ng8 8.
Ng1 Nb8 9. e4 g6 10. Nf3 d6 11. d4 Bg7 12. Bc4 h6 13. Nc3 e6 14. Be3 Nf6 15. Qe2
Ng4 16. Bf4 Nc6 17. O-O-O Qe7 18. Bg3 O-O 19. Bh4 Qe8 20. Nb5 Qd7 21. e5 d5 22.
Nh2 Nxh2 23. Rxh2 dxc4 24. Bf6 Nxe5 25. dxe5 Qxb5 26. Qe3 c3 27. bxc3 Qa6 28.
Rd3 g5 29. Rh5 Qa1+ 30. Kd2 Qf1 31. Rxg5 hxg5 32. Qxg5 Qxf2+ 33. Kc1 Qg1+ 34.
Rd1 Qe3+ 35. Qxe3 Rd8 36. Rxd8+ Kh7 37. Qe4+ Kh6 38. Qh4+ Kg6 39. Qg5+ Kh7 40.
Qxg7# 1-0 [/pgn]
-
- Posts: 1535
- Joined: Sun Oct 25, 2009 2:30 am
-
- Posts: 1242
- Joined: Sat Jul 05, 2014 7:54 am
- Location: Southwest USA
Re: Has Stockfish NNUE reached it's limit now?
It is a possibilty....ongoing RR test shows SugaR NN scoring about thesame as Crystal NNUE and Corechess NNUE....it is possible Stockfish has taken a "wrong turn" but still way ahead of LC0 on CPU (LC0 is closing in)....so that's a plus!
-
- Posts: 454
- Joined: Mon Nov 01, 2010 6:55 am
- Full name: Ted Wong
Re: Has Stockfish NNUE reached it's limit now?
What's product roadmap of Stockfish NNUE development? What's supposed to be done?
-
- Posts: 454
- Joined: Mon Nov 01, 2010 6:55 am
- Full name: Ted Wong
Re: Has Stockfish NNUE reached it's limit now?
Seriously, nobody else is doing it? The Discoard channel is always busy...Madeleine Birchfield wrote: ↑Wed Dec 09, 2020 5:19 pmSergio Vieri was the only person on the Stockfish team who really knew how to train nets, and when he suddenly stopped training nobody else could really step in and replicate his efforts. So they've had to start from scratch.