Code: Select all
+--------------------+-------+-----------|-------+-------+-------+-------+
| ENGINE | ELO | +/- | GAMES | SCORE | AvOp | DRAWS |
+--------------------+-------+-----------|-------+-------+-------+-------+
| Orion 0.7.nnue x64 | 2953 | +22 -22 | 1000 | 80% | 2712 | 19% |
| Orion 0.7 x64 | 2762 | +19 -18 | 1000 | 57% | 2711 | 33% |
+--------------------+-------+-----------|-------+-------+-------+-------+
For this experiment, I used exactly the same test conditions than for my recent release (v0.7), with same opponents (total: 10), same number of games (total: 1000), same opening book, same time controls (40/1), etc.
I didn't want to simply copy/paste available C++ code, but rather to understand the network architecture and the way final evaluation is computed, so I decided to write my own NNUE implementation in C, compatible with the current Stockfish's networks.
Network loading and evaluation represent around 250 lines of code. The only thing which is not yet implemented is the "capture" feature. I must admit that, for the moment, I found this quite strange : how the associated weights are computed since learning is performed using only fens (so from a static view of the board) ?
For the test, I used the current 'best' network available at https://tests.stockfishchess.org/nns (which is to date 'nn-82215d0fd0df.nnue').
So, what are my thoughts ?
I'm super happy ! I knew that Orion's evaluation was weak, so I'm not surprised. I have now a better idea of what remains to be done regarding other parts of the engine, especially search, if I want to make progress. Having a 'fixed' (and known as one of the best) evaluation should give a good basis to improve the rest of the engine.
But, wait, am I saying that next releases of Orion will now use Stockfich evaluation networks ?!
No ! It wouldn't be satisfactory from an intellectual perspective. My goal has always been - and remains - to understand concepts, try to implement them on my side, and then start to play with, in the sense of "try to improve if possible" !
So what's next ?
Next weeks will be busy, and I cannot imagine releasing another version before Orion v0.7 has been tested on CCRL 40/15 list (I put a lot of efforts on this version !). And, as said above, I cannot imagine releasing a version relying on a network not being built by myself !
If it was the case, what will be the benefit for me ? For sure, a better ranking. But it doesn't correspond at all to my wishes. I want to understand and experiment by myself !
Next steps for Orion will be : try to understand how to train networks, build my own trainer, try to mix concepts (the return of PBIL ?!), try to play with network architectures, implement SMP (!).
Then, a new version may see the light of day. I think, now that the code exists, that the next version will embed the capacity to use Stockfish networks, but not by default. This will offer the possibility for testers to play and compare both Orion's evaluations, or compare Orion with other engines - both using same networks : this could be a unexplored and new way to test (and rank ?) engines
One thing is sure : Orion won't use by default the evaluation of another engine. It shall remain a 100% original work (with the notable exception - for the moment !? - of Syzygy support introduced in the last release). What would be the a world where all engines would use the same eval ?! Competition requires to try different and distinct approaches !
To all engine developers : what are your plans ? Do you also currently work on NNUE integration in your engine ? Do you plan to replace (or mix) your engine evaluation by NNUE in the next future ? Does anyone has thoughts on the "capture" feature ?
Final note : for whom who are interested, here is an idea of the nps drop (using the Orion's builtin 'bench' command):
Code: Select all
+--------------------+------------------|----------------------------+
| ENGINE | popcount version | popcount+avx2+bmi2 version |
+--------------------+------------------|----------------------------+
| Orion 0.7.nnue x64 | 128 kn/s | 160 kn/s |
| Orion 0.7 x64 | 818 kn/s | 861 kn/s |
+--------------------+------------------|----------------------------+
