Orion 0.7 : NNUE experiment

[RubiChess]

I didn't want to simply copy/paste available C++ code, but rather to understand the network architecture and the way final evaluation is computed, so I decided to write my own NNUE implementation in C, compatible with the current Stockfish's networks.

...

But, wait, am I saying that next releases of Orion will now use Stockfich evaluation networks ?!

No ! It wouldn't be satisfactory from an intellectual perspective. My goal has always been - and remains - to understand concepts, try to implement them on my side, and then start to play with, in the sense of "try to improve if possible" !

Congrats and kudos for that!

You already went the way I will try to go for Rubi: Rewriting the complete NNUE code (to something I understand much better than this highly evolved C++ of the original) and within this learn the basics of NN.
This will be a hard and long way cause I don't have any knowledge about machine learning and NN yet (just a book that waits for reading) but it will be more worth than the "copy-and-paste in a rush" of some others.

Regards, Andreas

[mvanthoor]

This will be a hard and long way cause I don't have any knowledge about machine learning and NN yet (just a book that waits for reading) but it will be more worth than the "copy-and-paste in a rush" of some others.

Same here... but I even have to finish my engine first. Someday.

Because it's possible to reach at least 3400 ELO in a single-threaded alpha/beta search, I don't think that I'll be looking into techniques such as SMP and neural networks until my engine reaches at least around 2850 or even 3000, in a single-threaded a/b-search. That will take quite enough time already, after actually finishing it.

[David Carteau]

Thanks Andreas and Marcel for your feedback. Trying to understand concepts and then to implement them is a super challenge !

In the meanwhile, I carrefully looked at the Stockfish code to learn how intrinsics could speed up dot products computation. This picture helped me a lot to understand what was behind the obscure terms used :



I also implemented intrinsics for the ReLU layers, but with no real advantage in term of speed (so I decided to leave code commented).

I'm super excited with the results !!

Code: Select all

+--------------------+------------------|----------------------------+
| ENGINE             | popcount version | popcount+avx2+bmi2 version |
+--------------------+------------------|----------------------------+
| Orion 0.7.nnue x64 |     424 kn/s     |          578 kn/s          |
| Orion 0.7 x64      |     818 kn/s     |          861 kn/s          |
+--------------------+------------------|----------------------------+

The 'popcount' version now requires -mssse3 GCC's flag, which should not be a problem since 'popcount" instruction came with sse4 sets of instructions.

Speed of 'nnue' version is now around 51-67% of the 'classic' version ! On actual games, nps is more or less halved between two versions.

I'm going to launch a tournament (with always the same other engines) to see how it will impact the elo performance.

[David Carteau]

Here is the (spectacular !) result of the tournament :

Code: Select all

+--------------------+-------+-----------|-------+-------+-------+-------+
| ENGINE             |   ELO |       +/- | GAMES | SCORE |  AvOp | DRAWS |
+--------------------+-------+-----------|-------+-------+-------+-------+
| Orion 0.7.nnue x64 |  3091 |  +29  -27 |  1000 |   90% |  2712 |   10% |
| Orion 0.7 x64      |  2762 |  +19  -18 |  1000 |   57% |  2711 |   33% |
+--------------------+-------+-----------|-------+-------+-------+-------+

Which is... more than +300 elo !!

The rating must however be inflated due to the set of opponents, and the high score obtained against them (90%). Note that the tournament was run with the 'popcount' version, which is not the fastest implementation.

I could launch a new tournament with 3000+ elo engines, but for the moment, I will consider that my NNUE implementation is fast enough to switch to the training part. I will try to implement my own trainer, with the (first) objective to train a similar but smaller neural network than the Stockfish's one. The idea is not to get the highest possible ranking, but rather to try to see whether I can improve or not Orion's evaluation function by my own means

[ Edit to my previous post : source of image is here. ]

[Sylwy]

Waiting for your NNUE engine !

https://www.lrz.de/services/compute/cou ... tro_AI.pdf

[Frank Quisinsky]

Hi there,

the problem with NNUE is that it's harder to compare engine strength with non NNUE engines.

With the final result that all engines used NNUE should be run in an own rating list.

Means:
All engines should have the same conditions for engine testing (tournaments, rating list systems).
All with the same hashtables, all with 4-men or all with 5-men, all with the same book, all with contempt = 0 and so on.

In my opinion it make no sense to test NNUE engines with engines not used NNUE.

Or I can make the chaos perfect and start a bigger confusion if I test different engines with 4-men, different others with 5-men, or for group of engines I used 1 min. pro move for an other group 10 minutes / move and added all in one list.

For engine testing:
All should have the same conditions or nothing is to compare with others.

If programmers will start the way with NNUE, we have from this group of engines the last NON NNUE version for testing.
I think the NNUE engines (not important if a programmer goes here an own way or used the good known sources) should run in a new rating list.

Made no sense to compare:
Andscacs 1.95 NON NNUE with Igel 2.7.0 NNUE.

Really a big problem because a lot of engine programmers have strong engines but not working on it.

For a modern way of engine development (I think NNUE is interesting) we should go a modern way of testing. I will do that if more NNUE engines are available.

Best
Frank

[Sylwy]

Hi there,

the problem with NNUE is that it's harder to compare engine strength with non NNUE engines.

With the final result that all engines used NNUE should be run in an own rating list.

Gens una sumus. The motto of FIDE, the World Chess Federation - literally "we are one people", it's usually translated into English as "we are one family". Not two, or three, or four...........


Maybe better & clear test conditions.

From years the rating lists are full of nn-GPU chess engines. Without any problem. Take a look to the current top of CCRL-blitz rating list:



And yes, the era of classical chess engines is over. All will remain, in time, at the bottom of the rating lists. This is life. Only collectors remained. I'm one.

[Frank Quisinsky]

Hi,

not sure with "The era of classical chess engines is over".
Maybe for yourself?

Classical chess engines = wrong difination.
Right defination = computer chess engines

Not very interesting the extraction of CCRL rating list.
Because, most of that if the same ...
Very bad for all other engines in the list because the final result from such testing methods is an Elo inflation.
To tell the fortune with cards are maybe more interesting as to read Elo here?

But I think too that most of engines, still under development, will use NNUE.
Most of programmers with interest on NNUE are programmers using other good known ideas also.
This explain that many engines today started with 2.500 or an higher Elo.

If I remember the best amateurs around the year 2000 have 2.400 Elo.
Sure that not 50 or more genius-people was born in the last 20 years.

I think that is more the new era to use good known ideas.
Here I can give to 100% my agreement.
And NNUE is a fantastic example for it.
Can be used with a clear consicience.

In times Fruit are available ...
To used Fruit codes without to give an information about it are gentlemen like crime.
Many programmers will like more to used under a clear consicience as a gentlemen like crime.

That's my own opinion about it.
Again: 50 or more times a genius was born ...
I am far from believing it.

Again ...
I think we don't need chess programs stronger as Shredder 12 with around 2.800 Elo for self playing.
No human will have a chance vs. programs with higher Elo as 2850.

For analyzes of chess games stronger chess programs are interesting.
But I don't need 200 of such programs.

Sure if all the NNUE programs are later in an own rating list most of people have interest to test place 1 - maybe place 6 - 8. Because all of them are 400-700 Elo stronger as World Champion Carlsen. What can the people do with it. The people can say ... looking Nakamura vs. Carlson, so many mistakes are in the games. That's what people do with it without to understand the chess the engines can play.

NNUE isn't the end of the era from _classical chess engines_??!

Classical chess engines is wrong: "Computer chess engines" is more right.
NNUE and LCO are more: Special computer chess engines or we can say "Modern chess engines, used most of times good known ideas".

For LCO: I am thinking on the book about the topic (Nighmare programmer give it others in Leiden 1999 or 1998).
Not new ideas ... so better to write is:

Modern chess engines, used most of times good known ideas.
Classcial chess engines or better "Computer Chess Engines" should be programs the programmer used own ideas.

Copy and Paste in chess programming ...
Not an art in my eyes.

Best
Frank

[Sylwy]

Hi,

not sure with "The era of classical chess engines is over".
Maybe for yourself?

Classical chess engines = wrong difination.
Right defination = computer chess engines

Yes, I'm sure It's over. A few classic chess engines (100% handcrafted evaluation) will appear. Less and less.
Computer chess engines-all sorts (handcrafted evaluation, hybrid, pure NNUE, nn-GPU & many more) will exist for a long time. The classic ones (again, the handcrafted evaluation and even the hybrids) they will soon be history. Just my opinion !

Best
SilvianR

[MikeB]

Hi there,

the problem with NNUE is that it's harder to compare engine strength with non NNUE engines.

With the final result that all engines used NNUE should be run in an own rating list.

Means:
All engines should have the same conditions for engine testing (tournaments, rating list systems).
All with the same hashtables, all with 4-men or all with 5-men, all with the same book, all with contempt = 0 and so on.

In my opinion it make no sense to test NNUE engines with engines not used NNUE.

Or I can make the chaos perfect and start a bigger confusion if I test different engines with 4-men, different others with 5-men, or for group of engines I used 1 min. pro move for an other group 10 minutes / move and added all in one list.

For engine testing:
All should have the same conditions or nothing is to compare with others.

If programmers will start the way with NNUE, we have from this group of engines the last NON NNUE version for testing.
I think the NNUE engines (not important if a programmer goes here an own way or used the good known sources) should run in a new rating list.

Made no sense to compare:
Andscacs 1.95 NON NNUE with Igel 2.7.0 NNUE.

Really a big problem because a lot of engine programmers have strong engines but not working on it.

For a modern way of engine development (I think NNUE is interesting) we should go a modern way of testing. I will do that if more NNUE engines are available.

Best
Frank

HI Frank - you argument is a little bit silly -- how about ratings using check extensions and not using check extensions. It's a slippery slope my friend and it simply doesn't make sense. You're going to see an explosion of NN engines - with open source SF - as opposed to closed source Alpha Zero, it is the new paradigm.

All the best,
MikeB