Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Discussion of chess software programming and technical issues.

Moderators: hgm, Dann Corbit, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
David Carteau
Posts: 100
Joined: Sat May 24, 2014 7:09 am
Location: France
Full name: David Carteau
Contact:

Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by David Carteau » Mon Dec 07, 2020 9:50 am

A few days ago, I released a new version of my little engine Orion. After weeks of hard work, I managed to implement a working neural network trainer for NNUE-like architectures.

I decided to share my work by creating a library containing :
- a simple Python script for the trainer, based on Pytorch, and able to use (Nvidia) GPU if any to speed up the training ;
- a simple C program (with its header file) to convert and use the network file that have been generated.

The library is released under the MIT license : you can freely study it, copy/fork/clone it, or even embed it in your own projects.

My goal was not to design the most efficient code but to share simple code to help people not familiar with the NNUE concept to understand how it works. Despite being simple, the inference code supports incremental updates of the board state and uses some intrinsics to achieve the best possible performance.

Note that there are two major differences with the NNUE concept implemented in Stockfish :
- training produces float values, whereas Stockfish weights are integers : the inference is thus slower than in Stockfish ;
- the default network architecture uses 5 million weights (which is half of the 10 million weights used in Stockfish), but this can be easily changed.

Orion v0.8 uses the exact code that is published and it appears that first results are really good. It was a very exciting challenge and I'm happy to see that finally the training part works very well ;-)

What (ideally) remains to be done :
- find a better way to give insights on the training progress (I understood that cross-validation could help to estimate the quality of the current training) ;
- work on quantization, which could allow to produce small integers (int8) instead of float values, which could lead to better speed performance.

Do not hesitate to send me feedback or even contribute if you want : https://github.com/david-carteau/cerebrum

Regards, David.

Joerg Oster
Posts: 815
Joined: Fri Mar 10, 2006 3:29 pm
Location: Germany

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by Joerg Oster » Mon Dec 07, 2020 10:45 am

Thank you.
Very clean and readable code!
Jörg Oster

jdart
Posts: 4101
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by jdart » Mon Dec 07, 2020 3:12 pm

Thanks for making this available, especially under MIT License.

Madeleine Birchfield
Posts: 296
Joined: Tue Sep 29, 2020 2:29 pm
Location: Dublin, Ireland
Full name: Madeleine Birchfield

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by Madeleine Birchfield » Mon Dec 07, 2020 5:06 pm

Joerg Oster wrote:
Mon Dec 07, 2020 10:45 am
Thank you.
Very clean and readable code!
I'm not liking the tabs for indentation very much though, would rather prefer 4 spaces.

David Carteau
Posts: 100
Joined: Sat May 24, 2014 7:09 am
Location: France
Full name: David Carteau
Contact:

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by David Carteau » Tue Dec 08, 2020 7:28 am

Madeleine Birchfield wrote:
Mon Dec 07, 2020 5:06 pm
I'm not liking the tabs for indentation very much though, would rather prefer 4 spaces.
Fixed !

mar
Posts: 2320
Joined: Fri Nov 26, 2010 1:00 pm
Location: Czech Republic
Full name: Martin Sedlak

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by mar » Tue Dec 08, 2020 8:12 am

David Carteau wrote:
Tue Dec 08, 2020 7:28 am
Madeleine Birchfield wrote:
Mon Dec 07, 2020 5:06 pm
I'm not liking the tabs for indentation very much though, would rather prefer 4 spaces.
Fixed !
Noo :) Don't listen to this troll. I prefer tabs, because this way everybody can set any indentation level he wants - some indent to 2, 4 or 8 characters.
(Linux kernel uses tabs for a good reason)

And as a bonus, LZ-based compressors will compress source code with tabs better, because match distances will be smaller ;)
Martin Sedlak

David Carteau
Posts: 100
Joined: Sat May 24, 2014 7:09 am
Location: France
Full name: David Carteau
Contact:

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by David Carteau » Tue Dec 08, 2020 8:30 am

mar wrote:
Tue Dec 08, 2020 8:12 am
(...)
I prefer tabs, because this way everybody can set any indentation level he wants - some indent to 2, 4 or 8 characters.
(Linux kernel uses tabs for a good reason)
(...)
I also use tabs for all the good reasons you mentioned, but I must admit that in fact the source code on GitHub wasn't so readable, that's why I made the change. Is there a way to configure GitHub to visually shorten tabs ?

Rein Halbersma
Posts: 711
Joined: Tue May 22, 2007 9:13 am

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by Rein Halbersma » Tue Dec 08, 2020 9:40 am

mar wrote:
Tue Dec 08, 2020 8:12 am
Noo :) Don't listen to this troll. I prefer tabs, because this way everybody can set any indentation level he wants - some indent to 2, 4 or 8 characters.
(Linux kernel uses tabs for a good reason)

And as a bonus, LZ-based compressors will compress source code with tabs better, because match distances will be smaller ;)
It's not trolling to recommend 4 spaces for indentation. In fact, it's Pythonic to do so: https://www.python.org/dev/peps/pep-0008/#indentation
Most IDEs will have some PyLint plugin to automatically check it for you.

User avatar
Ozymandias
Posts: 1307
Joined: Sun Oct 25, 2009 12:30 am

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by Ozymandias » Tue Dec 08, 2020 10:11 am

How accurate is this parody on the subject?:

https://www.youtube.com/watch?v=SsoOG6ZeyUI

JohnWoe
Posts: 318
Joined: Sat Mar 02, 2013 10:31 pm

Re: Introducing the "Cerebrum" library (NNUE-like trainer and inference code)

Post by JohnWoe » Tue Dec 08, 2020 10:23 am

I think in general doing the exact opposite that troll tell you. Is a good start :lol:

Post Reply