New sf+nnue play-only compiles

[lkaufman]

Just to clarify any confusion in this thread...

We're seeing different Elo results presented
Those are of course the results of 'nnue' vs sf-dev test, and are very dependent on exactly which nn.bin is being tested
In the case of Larry and Mark, this detail is not indicated

This has much less to do with the quality of the compile (except for Ed's test of course), in which it's perfectly clear what's being tested.

My recommendation for a simple method of testing which compile is fastest on a particular system is:
1. make sure both binaries being compared load the same NNUE eval file (normally nn.bin)
2. type 'ucinewgame' and verify the nn.bin is found and loaded
(not needed for my recent compiles in which the nn.bin is loaded at startup)
3. run 'bench' at least twice for each compile, and average the result

(I know most here already know this, please don't be offended...I'm including it for clarity, and in case it helps someone).

Also, as Henk pointed out...to compare to a nodchip compile, make sure to select a 'nnue' version and make sure to type 'ucinewgame' to load the nn.bin before running 'bench'.

I believe Laskos 'nnue' vs sf-dev results with the newest SV net are significant...
I've been using ultra-fast for many years and have great confidence that ultra-fast results scale down in a meaningful way as TC increases.
At this point, I believe it's safe to say that nnue is pushing +60 elo or more? (Ed's results have it at +65)
which is fantastic considering it was at -50 to sf-dev in the very beginning, and just +30 just a short time ago

With the latest net (1817) vs. latest SF (july 17) at same 0.5' + 0.5", one thread, Hert lowdraw book, I got 60.5 out of 100, +74 elo, so far.

Test finished with 308.5 out of 500 games (56% draws), which is +83 elo! Quite remarkable given the not too short tc and fast (5 GHz) comp.

[Zenmastur]

On threadripper I got a HUGE regression in speed when using either the POPC or AVX2 version listed here vice using stockfish.avx2.no-nnue.nnue-gen-sfen-from-original-eval.2020-07-19.exe. I'm getting 900K to 1.2Mnps vice 2Mnps on a single core.

Regards,

Zenmastur

Because you are comparing regular SF's speed with the NNUE.

no-nnue means that there is no NNUE present.
nnue-gen-sfen-from-original-eval means it can generate training data and that it will generate it from SF's original eval.
[/quote]

Yeah, that figures... guess I need to read the descriptions a of the .exe's

[Rebel]

Hi all-
I released new PO (play-only) compiles
which seem to be significantly faster
...more than 10% on my system

Tested it, the usual 2000 games at 40m/20s, got the best result so far.
sergio-2344 57.2% with the old executable
Sergio-2344 59.3% with Norman's compile

Will test this net on CCRL blitz level, curious how it scales, 1000 games.

Decided to play 2000 games instead of 1000.

First 1000 finished, so far more than +70 elo.

Code: Select all

Finished game 1018 (sf11 vs Norman): 1/2-1/2 {Draw by 3-fold repetition}
Score of Norman vs sf11: 354 - 146 - 509  [0.603] 1009

Scales well.

[lkaufman]

Just to clarify any confusion in this thread...

We're seeing different Elo results presented
Those are of course the results of 'nnue' vs sf-dev test, and are very dependent on exactly which nn.bin is being tested
In the case of Larry and Mark, this detail is not indicated

This has much less to do with the quality of the compile (except for Ed's test of course), in which it's perfectly clear what's being tested.

My recommendation for a simple method of testing which compile is fastest on a particular system is:
1. make sure both binaries being compared load the same NNUE eval file (normally nn.bin)
2. type 'ucinewgame' and verify the nn.bin is found and loaded
(not needed for my recent compiles in which the nn.bin is loaded at startup)
3. run 'bench' at least twice for each compile, and average the result

(I know most here already know this, please don't be offended...I'm including it for clarity, and in case it helps someone).

Also, as Henk pointed out...to compare to a nodchip compile, make sure to select a 'nnue' version and make sure to type 'ucinewgame' to load the nn.bin before running 'bench'.

I believe Laskos 'nnue' vs sf-dev results with the newest SV net are significant...
I've been using ultra-fast for many years and have great confidence that ultra-fast results scale down in a meaningful way as TC increases.
At this point, I believe it's safe to say that nnue is pushing +60 elo or more? (Ed's results have it at +65)
which is fantastic considering it was at -50 to sf-dev in the very beginning, and just +30 just a short time ago

With the latest net (1817) vs. latest SF (july 17) at same 0.5' + 0.5", one thread, Hert lowdraw book, I got 60.5 out of 100, +74 elo, so far.

Test finished with 308.5 out of 500 games (56% draws), which is +83 elo! Quite remarkable given the not too short tc and fast (5 GHz) comp.

With a later net (0109), 312.5 out of 500 (57% draws) for same test, +89 elo.

[cc2150dx]

Hi all-
I released new PO (play-only) compiles
which seem to be significantly faster
...more than 10% on my system

Intel® Core™ i9-9900K Processor

Bench was run just once, not averaged after several runs, so these are just estimates:

Code: Select all

PO
-------------------------
sf+nnue-po.270720.halfkp_256x2-32-32.x64.bmi2.exe
Total time (ms) : 2289
Nodes searched  : 3355738
Nodes/second    : 1466027

sf+nnue-po.270720.halfkp_256x2-32-32.x64.avx2.exe
Total time (ms) : 2305
Nodes searched  : 3355738
Nodes/second    : 1452071

sf+nnue-po.270720.halfkp_256x2-32-32.x64.popc.exe
Total time (ms) : 2684
Nodes searched  : 3355738
Nodes/second    : 1247022
-------------------------

NODCHIP
stockfish.bmi2.halfkp_256x2-32-32.profile-nnue.2020-07-19.exe
Total time (ms) : 2608
Nodes searched  : 4049933
Nodes/second    : 1552888

stockfish.avx2.halfkp_256x2-32-32.profile-nnue.2020-07-19.exe
Total time (ms) : 2609
Nodes searched  : 4049933
Nodes/second    : 1552293

stockfish.sse42.halfkp_256x2-32-32.profile-nnue.2020-07-19.exe
Total time (ms) : 3041
Nodes searched  : 4049933
Nodes/second    : 1331776
-------------------------

AIO
sf+nnue-aio.270720.halfkp_256x2-32-32.x64.bmi2.exe
Total time (ms) : 3086
Nodes searched  : 4049933
Nodes/second    : 1312356

sf+nnue-aio.270720.halfkp_256x2-32-32.x64.avx2.exe
Total time (ms) : 3124
Nodes searched  : 4049933
Nodes/second    : 1296393

sf+nnue-aio.270720.halfkp_256x2-32-32.x64.popc.exe
Total time (ms) : 3661
Nodes searched  : 4049933
Nodes/second    : 1106236

https://github.com/FireFather/sf-nnue/releases

I hope they prove fast on other systems as well...
PS they have not been widely tested

I inserted a 30 MB nn.bin file into the 'eval' folder and got this error:

SF+NNUE PO 270720 x64 avx2 by Stockfish+NNUE team
Error! ./eval/nn.bin not found or wrong format

any suggestions?

Thanks

[kranium]

Hi all-
I released new PO (play-only) compiles
which seem to be significantly faster
...more than 10% on my system

Intel® Core™ i9-9900K Processor

Bench was run just once, not averaged after several runs, so these are just estimates:

https://github.com/FireFather/sf-nnue/releases

I hope they prove fast on other systems as well...
PS they have not been widely tested

I inserted a 30 MB nn.bin file into the 'eval' folder and got this error:

SF+NNUE PO 270720 x64 avx2 by Stockfish+NNUE team
Error! ./eval/nn.bin not found or wrong format

any suggestions?

Thanks

These PO compiles are for halfkp_256x2-32-32 eval nets only, as indicated via the filename
(halfkp_256x2-32-32 is 20 MB)

Looks like you are trying to load a bigger 30 MB halfkp_384x2-32-32 net...

all of the strong nets being tested in this thread are 256
and come from here:
https://www.comp.nus.edu.sg/~sergio-v/nnue/

[cc2150dx]

Hi all-
I released new PO (play-only) compiles
which seem to be significantly faster
...more than 10% on my system

Intel® Core™ i9-9900K Processor

Bench was run just once, not averaged after several runs, so these are just estimates:

https://github.com/FireFather/sf-nnue/releases

I hope they prove fast on other systems as well...
PS they have not been widely tested

I inserted a 30 MB nn.bin file into the 'eval' folder and got this error:

SF+NNUE PO 270720 x64 avx2 by Stockfish+NNUE team
Error! ./eval/nn.bin not found or wrong format

any suggestions?

Thanks

These PO compiles are for halfkp_256x2-32-32 eval nets only, as indicated via the filename
(halfkp_256x2-32-32 is 20 MB, not 30)

Looks like you are trying to load a bigger halfkp_384x2-32-32 net...

yes! I didn't read it right Do you know where I can get a good 384 which is Fritz/Chessbase compatible ?

Thanks

[kranium]

Hi all-
I released new PO (play-only) compiles
which seem to be significantly faster
...more than 10% on my system

Intel® Core™ i9-9900K Processor

Bench was run just once, not averaged after several runs, so these are just estimates:

https://github.com/FireFather/sf-nnue/releases

I hope they prove fast on other systems as well...
PS they have not been widely tested

I inserted a 30 MB nn.bin file into the 'eval' folder and got this error:

SF+NNUE PO 270720 x64 avx2 by Stockfish+NNUE team
Error! ./eval/nn.bin not found or wrong format

any suggestions?

Thanks

These PO compiles are for halfkp_256x2-32-32 eval nets only, as indicated via the filename
(halfkp_256x2-32-32 is 20 MB, not 30)

Looks like you are trying to load a bigger halfkp_384x2-32-32 net...

yes! I didn't read it right Do you know where I can get a good 384 which is Fritz/Chessbase compatible ?

Thanks

The only 384's I'm aware of were from Josh's Patreon site...but that was some time ago.
I'm pretty sure these new 256's from the sergio-v are strongest at the moment.
The nn.bin has nothing to do with Fritz/Chessbase compatibly...that's dependent on the binary.
Going to have a beer...you'll have to dig up the link yourself.

[cc2150dx]

Hi all-
I released new PO (play-only) compiles
which seem to be significantly faster
...more than 10% on my system

Intel® Core™ i9-9900K Processor

Bench was run just once, not averaged after several runs, so these are just estimates:

https://github.com/FireFather/sf-nnue/releases

I hope they prove fast on other systems as well...
PS they have not been widely tested

I inserted a 30 MB nn.bin file into the 'eval' folder and got this error:

SF+NNUE PO 270720 x64 avx2 by Stockfish+NNUE team
Error! ./eval/nn.bin not found or wrong format

any suggestions?

Thanks

These PO compiles are for halfkp_256x2-32-32 eval nets only, as indicated via the filename
(halfkp_256x2-32-32 is 20 MB, not 30)

Looks like you are trying to load a bigger halfkp_384x2-32-32 net...

yes! I didn't read it right Do you know where I can get a good 384 which is Fritz/Chessbase compatible ?

Thanks

The only 384's I'm aware of were from Josh's Patreon site...but that was some time ago.
I'm pretty sure these new 256's from the sergio-v are strongest at the moment.
The nn.bin has nothing to do with Fritz/Chessbase compatibly...that's dependent on the binary.
Going to have a beer...you'll have to dig up the link yourself.

What I meant was a exec that supports 384 net sizes...for example here: https://github.com/nodchip/Stockfish/releases. Anyway enjoy your beer

[carldaman]

Here:

https://github.com/nodchip/Stockfish/re ... 2020-07-19