New sf+nnue play-only compiles

[Laskos]

Just to clarify any confusion in this thread...

We're seeing different Elo results presented
Those are of course the results of 'nnue' vs sf-dev test, and are very dependent on exactly which nn.bin is being tested
In the case of Larry and Mark, this detail is not indicated

This has much less to do with the quality of the compile (except for Ed's test of course), in which it's perfectly clear what's being tested.

My recommendation for a simple method of testing which compile is fastest on a particular system is:
1. make sure both binaries being compared load the same NNUE eval file (normally nn.bin)
2. type 'ucinewgame' and verify the nn.bin is found and loaded
(not needed for my recent compiles in which the nn.bin is loaded at startup)
3. run 'bench' at least twice for each compile, and average the result

(I know most here already know this, please don't be offended...I'm including it for clarity, and in case it helps someone).

Also, as Henk pointed out...to compare to a nodchip compile, make sure to select a 'nnue' version and make sure to type 'ucinewgame' to load the nn.bin before running 'bench'.

I believe Laskos 'nnue' vs sf-dev results with the newest SV net are significant...
I've been using ultra-fast for many years and have great confidence that ultra-fast results scale down in a meaningful way as TC increases.
At this point, I believe it's safe to say that nnue is pushing +60 elo or more? (Ed's results have it at +65)
which is fantastic considering it was at -50 to sf-dev in the very beginning, and just +30 just a short time ago

With the latest net (1817) vs. latest SF (july 17) at same 0.5' + 0.5", one thread, Hert lowdraw book, I got 60.5 out of 100, +74 elo, so far.

Test finished with 308.5 out of 500 games (56% draws), which is +83 elo! Quite remarkable given the not too short tc and fast (5 GHz) comp.

With a later net (0109), 312.5 out of 500 (57% draws) for same test, +89 elo.

Finally something closer to my results. I don't quite understand recent +40 - +60 Elo results at fast TC, either the net is old or compile is old or the openings are flattening the difference.

[Rebel]

Hi all-
I released new PO (play-only) compiles
which seem to be significantly faster
...more than 10% on my system

Tested it, the usual 2000 games at 40m/20s, got the best result so far.
sergio-2344 57.2% with the old executable
Sergio-2344 59.3% with Norman's compile

Will test this net on CCRL blitz level, curious how it scales, 1000 games.

Decided to play 2000 games instead of 1000.

First 1000 finished, so far more than +70 elo.

Code: Select all

Finished game 1018 (sf11 vs Norman): 1/2-1/2 {Draw by 3-fold repetition}
Score of Norman vs sf11: 354 - 146 - 509  [0.603] 1009

Scales well.

Final result 59.7%.

Games: http://rebel13.nl/sergio-2344.7z

[Werner]

I repeated here a game with the play-only.exe and got a better result:

1 SFNNUE 270720 x64 SV2141 +70 +22/=76/-2 60.00% 60.0/100
2 Stockfish 11.0 x64 1CPU -70 +2/=76/-22 40.00% 40.0/100

1 SFNNUE 200719 SV2141 +45 +16/=81/-3 56.50% 56.5/100
2 Stockfish 11.0 x64 1CPU -45 +3/=81/-16 43.50% 43.5/100

[Rebel]

Sergio-0109, 5000 games, tc 40m/20s, 96 elo stronger than Stockfish 11.

Code: Select all

Score of Norman vs sf11: 2076 - 720 - 2204  [0.636] 5000
...      Norman playing White: 1225 - 262 - 1013  [0.693] 2500
...      Norman playing Black: 851 - 458 - 1191  [0.579] 2500
...      White vs Black: 1683 - 1113 - 2204  [0.557] 5000
Elo difference: 96.6 +/- 7.2, LOS: 100.0 %, DrawRatio: 44.1 %
Finished match

Incredible.

What net next?

[lkaufman]

Sergio-0109, 5000 games, tc 40m/20s, 96 elo stronger than Stockfish 11.

Code: Select all

Score of Norman vs sf11: 2076 - 720 - 2204  [0.636] 5000
...      Norman playing White: 1225 - 262 - 1013  [0.693] 2500
...      Norman playing Black: 851 - 458 - 1191  [0.579] 2500
...      White vs Black: 1683 - 1113 - 2204  [0.557] 5000
Elo difference: 96.6 +/- 7.2, LOS: 100.0 %, DrawRatio: 44.1 %
Finished match

Incredible.

What net next?

At the much slower tc of 5' + 3" (one thread, 5 GHz i7, net 1218, 244 games, 75% draws) I got +60 elo over SF July 17, quite good for that TC.

[Ozymandias]

For some reason, this compile loses 100% of the games under cutechess-cli.

This is the log file.

This is the PGN.

This is the engines.json config for this engine:

Code: Select all

  { 
    "command" : "sf+nnue-po.270720.halfkp_256x2-32-32.x64.bmi2.exe", 
    "name" : "nnue", 
    "options" : [ 
      { 
        "name" : "Debug Log File", 
        "value" : "log.txt" 
      }  
    ] 
  }, 

This is the cli command:

Code: Select all

cutechess-cli -engine conf=S11 tc=inf depth=13 -engine conf=nnue tc=inf depth=13 -openings file=SALC_V5_hc_10m.pgn start=1 -site test -event 10 -games 2 -rounds 5 -repeat -concurrency 1 -pgnout run.pgn -recover -resign movecount=1 score=300 -draw movenumber=25 movecount=5 score=5 -each proto=uci

No tablebases.

[JohnS]

For some reason, this compile loses 100% of the games under cutechess-cli.

This is the log file.

This is the PGN.

This is the engines.json config for this engine:

Code: Select all

  { 
    "command" : "sf+nnue-po.270720.halfkp_256x2-32-32.x64.bmi2.exe", 
    "name" : "nnue", 
    "options" : [ 
      { 
        "name" : "Debug Log File", 
        "value" : "log.txt" 
      }  
    ] 
  }, 

This is the cli command:

Code: Select all

cutechess-cli -engine conf=S11 tc=inf depth=13 -engine conf=nnue tc=inf depth=13 -openings file=SALC_V5_hc_10m.pgn start=1 -site test -event 10 -games 2 -rounds 5 -repeat -concurrency 1 -pgnout run.pgn -recover -resign movecount=1 score=300 -draw movenumber=25 movecount=5 score=5 -each proto=uci

No tablebases.

Probably not loading net.

[Ozymandias]

Probably not loading net.

When firing up the command line, this is what I get:

[JohnS]

Probably not loading net.

When firing up the command line, this is what I get:

Why are you using depth instead of normal time control. Still not clear net is being loaded looking at he log file. Maybe specify net on cutechess command.

[Ozymandias]

What would be the syntax for specifying the net on cutechess command?