Lc0 with latest test30 nets is vastly superior positionally

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Rebel, chrisw

User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Laskos »

A new check with the positional test suite Openings200, included are NNUE engines too. Top places go to NN engines, which haven't appeared yet when I built the test suite in 2016.

EPD test using Polyglot. Time is hard cut-off at 1s/position:

Code: Select all

polyglot.exe polyglot.ini epd-test -epd Openings200.epd -min-time 1 -max-time 1 -depth-delta 0
Lc0 runs on RTX 2070, multithreaded engines on 4 cores.

Red engines are Lc0 NN. Blue engines are NNUE engines. Black engines are classical eval engines.
Solutions found out of 200:

Lc0 LS15: 152
Lc0 J92-190: 145
Lc0 SV1541: 141
SFdev Hybrid: 138
SFdev Pure: 132
SF11: 110
Nemorino 60: 105
Igel 280: 102
Shredder 12: 98
SFdev Classic: 96
Ethereal 1262: 96
Komodo 14: 90
Nemorino 60 Classic: 88
Ethereal 1250: 81
Andscacs 095: 79
Shredder 13: 74
Meph. Gideon Pro: 74
Shredder 10: 72
Rebel Century: 66
Shredder 6PB: 65
Fruit 21: 63
Meph. Amsterdam: 42
Sargon 1978: 38

Download link to the test suite:
https://gofile.io/d/oqOBxF
Top
Stephen Ham
Posts: 2488
Joined: Wed Mar 08, 2006 9:40 pm
Location: Eden Prairie, Minnesota
Full name: Stephen Ham

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Stephen Ham »

Hello Kai,

Please also test the latest iteration of Fat Fritz. I'm a fan, and curious how it compares to Leela and Stockfish NNUE.

All the best,
-Steve-
Top
Dann Corbit
Posts: 12540
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Dann Corbit »

I am running your test suite at 15 minutes per position with LC0 26.2 using the LC0 net used at TCEC
This link:
https://github.com/jhorthos/lczero-trai ... a-Training
Provides strong evidence that lc0.net.J92-190 is best.

Here is the output so far:

Code: Select all

Analysis of C:\Users\dannc\Downloads\Openings200.epd   
Analyzing engine: Lc0-26-2
10/4/2020 10:50:35 PM Level: 900 Seconds
   Source                    : List of EPD/PGN files
      C:\Users\dannc\Downloads\Openings200.epd
   Colours                   : White moves=True, Black moves=True
   Direction                 : forward
   Games (PGN)               : From 1 to 999999
   Moves                     : All moves
   Positions (EPD)           : All positions
   Use Engine(s)             : List
      Lc0-26-2
   Use seconds per move      : 900 Seconds per move
   Analysis Lines            : Minimum search depth=2
   Activate abort analysis   : False
  1) .. Ng8-f6          Ng8-f6  * 0 Seconds
  2) .. Bf8-e7, Nf6xe4  Nf6xe4  * 43 Seconds
  3) e4-e5, Bf1-d3      Bf1-d3  * 0 Seconds
  4) f2-f4, g2-g3       Bc1-a3   
  5) c2-c3, O-O         c2-c3   * 3 Seconds
  6) .. O-O, a7-a6      a7-a6   * 65 Seconds
  7) Qd1-b3             Qd1-b3  * 0 Seconds
  8) O-O                O-O     * 484 Seconds
  9) a2-a3, Qd1-g4      a2-a3   * 0 Seconds
 10) Qd1-g4, Ng1-f3     Qd1-g4  * 84 Seconds
 11) Ng1-f3             Ng1-f3  * 0 Seconds
 12) Bc1-e3             O-O      
 13) .. Bc8-f5, c6-c5   Bc8-f5  * 0 Seconds
 14) .. h7-h6, c6-c5    h7-h6   * 0 Seconds
 15) Ng1-f3             c2-c4    
 16) e5xd6              f2-f4    
 17) Bf1-e2             Bc1-e3   
 18) O-O                O-O     * 468 Seconds
 19) Ng1-f3             Nc3-b5   
 20) c2-c4              a2-a4    
 21) .. e7-e5, e7-e6    e7-e5   * 84 Seconds
 22) f2-f3              f2-f3   * 897 Seconds
 23) Bc1-g5             Nc3-d5   
 24) .. b7-b5           b7-b5   * 0 Seconds
 25) Nc3-a4             Nc3-a4  * 66 Seconds
 26) Qd1-d3, b2-b3      Qd1-d3  * 0 Seconds
 27) Ng1-e2, f2-f4      Ng1-e2  * 0 Seconds
 28) O-O                a2-a4    
 29) .. d6xe5           d6xe5   * 0 Seconds
 30) .. b7-b6, Qd8-e7   g7-g6    
 31) O-O                O-O     * 0 Seconds
 32) Rf1-e1             Rf1-e1  * 0 Seconds
 33) f2-f3              f2-f3   * 0 Seconds
 34) O-O-O              O-O-O   * 0 Seconds
 35) Ng1-f3             Ng1-f3  * 2 Seconds
 36) .. d5xc4, Qd8-c7   Nc6-a5   
 37) g2-g3              g2-g3   * 0 Seconds
 38) .. g5-g4           g5-g4   * 102 Seconds
 39) Ne5xd7             O-O      
 40) O-O, b2-b3         O-O     * 0 Seconds
 41) b2-b3              Rf1-e1   
 42) .. Bf8-d6, Bf8-e7, Qd8-b6  Nf6-h5   
 43) e2-e4              e2-e4   * 0 Seconds
Here is the log for the first position that differed from the proposed move:

Code: Select all

 4 f4 g3; c0 "2"; id "k 
    Searching move: f2-f4, g2-g3
    Best move (Lc0-26-2): Bc1-a3
    Not found in: 15:00
     2/3	00:00	           7	583	-0.02	Bb2  Bg7  g3
     2/4	00:00	          11	611	-0.03	Bb2  Bg7  g3  0-0
     3/5	00:00	          21	875	+0.02	Ba3  Nb4  Bb2  Bg7  a3
     3/6	00:00	          44	1k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5
     4/6	00:00	          51	1k	+0.05	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  Rb1
     4/7	00:00	          62	1k	+0.05	Bb2  Bg7  Nd2  Nb4  Ne4  Bxe5  0-0-0
     4/8	00:00	          86	2k	+0.01	Bb2  Bg7  Nd2  Nb4  Ne4  Bxe5  0-0-0
     5/8	00:00	         104	2k	-0.01	Bb2  Bg7  Nd2  Nb4  Ne4  Bxe5  0-0-0
     5/9	00:00	         133	2k	+0.05	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2
     5/10	00:00	         188	2k	+0.04	Ba3  Nb4  Bb2  Bg7  g3  0-0  a3  Nd5  Bg2
     6/11	00:00	         282	3k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rab8  Nd2
     6/12	00:00	         368	3k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rab8  Nd2  Ra8
     6/13	00:00	         475	4k	+0.03	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6
     6/14	00:00	         555	4k	+0.04	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Kb1
     7/15	00:00	         653	4k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5
     7/16	00:00	         687	4k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5
     7/19	00:00	         946	4k	+0.04	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5
     8/20	00:00	          1k	4k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5
     8/21	00:00	          2k	5k	+0.04	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     8/22	00:00	          2k	5k	+0.03	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     8/22	00:00	          2k	5k	+0.04	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  a3  Nd5  0-0-0  Bxe5  Bxe5  Qxe5  Nc5
     8/23	00:00	          3k	5k	+0.03	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  a3  Nd5  0-0-0  Bxe5  Bxe5  Qxe5  Nc5  Qa1+
     9/23	00:00	          3k	6k	+0.02	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  a3  Nd5  0-0-0  Bxe5  Bxe5  Qxe5  Nc5  Qa1+
     9/23	00:00	          5k	6k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     9/23	00:00	          5k	6k	 0.00	Bb2  Bg7  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Na4  Rb8  Nc5  Re7
     9/23	00:00	          5k	6k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     9/23	00:00	          6k	6k	 0.00	Bb2  Bg7  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Na4  Rb8  Nc5  Re7  Rc1
    10/23	00:00	          6k	6k	 0.00	Bb2  Bg7  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Na4  Rb8  Nc5  Re7  Rc1
    10/23	00:01	          6k	6k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  Bb7  h4
    10/24	00:01	         10k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  h4  Nf4  Qe3  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5
    10/25	00:01	         12k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  b4  Qf8
    11/25	00:01	         12k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  b4  Qf8  h4  Bxc4  h5
    11/26	00:02	         15k	6k	-0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5  h4  Bb7  h5  Bxf3  gxf3
    12/26	00:02	         18k	6k	-0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Rb8  Qc2  Nf4  g3  Ne6  Bh3  c5  Rhe1
    12/26	00:03	         19k	6k	-0.02	g3  Bg7  Bb2  0-0-0  Bg2  Rhe8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Rc1  Kb7  Bf3  a5  Na4  Kc7  Nb2
    12/26	00:03	         21k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Rb8  Qc2  Nf4  g3  Ne6  Bh3  c5  Rhe1
    12/29	00:03	         21k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Rb8  Qc2  Nf4  g3  Ne6  Bh3  c5  Rhe1
    12/30	00:03	         22k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa3+
    12/31	00:03	         22k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa1+  Kc2  Qa2+
    13/31	00:04	         29k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Nb6  h4  Rae8  f4  d6  h5  dxe5  Re1  Bc8  g4  e4  Nxe4  f5  gxf5  Bxf5
    13/32	00:05	         38k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  d6  Qg4  Nb6  Qg5  Rfe8  Qxe7  Rxe7  exd6  cxd6  Bxg7  Kxg7  c5  Bxf1
    13/35	00:06	         42k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  d6  Qg4  Nb6  Qg5  Rfe8  Qxe7  Rxe7  exd6  cxd6  Bxg7  Kxg7  c5  Bxf1
    13/36	00:06	         46k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa1+  Kc2  Qa2+
    14/36	00:06	         49k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa1+  Kc2  Qa2+
    14/37	00:09	         65k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  Rfe8  Re1  Nb6  Qe3  d5  c5  Bxf1  cxb6  Bxg2  bxa7  Ra8  Rh2  Be4  Nxe4  dxe4
    14/38	00:09	         65k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  Rfe8  Re1  Nb6  Qe3  d5  c5  Bxf1  cxb6  Bxg2  bxa7  Ra8  Rh2  Be4  Nxe4  dxe4
    15/38	00:10	         79k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  h4  dxc4  b4  Qf5  Qxf5  gxf5  g4  fxg4  h5  Rad8  Rxd8  Rxd8  Rg1  Bc8  Be2  Rd5  Bxg4  Bxg4  Rxg4+
    16/38	00:14	        104k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  Qf4  dxc4  Qh6  Qf8  Qxf8+  Kxf8  a4  Re4  h4  h5  f3  Re3  a5  Nd5  Bxc4  Bxc4
    16/39	00:14	        105k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  Qf4  dxc4  Qh6  Qf8  Qxf8+  Kxf8  a4  Re4  h4  h5  f3  Re3  a5  Nd5  Bxc4  Bxc4
    16/40	00:14	        106k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  Qf4  dxc4  Qh6  Qf8  Qxf8+  Kxf8  a4  Re4  h4  h5  f3  Re3  a5  Nd5  Bxc4  Bxc4
    16/41	00:16	        132k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    16/42	00:17	        133k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/43	00:17	        135k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/44	00:17	        135k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/45	00:21	        186k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/46	00:26	        224k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/47	00:26	        225k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/47	00:27	        248k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/47	00:27	        248k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/47	00:27	        249k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/47	00:32	        308k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/50	00:40	        382k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    19/50	00:41	        399k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    19/51	00:44	        438k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    19/54	00:56	        576k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    20/54	00:58	        600k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    20/54	01:03	        649k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    20/59	01:59	      1,207k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    21/59	02:03	      1,254k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    21/59	02:08	      1,314k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    21/64	02:39	      1,657k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    22/64	02:43	      1,713k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    22/65	02:43	      1,716k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    22/66	04:12	      2,752k	11k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    23/66	04:17	      2,805k	11k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    23/66	04:22	      2,869k	11k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    23/67	05:55	      4,093k	12k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/67	05:59	      4,144k	12k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/67	06:04	      4,207k	12k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/70	09:23	      7,058k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/70	09:27	      7,123k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/70	09:27	      7,123k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/70	09:27	      7,124k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/70	09:32	      7,209k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/71	13:39	     10,801k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    26/71	13:43	     10,855k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    26/71	13:48	     10,927k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    26/73	14:58	     11,953k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
   10/4/2020 11:50:41 PM, Time for this analysis: 00:15:00, Rated time: 15:43
I may do another pass with a smaller net using LC0, and yet another with bleeding edge SF to see how the three differ.

I did not forget about your request to give additional filtering to the data you requested, but I have come to the conclusion that I am not able.
There are two problems. The first is that not all data uses the same version of Stockfish. It is almost like a collection of different engines. The second (which is much more serious) is that the data is not all at the same depth.

So consider position X.
For position X, at depth 36 we have a score of +44 for move y
For position X, at depth 43 we have a score of +25 for move z

Now, +25 is a better score than +44 because the depth is significantly deeper (unless it was made with a very old Stockfish).
So how do we compare the +44 score with the +25 score? Any way that I can think of is utterly arbitrary.

My conclusion is that I do not know how to do what you want.
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Top
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Laskos »

Dann Corbit wrote: ↑Mon Oct 05, 2020 6:54 pm I am running your test suite at 15 minutes per position with LC0 26.2 using the LC0 net used at TCEC
This link:
https://github.com/jhorthos/lczero-trai ... a-Training
Provides strong evidence that lc0.net.J92-190 is best.

Here is the output so far:

Code: Select all

Analysis of C:\Users\dannc\Downloads\Openings200.epd   
Analyzing engine: Lc0-26-2
10/4/2020 10:50:35 PM Level: 900 Seconds
   Source                    : List of EPD/PGN files
      C:\Users\dannc\Downloads\Openings200.epd
   Colours                   : White moves=True, Black moves=True
   Direction                 : forward
   Games (PGN)               : From 1 to 999999
   Moves                     : All moves
   Positions (EPD)           : All positions
   Use Engine(s)             : List
      Lc0-26-2
   Use seconds per move      : 900 Seconds per move
   Analysis Lines            : Minimum search depth=2
   Activate abort analysis   : False
  1) .. Ng8-f6          Ng8-f6  * 0 Seconds
  2) .. Bf8-e7, Nf6xe4  Nf6xe4  * 43 Seconds
  3) e4-e5, Bf1-d3      Bf1-d3  * 0 Seconds
  4) f2-f4, g2-g3       Bc1-a3   
  5) c2-c3, O-O         c2-c3   * 3 Seconds
  6) .. O-O, a7-a6      a7-a6   * 65 Seconds
  7) Qd1-b3             Qd1-b3  * 0 Seconds
  8) O-O                O-O     * 484 Seconds
  9) a2-a3, Qd1-g4      a2-a3   * 0 Seconds
 10) Qd1-g4, Ng1-f3     Qd1-g4  * 84 Seconds
 11) Ng1-f3             Ng1-f3  * 0 Seconds
 12) Bc1-e3             O-O      
 13) .. Bc8-f5, c6-c5   Bc8-f5  * 0 Seconds
 14) .. h7-h6, c6-c5    h7-h6   * 0 Seconds
 15) Ng1-f3             c2-c4    
 16) e5xd6              f2-f4    
 17) Bf1-e2             Bc1-e3   
 18) O-O                O-O     * 468 Seconds
 19) Ng1-f3             Nc3-b5   
 20) c2-c4              a2-a4    
 21) .. e7-e5, e7-e6    e7-e5   * 84 Seconds
 22) f2-f3              f2-f3   * 897 Seconds
 23) Bc1-g5             Nc3-d5   
 24) .. b7-b5           b7-b5   * 0 Seconds
 25) Nc3-a4             Nc3-a4  * 66 Seconds
 26) Qd1-d3, b2-b3      Qd1-d3  * 0 Seconds
 27) Ng1-e2, f2-f4      Ng1-e2  * 0 Seconds
 28) O-O                a2-a4    
 29) .. d6xe5           d6xe5   * 0 Seconds
 30) .. b7-b6, Qd8-e7   g7-g6    
 31) O-O                O-O     * 0 Seconds
 32) Rf1-e1             Rf1-e1  * 0 Seconds
 33) f2-f3              f2-f3   * 0 Seconds
 34) O-O-O              O-O-O   * 0 Seconds
 35) Ng1-f3             Ng1-f3  * 2 Seconds
 36) .. d5xc4, Qd8-c7   Nc6-a5   
 37) g2-g3              g2-g3   * 0 Seconds
 38) .. g5-g4           g5-g4   * 102 Seconds
 39) Ne5xd7             O-O      
 40) O-O, b2-b3         O-O     * 0 Seconds
 41) b2-b3              Rf1-e1   
 42) .. Bf8-d6, Bf8-e7, Qd8-b6  Nf6-h5   
 43) e2-e4              e2-e4   * 0 Seconds
Here is the log for the first position that differed from the proposed move:

Code: Select all

 4 f4 g3; c0 "2"; id "k 
    Searching move: f2-f4, g2-g3
    Best move (Lc0-26-2): Bc1-a3
    Not found in: 15:00
     2/3	00:00	           7	583	-0.02	Bb2  Bg7  g3
     2/4	00:00	          11	611	-0.03	Bb2  Bg7  g3  0-0
     3/5	00:00	          21	875	+0.02	Ba3  Nb4  Bb2  Bg7  a3
     3/6	00:00	          44	1k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5
     4/6	00:00	          51	1k	+0.05	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  Rb1
     4/7	00:00	          62	1k	+0.05	Bb2  Bg7  Nd2  Nb4  Ne4  Bxe5  0-0-0
     4/8	00:00	          86	2k	+0.01	Bb2  Bg7  Nd2  Nb4  Ne4  Bxe5  0-0-0
     5/8	00:00	         104	2k	-0.01	Bb2  Bg7  Nd2  Nb4  Ne4  Bxe5  0-0-0
     5/9	00:00	         133	2k	+0.05	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2
     5/10	00:00	         188	2k	+0.04	Ba3  Nb4  Bb2  Bg7  g3  0-0  a3  Nd5  Bg2
     6/11	00:00	         282	3k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rab8  Nd2
     6/12	00:00	         368	3k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rab8  Nd2  Ra8
     6/13	00:00	         475	4k	+0.03	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6
     6/14	00:00	         555	4k	+0.04	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Kb1
     7/15	00:00	         653	4k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5
     7/16	00:00	         687	4k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5
     7/19	00:00	         946	4k	+0.04	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5
     8/20	00:00	          1k	4k	+0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5
     8/21	00:00	          2k	5k	+0.04	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     8/22	00:00	          2k	5k	+0.03	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     8/22	00:00	          2k	5k	+0.04	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  a3  Nd5  0-0-0  Bxe5  Bxe5  Qxe5  Nc5
     8/23	00:00	          3k	5k	+0.03	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  a3  Nd5  0-0-0  Bxe5  Bxe5  Qxe5  Nc5  Qa1+
     9/23	00:00	          3k	6k	+0.02	Bb2  Bg7  Nd2  Nb4  Ne4  0-0  a3  Nd5  0-0-0  Bxe5  Bxe5  Qxe5  Nc5  Qa1+
     9/23	00:00	          5k	6k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     9/23	00:00	          5k	6k	 0.00	Bb2  Bg7  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Na4  Rb8  Nc5  Re7
     9/23	00:00	          5k	6k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5
     9/23	00:00	          6k	6k	 0.00	Bb2  Bg7  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Na4  Rb8  Nc5  Re7  Rc1
    10/23	00:00	          6k	6k	 0.00	Bb2  Bg7  g3  0-0  Bg2  Rae8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Na4  Rb8  Nc5  Re7  Rc1
    10/23	00:01	          6k	6k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  Bb7  h4
    10/24	00:01	         10k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  h4  Nf4  Qe3  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5
    10/25	00:01	         12k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  b4  Qf8
    11/25	00:01	         12k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  b4  Qf8  h4  Bxc4  h5
    11/26	00:02	         15k	6k	-0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Nb6  Qd2  c5  h4  Bb7  h5  Bxf3  gxf3
    12/26	00:02	         18k	6k	-0.02	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Rb8  Qc2  Nf4  g3  Ne6  Bh3  c5  Rhe1
    12/26	00:03	         19k	6k	-0.02	g3  Bg7  Bb2  0-0-0  Bg2  Rhe8  0-0  Bxe5  Qxe5  Qxe5  Bxe5  Rxe5  cxd5  Bxf1  Kxf1  cxd5  Nc3  c6  Rc1  Kb7  Bf3  a5  Na4  Kc7  Nb2
    12/26	00:03	         21k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Rb8  Qc2  Nf4  g3  Ne6  Bh3  c5  Rhe1
    12/29	00:03	         21k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Nf3  Rb8  Qc2  Nf4  g3  Ne6  Bh3  c5  Rhe1
    12/30	00:03	         22k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa3+
    12/31	00:03	         22k	6k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa1+  Kc2  Qa2+
    13/31	00:04	         29k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Nb6  h4  Rae8  f4  d6  h5  dxe5  Re1  Bc8  g4  e4  Nxe4  f5  gxf5  Bxf5
    13/32	00:05	         38k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  d6  Qg4  Nb6  Qg5  Rfe8  Qxe7  Rxe7  exd6  cxd6  Bxg7  Kxg7  c5  Bxf1
    13/35	00:06	         42k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  d6  Qg4  Nb6  Qg5  Rfe8  Qxe7  Rxe7  exd6  cxd6  Bxg7  Kxg7  c5  Bxf1
    13/36	00:06	         46k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa1+  Kc2  Qa2+
    14/36	00:06	         49k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qxa3+  Kc2  Qa2+  Kc1  Qa1+  Kc2  Qa2+
    14/37	00:09	         65k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  Rfe8  Re1  Nb6  Qe3  d5  c5  Bxf1  cxb6  Bxg2  bxa7  Ra8  Rh2  Be4  Nxe4  dxe4
    14/38	00:09	         65k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rab8  h4  Rfe8  Re1  Nb6  Qe3  d5  c5  Bxf1  cxb6  Bxg2  bxa7  Ra8  Rh2  Be4  Nxe4  dxe4
    15/38	00:10	         79k	7k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  h4  dxc4  b4  Qf5  Qxf5  gxf5  g4  fxg4  h5  Rad8  Rxd8  Rxd8  Rg1  Bc8  Be2  Rd5  Bxg4  Bxg4  Rxg4+
    16/38	00:14	        104k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  Qf4  dxc4  Qh6  Qf8  Qxf8+  Kxf8  a4  Re4  h4  h5  f3  Re3  a5  Nd5  Bxc4  Bxc4
    16/39	00:14	        105k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  Qf4  dxc4  Qh6  Qf8  Qxf8+  Kxf8  a4  Re4  h4  h5  f3  Re3  a5  Nd5  Bxc4  Bxc4
    16/40	00:14	        106k	7k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  d5  Nf6+  Bxf6  exf6  Qc5  Qf4  dxc4  Qh6  Qf8  Qxf8+  Kxf8  a4  Re4  h4  h5  f3  Re3  a5  Nd5  Bxc4  Bxc4
    16/41	00:16	        132k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    16/42	00:17	        133k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/43	00:17	        135k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/44	00:17	        135k	8k	+0.01	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/45	00:21	        186k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rfe8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kf8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/46	00:26	        224k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/47	00:26	        225k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/47	00:27	        248k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    17/47	00:27	        248k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/47	00:27	        249k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/47	00:32	        308k	9k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    18/50	00:40	        382k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    19/50	00:41	        399k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    19/51	00:44	        438k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    19/54	00:56	        576k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    20/54	00:58	        600k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    20/54	01:03	        649k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    20/59	01:59	      1,207k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    21/59	02:03	      1,254k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    21/59	02:08	      1,314k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    21/64	02:39	      1,657k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    22/64	02:43	      1,713k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    22/65	02:43	      1,716k	10k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    22/66	04:12	      2,752k	11k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    23/66	04:17	      2,805k	11k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    23/66	04:22	      2,869k	11k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    23/67	05:55	      4,093k	12k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/67	05:59	      4,144k	12k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/67	06:04	      4,207k	12k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/70	09:23	      7,058k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/70	09:27	      7,123k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    24/70	09:27	      7,123k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/70	09:27	      7,124k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/70	09:32	      7,209k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    25/71	13:39	     10,801k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    26/71	13:43	     10,855k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    26/71	13:48	     10,927k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
    26/73	14:58	     11,953k	13k	 0.00	Ba3  Nb4  Bb2  Bg7  a3  Nd5  Nd2  0-0  0-0-0  Rae8  Qf3  Nb6  Ne4  Bxe5  Bxe5  Qxe5  Nf6+  Kh8  Nxe8  Qa1+  Kc2  Qa2+  Kc1  Qa1+
   10/4/2020 11:50:41 PM, Time for this analysis: 00:15:00, Rated time: 15:43
I may do another pass with a smaller net using LC0, and yet another with bleeding edge SF to see how the three differ.

I did not forget about your request to give additional filtering to the data you requested, but I have come to the conclusion that I am not able.
There are two problems. The first is that not all data uses the same version of Stockfish. It is almost like a collection of different engines. The second (which is much more serious) is that the data is not all at the same depth.

So consider position X.
For position X, at depth 36 we have a score of +44 for move y
For position X, at depth 43 we have a score of +25 for move z

Now, +25 is a better score than +44 because the depth is significantly deeper (unless it was made with a very old Stockfish).
So how do we compare the +44 score with the +25 score? Any way that I can think of is utterly arbitrary.

My conclusion is that I do not know how to do what you want.
Thanks Dann, I managed to somehow solve the issue with one of the Ferdy tools he is always kind to provide. Only that my own engine analyses are very weak compared to yours.
As for this test suite and Lc0, you are performing a humongous analysis and I am very curious about the results. Yes, J92-190 is probably the best net, at least at slower than Blitz TC, therefore in analysis too. I am not expecting much improvement with longer times on this suite with Lc0, but I guess at 15 minutes per positions, Stockfish_dev might score even better than Lc0, AB engines do improve with time quite a lot positionally. The test suite is also flawed beyond a certain threshold, I believe ~30 or even more positions have wrong solutions. The positions are taken from almost purely human databases with often dubious statistic of outcomes, the main criterion for selection to the suite. Maybe I will use your analyses with Lc0 and SF_dev to make some modifications to the suite, so keep me updated on the analysis. Thanks!
Top
User avatar
Laskos
Posts: 10948
Joined: Wed Jul 26, 2006 10:21 pm
Full name: Kai Laskos

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Laskos »

Stephen Ham wrote: ↑Sun Oct 04, 2020 11:57 pm Hello Kai,

Please also test the latest iteration of Fat Fritz. I'm a fan, and curious how it compares to Leela and Stockfish NNUE.

All the best,
-Steve-
I don't currently have the Fritz 17 GUI with Fat Fritz, when I will re-install it and get the updates to Fat Fritz, I will check. It seemed to me hard to distinguish from T30 and then T40 nets, but I am a weak player and can easily miss some stylistic differences.
Top
Dann Corbit
Posts: 12540
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Dann Corbit »

Here is a link to some of the analysis:


After reading about the improved performance, I switched to the beta engine, so the attached data is only a partial set, and I am redoing all of them
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Top
Dann Corbit
Posts: 12540
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Dann Corbit »

Early returns...
Here is the LC0 analysis with the big net used in TCEC:

The archive contains an Arena log, the report file, and the resulting EPD records.
138 of 200 matching moves

And here are the first 170 or so EPD records using SF (bleeding edge) at 15 minutes with 60 threads:
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Top
Vinvin
Posts: 5228
Joined: Thu Mar 09, 2006 9:40 am
Full name: Vincent Lejeune

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Vinvin »

Laskos wrote: ↑Sun Oct 04, 2020 5:22 pm ...
Lc0 LS15: 152
Lc0 J92-190: 145
Lc0 SV1541: 141
SFdev Hybrid: 138...
So, as SF is better than Lc0, that probably means that's not a good idea to be better positionally ? :lol:
Top
BrendanJNorman
Posts: 2526
Joined: Mon Feb 08, 2016 12:43 am
Full name: Brendan J Norman

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by BrendanJNorman »

Vinvin wrote: ↑Thu Oct 08, 2020 3:27 am
Laskos wrote: ↑Sun Oct 04, 2020 5:22 pm ...
Lc0 LS15: 152
Lc0 J92-190: 145
Lc0 SV1541: 141
SFdev Hybrid: 138...
So, as SF is better than Lc0, that probably means that's not a good idea to be better positionally ? :lol:
Or perhaps tactics play a bigger part in chess than you thought? :wink:
Top
Dann Corbit
Posts: 12540
Joined: Wed Mar 08, 2006 8:57 pm
Location: Redmond, WA USA

Re: Lc0 with latest test30 nets is vastly superior positionally

Post by Dann Corbit »

Vinvin wrote: ↑Thu Oct 08, 2020 3:27 am
Laskos wrote: ↑Sun Oct 04, 2020 5:22 pm ...
Lc0 LS15: 152
Lc0 J92-190: 145
Lc0 SV1541: 141
SFdev Hybrid: 138...
So, as SF is better than Lc0, that probably means that's not a good idea to be better positionally ? :lol:
I guess that further data will modify the test a bit (while on the other hand we can still see that the best programs are at the top and we also suspect that the giant neural nets of LC0 are really good at quiet positions). But opening positions are just one aspect of the game of chess. While I would agree that quiet moves are more important that captures and other "noisy" moves, both kinds are important. And here is a mystery. Are there moves that are neither stragegic nor tactical but are just as important or more important? The reason I ask is the mystery of the nodes files. Look at the CPU horsepower and time devoted to improving the SF eval. Yet this tiny cluster of nodes out-performs it (if you consider 20MB tiny). What, exactly, did we miss such that a wimply little CPU can Poleaxe a mighty GPU, many, many time stronger than it is?
Taking ideas is not a vice, it is a virtue. We have another word for this. It is called learning.
But sharing ideas is an even greater virtue. We have another word for this. It is called teaching.
Top

Return to “Computer Chess Club: General Topics”