Analysis of policy nets

[chrisw]

One way to analyse current neural nets is to take a look at their components parts. I played some games between a young NN of mine and LCZero, where LC0 was set to nodes=0, with the result it just returns its top policy move. Top policy move is supposed to be the best/most probable/whatever move at that position. I read that Alexander wrote that policy alone is responsible for 2400 ELO.
So LCZero raw policy versus my raw policy (mine of course got trashed) shows a few things.

Here, in one game, LCZero policy just leaves on a nite fork KQ ...... (move 16). Looks like it is not correct so say NN policies "understand" nite fork tactics. Other thing to notice is that Policy starts producing quite silly moves when position is objectively already "lost". Please don't answer with boring questions like which net (it is one I downloaded last month), this is not about individual nets, its about current NNs in general. If people are interested, I have several more examples that NN-policy is not exactly accurate in various weird and wonderful ways, to post.

LCZero policy no lookahead vs testNet 2019.03.26 0-1
[pgn] 1.e4 c5 2.Nf3 Nf6 3.e5 Nd5 4.Nc3 e6 5.Nxd5 exd5 6.d4 Nc6 7.dxc5 Bxc5 8.Qxd5 O-O 9.Qxc5 Re8 10.Be3 b6 11.Qc3 Bb7 12.O-O-O Qe7 13.e6 dxe6 14.Ng5 h6 15.Ne4 Nb4 16.Nd6 Nxa2+ 17.Kd2 Nxc3 18.Kxc3 Red8 19.Bd3 Bxg2 20.Rd2 Bxh1 21.Bg6 fxg6 22.b4 Rxd6 23.Rxd6 Qxd6 24.b5 Rc8+ 25.Kb2 Qb4+ 26.Ka1 Rxc2 27.Bxb6 axb6 28.f3 Rxh2 29.f4 Qxf4 30.Kb1 Qb4+ 31.Ka1 Qb2# 0-1 [/pgn]

[jp]

If it's really 2400 on any human or computer scale I'm surprised.
When they had nets to play with on the Lc website, I doubt it was anywhere near that strong. They stopped that, so it must have been many months ago.

[chrisw]

this one shows well how with 2Q or 3Q, LC0 Policy doesn't really know what it is doing. Playing chess apparently. All quite funny to watch after about move 30 or so ....

[pgn][Event "LZ vs OurNet testing"] [Site "PC"] [Date "2019.03.26"] [Round "3"] [White "Our TestNet"] [Black "LCZero"] [Result "1/2-1/2"] 1.e4 e6 2.Bc4 d5 3.exd5 exd5 4.Bb3 a5 5.Ne2 a4 6.O-O axb3 7.axb3 Rxa1 8.d3 Rxb1 9.Re1 Be7 10.Nd4 Nf6 11.h3 O-O 12.Bf4 Rxd1 13.Rxd1 c5 14.Nb5 Nc6 15. c3 d4 16.Na3 dxc3 17.bxc3 Nd5 18.Bd2 f5 19.Nc4 f4 20.Ra1 f3 21.Re1 fxg2 22.Kxg2 b5 23.Ne3 b4 24.Nxd5 Qxd5+ 25.Re4 Qxd3 26.Re3 Qxd2 27.Rf3 bxc3 28. Rxf8+ Kxf8 29.h4 Bxh4 30.Kf3 c2 31.Kg2 c1=Q 32.Kf3 Qdd1+ 33.Kg2 Qb1 34.f3 Qa1 35.f4 Qab1 36.Kh2 Qa1 37.Kg2 h6 38.f5 Bb7 39.Kh3 h5 40.Kxh4 Qab1 41. Kg5 h4 42.Kxh4 Qa1 43.Kg3 Qab1 44.Kf4 Qa1 45.Kg3 Ba6 46.Kf4 Bb7 47.Kg3 Qab1 48.Kf4 Ba6 49.Kg3 Bb7 50.Kf4 Ba6 51.Kg3 g6 52.fxg6 Bb7 53.Kf4 Qa1 54. Kg5 Qab1 55.Kf4 Nd4 56.Kg5 Qa1 57.Kf4 Qab1 58.Kg5 Ne2 59.b4 Qa1 60.bxc5 Qab1 61.c6 Qa1 62.cxb7 Qab1 63.b8=Q+ Qxb8 64.Kf6 Kg8 65.Ke6 Kh8 66.Kf5 Kg7 67.Kg5 Kg8 68.Kf6 Kf8 69.g7+ Ke8 70.g8=Q+ Kd7 71.Qxb8 Qd4+ 72.Kf7 Qd5+ 73. Kg7 Qg5+ 74.Kf7 Qf5+ 75.Kg7 Qg5+ 76.Kf7 Qf5+ 77.Kg7 Qg5+ 78.Kf7 Qf5+ 79. Kg7 Qg5+ 80.Kf7 Qf5+ 81.Kg7 Qg5+ 1/2-1/2 [/pgn]

[jp]

Way back then, even in normal positions it would play a bit like a hustler from time to time.