"Leela is an opening book" nonsense

[dkappe]

This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?

It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.

Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.

[AdminX]

This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?

It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.

Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.

Problem solved and case closed! Case dismissed due to lack of evidence.

[jhellis3]

Nice strawman bro?

[dkappe]

Nice strawman bro?

I wish it were.

[jhellis3]

Why do you have to wish?

[the_real_greco]

--aha. EXCELLENT IDEA!
1: Play {SF, Komodo, Ethereal, etc} vs Lc0 chess960 tournament.
2: Also play {SF, K, E etc} vs Lc0 ordinary-chess tourney as
"experimental control."
(But not just 10 games in each tourney; you will need far more games than that.)

CCC could run this test. Might be fun (although I suspect kind of pointless).

[MikeB]

This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?

It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.

Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.

I don't think anybody who is knowledgeable thinks that now, but yea at first glance it might look like that to someone who doesn't understand. One big difference , as you illustrate quite well in your post , an opening book no longer works once it gets out of book, while a NN engine can still make an pretty decent move even if it has never seen the position before. Your post is akin to proving the earth is round, there is no dispute unless somebody just wants to be in self denial - which we cannot control. Thanks for posting.

[Rebel]

Nice strawman bro?

I wish it were.

You are right, I tried, see subject Amazing Leela - http://rebel13.nl/home/whatever.html

Encouraged by the first excellent results at D=1 I tried D=2 | D=3 | D=4 | D=5 and instead of getting somewhat stronger at each deeper depth strangely enough the opposite happened, the deeper depth the weaker Leela performed. Ra ra...

[corres]

This notion that neural networks are actually opening books in disguise keeps rearing its silly head. I decided to test this opening book hypothesis with one of my favorite nets. With 3 seconds per move on a 2070, I investigated a few opening lines, taking the top choice, except in the case of the first or second move as noted. So, what openings does this net like?
It likes e4 and d4. On e4, it likes e6 and c5. On e6, it likes d4. Looking a few moves deeper, we get 1. e4 e6 2. d4 d5 3. Nd2 c5 4. Ngf3 cxd4 5. Nxd4 Nf6 -- pretty main line Tarrash French. If we go down the Sicilian route, we get 1. e4 c5 2. Nf3 d6 3. Bb5+ Bd7 4. Bxd7+ Qxd7 5. O-O Nc6 -- main line Rossolimo. And on d4, 1. d4 Nf6 2. Nf3 e6 3. c4 d5 4. Nc3 Be7 5. Bg5 O-O is pretty main line QGD.
Good evidence that this net has memorized opening moves? Not so fast. This is Little Ender, an endgame net that has never seen opening positions. In fact, it has never seen a position with more than 18 pieces.

I do not know your net contains policy head or not.
But it has or not your experience proves nothing but search of Leela is rather effective even if it uses randomly generated search moves.
So if you want to prove "There is no any such a behavior of NN what we can use such a mode than how we use an opening book" (as I stated "NN is a KIND of opening book") you have to gain information without any search. Using Leela we need at least 1 node/second speed and Leela use not only her Value Head but her Policy Head too. Because the meaning of Policy Head is not unambiguous for everybody, I should state under Policy Head I mean "Those part of NN what NN engines use helping their search and it do not contain value for the searched position - or if it contains value the search does not take it into consideration." So you use Leela with 1 node/second speed AND Leela does not use Policy Head the gotten results prove what Leela knows about openings.
I think the result will show such "book" in what there are holes and as the distance from the start position is larger as the number of holes will be higher.