A question abou LCZero for the experts

[Dann Corbit]

The LCZero training creates a big pile of numbers in a file.
They have to do with features LCZero has derived from observable patterns.

My question is this:

Is it somehow possible to reverse engineer this pile of numbers to tell us what they mean?

I would be keen to know if there is some remarkable and fresh insight contained in this nebulous pile of digits.

If we can't can someone give me the layman's explanation of why it cannot be done.

[Albert Silver]

The LCZero training creates a big pile of numbers in a file.
They have to do with features LCZero has derived from observable patterns.

My question is this:

Is it somehow possible to reverse engineer this pile of numbers to tell us what they mean?

I would be keen to know if there is some remarkable and fresh insight contained in this nebulous pile of digits.

If we can't can someone give me the layman's explanation of why it cannot be done.

Right now, as far as I know we cannot. Neural Networks are generally referred to as Black Boxes, with indecipherable information. I saw a lot of this consternation in backgammon, where NNs have dominated top programs for well over a decade. It is not a new thing, and is the reason why one of the latest trends in computing is developing means to understand the whys behind the decisions, possibly create or insert something in the middle that allows for later decrypting. The concern is mostly related to AI and fear that we see important decisions that are not able to be explained.

[Evert]

I would be keen to know if there is some remarkable and fresh insight contained in this nebulous pile of digits.

That would, indeed, be extremely interesting. If it could be done.

If we can't can someone give me the layman's explanation of why it cannot be done.

The amount of data is enormous. It's also the way knowledge is encoded in the network. It's not contained in a few easily identifiable neurons, where you can work out what they do on a piece of paper. It's distributed in the connections throughout the network.

[Gian-Carlo Pascutto]

There's been research on deconvolution in image recognition networks.

i.e. for a network that classifies what is in an image, set the output to "car" and run the network in reverse, then see what it produces at the input end.

The output space for LC0 may mean that such approaches won't work well. But perhaps you could ask the network to produce, for example, positions where it's 100% certain that Bxh7 is the best move.

[Albert Silver]

There's been research on deconvolution in image recognition networks.

i.e. for a network that classifies what is in an image, set the output to "car" and run the network in reverse, then see what it produces at the input end.

The output space for LC0 may mean that such approaches won't work well. But perhaps you could ask the network to produce, for example, positions where it's 100% certain that Bxh7 is the best move.

Well, if you ever consider changing Leela's name, it should be Bogo. Efim Bogoljubow, for those who don't understand the reference, a great player in the 30s, was famous for his optimism and saying "When I am White I win because I am White. When I am Black I win because I am Bogoljubow.".

I ran a match with 50 typical and varied opening positions, alternating colors, and no matter the color, Leela always said she was better in the first move.

[Gian-Carlo Pascutto]

I ran a match with 50 typical and varied opening positions, alternating colors, and no matter the color, Leela always said she was better in the first move.

Even considering that internally the colors are flipped to always be white, this still sounds a bit unexpected to me.