Copyright and Machine Learning IP

[syzygy]

I am also a bit uncomfortable with this focusing on EU law, now you mentioned that member states are free to adopt laws to protect sports events not protected by EU law. That woke me up to the fact that EU law is not a complete system of laws of a civilized society, but just the part of it that could be harmonized over all its member states.

Copyright is part of the law that has been harmonised in the EU. The special protection for sporting events will have to be separate from copyright. It also has to comply with a series of requirements in order for it to be compatible with the free movement of services, as the CJEU explains in C-403/08. I doubt that neural networks play such an important role in any of the EU member states' traditions that special protection can be justified. But if anyone is aware of any special national legislation affecting NNs, please let us hear.

In C-403/08, the CJEU declared incompatible with EU law a national law forbidding the import, sale and use of foreign decoding devices giving access to an encrypted satellite broadcasting service from another EU member state.

[syzygy]

Well, the architecture of a neural net maps directly to a computer program. You could say that the net inputs (chess position eg) are variables, the fixed weights as program constants and the net architecture as the program. If you really wanted, you could unroll the program loops and insert the constant weights appropriately in each unrolled section. And represent the whole thing (weights and architecture) as source code. You could get the training program to spew the whole thing out as source or object code if you really wanted, at which point you really have the ingenious machine that really has written a useful program. It's written all the source code of an evaluation function for chess positions.

It would just be a mechanical translation. The result would be copyrighted only if the NN was copyrighted. Just like the output of a compiler is copyrighted only if the source code given to it is copyrighted. Looking at a neural network as object code (or as the equivalent extremely boring uncreative source code) does not change anything, as far as I can see.

[chrisw]

Well, the architecture of a neural net maps directly to a computer program. You could say that the net inputs (chess position eg) are variables, the fixed weights as program constants and the net architecture as the program. If you really wanted, you could unroll the program loops and insert the constant weights appropriately in each unrolled section. And represent the whole thing (weights and architecture) as source code. You could get the training program to spew the whole thing out as source or object code if you really wanted, at which point you really have the ingenious machine that really has written a useful program. It's written all the source code of an evaluation function for chess positions.

It would just be a mechanical translation. The result would be copyrighted only if the NN was copyrighted. Just like the output of a compiler is copyrighted only if the source code given to it is copyrighted. Looking at a neural network as object code (or as the equivalent extremely boring uncreative source code) does not change anything, as far as I can see.

Nevertheless we've written a program that has written a program. True that we told our written program what to do and how to do it, but we also told it, here's a bunch of games, build a chess playing program according to this blueprint and mine the games for the smart stuff you need. Go.

Alternatively, in some other universe, someone says to a machine. Write a chess program. Go.

Where in between here and there does the mechanical-functional argument fail? What isn't mechanical-functional?

[syzygy]

Well, the architecture of a neural net maps directly to a computer program. You could say that the net inputs (chess position eg) are variables, the fixed weights as program constants and the net architecture as the program. If you really wanted, you could unroll the program loops and insert the constant weights appropriately in each unrolled section. And represent the whole thing (weights and architecture) as source code. You could get the training program to spew the whole thing out as source or object code if you really wanted, at which point you really have the ingenious machine that really has written a useful program. It's written all the source code of an evaluation function for chess positions.

It would just be a mechanical translation. The result would be copyrighted only if the NN was copyrighted. Just like the output of a compiler is copyrighted only if the source code given to it is copyrighted. Looking at a neural network as object code (or as the equivalent extremely boring uncreative source code) does not change anything, as far as I can see.

Nevertheless we've written a program that has written a program. True that we told our written program what to do and how to do it, but we also told it, here's a bunch of games, build a chess playing program according to this blueprint and mine the games for the smart stuff you need. Go.

Alternatively, in some other universe, someone says to a machine. Write a chess program. Go.

Where in between here and there does the mechanical-functional argument fail? What isn't mechanical-functional?

I'm talking about the mechanical translation NN weights -> equivalent program.

My point is that the existence of this mapping together with the argument "there is copyright on source code/object code" does not work to show that there is copyright on the NN. (Because there is in fact no copyright on the mechanical translation of non-copyrighted material, even if the translation is code.)

[chrisw]

Well, the architecture of a neural net maps directly to a computer program. You could say that the net inputs (chess position eg) are variables, the fixed weights as program constants and the net architecture as the program. If you really wanted, you could unroll the program loops and insert the constant weights appropriately in each unrolled section. And represent the whole thing (weights and architecture) as source code. You could get the training program to spew the whole thing out as source or object code if you really wanted, at which point you really have the ingenious machine that really has written a useful program. It's written all the source code of an evaluation function for chess positions.

It would just be a mechanical translation. The result would be copyrighted only if the NN was copyrighted. Just like the output of a compiler is copyrighted only if the source code given to it is copyrighted. Looking at a neural network as object code (or as the equivalent extremely boring uncreative source code) does not change anything, as far as I can see.

Nevertheless we've written a program that has written a program. True that we told our written program what to do and how to do it, but we also told it, here's a bunch of games, build a chess playing program according to this blueprint and mine the games for the smart stuff you need. Go.

Alternatively, in some other universe, someone says to a machine. Write a chess program. Go.

Where in between here and there does the mechanical-functional argument fail? What isn't mechanical-functional?

I'm talking about the mechanical translation NN weights -> equivalent program.

My point is that the existence of this mapping together with the argument "there is copyright on source code/object code" does not work to show that there is copyright on the NN. (Because there is in fact no copyright on the mechanical translation of non-copyrighted material, even if the translation is code.)

Agreed. For this particular ingenious machine, the key step is the process "transformation of input game database into a probe-able knowledge engine", it wasn't possible to probe the database for this "knowledge", but it is possible to probe its transformation, or whatever you want to call it.

(there is also the case, that we don't even need a game database, games could be generated on the fly, as they more or less are in the Zero model).

And you argue, I think, that this transformation is functional. Each repetitive step is functional. But why not also creative? Does functional exclude the elements necessary to be called creative? What does a step do? Before, there was encoded a unique chess knowledge, never known before or to be known again (nature of size of network). Each step extracts a tiny piece of chess knowledge, minutely transforming the weights matrix, creating a new chess knowledge, never known before or to be known again. Until the steps stop.

You don't know the knowledge that's been got, but you know it has been got, and you know the state beforehand, the change in state and the state after are all totally unique. And that they represent intangible knowledge. Doesn't that step constitute an original creation?

[syzygy]

And you argue, I think, that this transformation is functional.

No, I'm saying that the transformation is mechanical.

Converting a book from ASCII to Morse code is a mechanical and can therefore not result in the addition of creative elements.

Mechanically converting an NN weight into a few lines of code implementing the corresponding neuron, same story.

Does functional exclude the elements necessary to be called creative?

It does, but that's not really the point here I think.

[chrisw]

And you argue, I think, that this transformation is functional.

No, I'm saying that the transformation is mechanical.

Converting a book from ASCII to Morse code is a mechanical and can therefore not result in the addition of creative elements.

Mechanically converting an NN weight into a few lines of code implementing the corresponding neuron, same story.

Does functional exclude the elements necessary to be called creative?

It does, but that's not really the point here I think.

I think we are referring to different things with the same word, transformation. I meant the transformation games to weights. Which is not reversible (as would be book to morse). Where games to weights really means: extracting knowledge from each game, altering the weights, such that the final result is a knowledge probe-able network. The games are not usefully knowledge probe-able in the same way, they require creative interpretation by a human to give up their knowledge (eg how a GM learns by studying).

[syzygy]

And you argue, I think, that this transformation is functional.

No, I'm saying that the transformation is mechanical.

Converting a book from ASCII to Morse code is a mechanical and can therefore not result in the addition of creative elements.

Mechanically converting an NN weight into a few lines of code implementing the corresponding neuron, same story.

Does functional exclude the elements necessary to be called creative?

It does, but that's not really the point here I think.

I think we are referring to different things with the same word, transformation. I meant the transformation games to weights. Which is not reversible (as would be book to morse). Where games to weights really means: extracting knowledge from each game, altering the weights, such that the final result is a knowledge probe-able network. The games are not usefully knowledge probe-able in the same way, they require creative interpretation by a human to give up their knowledge (eg how a GM learns by studying).

Yes, that is not what I was referring to.

I was referring to this:

Well, the architecture of a neural net maps directly to a computer program. You could say that the net inputs (chess position eg) are variables, the fixed weights as program constants and the net architecture as the program. If you really wanted, you could unroll the program loops and insert the constant weights appropriately in each unrolled section. And represent the whole thing (weights and architecture) as source code. You could get the training program to spew the whole thing out as source or object code if you really wanted, at which point you really have the ingenious machine that really has written a useful program. It's written all the source code of an evaluation function for chess positions.

So my point is that the set of weights will be just as (un)copyrighted as the equivalent source or object code. The extra mechanical transformation (the "direct map") makes no difference.

[chrisw]

syzygy wrote: So my point is that the set of weights will be just as (un)copyrighted as the equivalent source or object code. The extra mechanical transformation (the "direct map") makes no difference.

Yes, I agreed with you earlier and switched to:

Chrisw wrote: Agreed. For this particular ingenious machine, the key step is the process "transformation of input game database into a probe-able knowledge engine", it wasn't possible to probe the database for this "knowledge", but it is possible to probe its transformation, or whatever you want to call it.

as the creative and potentially copyright giving step.

[syzygy]

syzygy wrote: So my point is that the set of weights will be just as (un)copyrighted as the equivalent source or object code. The extra mechanical transformation (the "direct map") makes no difference.

Yes, I agreed with you earlier and switched to:

Chrisw wrote: Agreed. For this particular ingenious machine, the key step is the process "transformation of input game database into a probe-able knowledge engine", it wasn't possible to probe the database for this "knowledge", but it is possible to probe its transformation, or whatever you want to call it.

as the creative and potentially copyright giving step.

Yes, I see now. I skipped a paragraph I shouldn't have skipped.

OK, then I'd say the transformation is still mechanical and irreversible. Since it is mechanical, no copyrightable elements are added. Since it is irreversible, some copyrightable elements might be lost. (In view of section 178, UK copyright law seems to accept that (non-trivial) mechanical transformations can in theory add copyrightable elements, though.)

And I think what normally, if not always, will happen is that everything copyrightable about the initial set of games will be lost.

But I might be able to come up with an exception. The game of chess itself, as a set of game rules, is probably copyrightable (were it not that the rules of chess are in the public domain). Suppose you start with an uninitialised NN for playing a turn-based two-person board game, where the NN does not have any hard-coded game rules because of how the inputs are connected. Now you (creatively) come up with a set of game rules, generate a large amount of games adhering to those game rules, and train the network on those games.

If you play the resulting NN, you will probably be able to recognise some of the game rules (even though the NN might not have acquired the rules with 100% accuracy). So some of the creativity that went into coming up with the game rules will have been retained in the NN weights.

Going back to Lc0, something similar might be possible. Select training games having some silly pattern not connected to the rules of chess and hope you can find back that pattern in the games played by the resulting NN (showing that the NN has some observable creative characteristic created by the person selecting the training games). This does not seem to be easy to achieve, but may not be completely impossible.