Neural Nets can't explain what they do and this is a problem

[althus]

I've come across this article at the NYTimes about deep neural networks and the amazing things they do that we just cannot. The problem is that if a human has to make an important decision, like a medical diagnosis, based on an AI recommendation, will he/she do so with no understanding at all of how the AI came up with it, even though it is likely correct?

It reminds me of the way computer chess tends to be baffling to us humans, but is nonetheless "superhuman" at the same time.

Am curious what talkchess thinks of the piece. Sorry if it's been posted before.

[jdart]

It's a well known issue and also applies to other kinds of complex learned models, including random forests, SVM with non-linear kernels, etc.

--Jon

[Michel]

I find this even more worrisome...

https://arxiv.org/abs/1412.1897

[ZirconiumX]

I've come across this article at the NYTimes about deep neural networks and the amazing things they do that we just cannot. The problem is that if a human has to make an important decision, like a medical diagnosis, based on an AI recommendation, will he/she do so with no understanding at all of how the AI came up with it, even though it is likely correct?

It reminds me of the way computer chess tends to be baffling to us humans, but is nonetheless "superhuman" at the same time.

Am curious what talkchess thinks of the piece. Sorry if it's been posted before.

I'd like to point out that also applies to humans. We don't know how the brain works, and possibly won't for a long time.

If the brain was so simple we could understand it, we would be so simple we couldn't understand it. The same goes for neural nets.

[Henk]

Artist can't create good art if he follows rules only.