Neural Nets can't explain what they do and this is a problem

Discussion of anything and everything relating to chess playing software and machines.

Moderators: hgm, Harvey Williamson, bob

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
Post Reply
althus
Posts: 4
Joined: Sun Aug 20, 2017 2:09 am

Neural Nets can't explain what they do and this is a problem

Post by althus » Sun Nov 26, 2017 10:43 pm

I've come across this article at the NYTimes about deep neural networks and the amazing things they do that we just cannot. The problem is that if a human has to make an important decision, like a medical diagnosis, based on an AI recommendation, will he/she do so with no understanding at all of how the AI came up with it, even though it is likely correct?

It reminds me of the way computer chess tends to be baffling to us humans, but is nonetheless "superhuman" at the same time.

Am curious what talkchess thinks of the piece. Sorry if it's been posted before.

jdart
Posts: 3675
Joined: Fri Mar 10, 2006 4:23 am
Location: http://www.arasanchess.org

Re: Neural Nets can't explain what they do and this is a pro

Post by jdart » Sun Nov 26, 2017 11:39 pm

It's a well known issue and also applies to other kinds of complex learned models, including random forests, SVM with non-linear kernels, etc.

--Jon

Michel
Posts: 2001
Joined: Sun Sep 28, 2008 11:50 pm

Re: Neural Nets can't explain what they do and this is a pro

Post by Michel » Mon Nov 27, 2017 7:45 am

I find this even more worrisome...

https://arxiv.org/abs/1412.1897
Ideas=science. Simplification=engineering.
Without ideas there is nothing to simplify.

ZirconiumX
Posts: 1327
Joined: Sun Jul 17, 2011 9:14 am

Re: Neural Nets can't explain what they do and this is a pro

Post by ZirconiumX » Mon Nov 27, 2017 7:55 am

althus wrote:I've come across this article at the NYTimes about deep neural networks and the amazing things they do that we just cannot. The problem is that if a human has to make an important decision, like a medical diagnosis, based on an AI recommendation, will he/she do so with no understanding at all of how the AI came up with it, even though it is likely correct?

It reminds me of the way computer chess tends to be baffling to us humans, but is nonetheless "superhuman" at the same time.

Am curious what talkchess thinks of the piece. Sorry if it's been posted before.
I'd like to point out that also applies to humans. We don't know how the brain works, and possibly won't for a long time.

If the brain was so simple we could understand it, we would be so simple we couldn't understand it. The same goes for neural nets.
Some believe in the almighty dollar.

I believe in the almighty printf statement.

Henk
Posts: 5491
Joined: Mon May 27, 2013 8:31 am

Re: Neural Nets can't explain what they do and this is a pro

Post by Henk » Mon Nov 27, 2017 4:17 pm

Artist can't create good art if he follows rules only.

Post Reply