Created Tuesday 31 December 2013
Two neural networks are at a bar. One says to the other, "Hey, I learned that if A then B". The other replies, "Wow, I know that if B then C". "Why don't we teach ourselves if A then C?" A while later, they ask each other "What was it what we were talking about?"
The joke is in reference to the fact that while neural networks can be used to create powerful technologies (self-driving car, voice recognition like Google Now, etc), neural networks are actually pretty stupid. The reality is that machine learning hasn't advanced much when it comes to creating a machine that can actually think for itself i.e., "sentient" artificial intelligence.
As per this TED talk, the real problem is that there doesn't seem to be a common "theory of learning."
A Brief History
- Neurons were researched over 100 years ago. Scientists discovered that our brain is made up of millions of connected neurons.
- first mathematical model of a neuron was created in 1943 by Warren McCulloch and Walter Pitts
- The original neuron model is basically a linear function. Each coordinate is multiplied by a weight and the sum of each product is the output of the neuron.
- Neurons working together in parallel is a single layer of a neural network. Layers can be stacked together to create multi-layer neural networks.
- By the 60s, NNs had a lot of hype---robotic servants, etc.
- A book in 1969 by Minsky helped destroy interest in neural networks for the next decade.
- Minksy pointed out the elephant in the room. Single layer neural networks can model only linear data.
- Research decreased during the 70s because: (1) computers weren't (yet) powerful enough for research and (2) no motivation or $money$ (thanks to Minksy)
- It wasn't until the 80's that interest in NNs surged again. In particular, the backpropagation algorithm was popularized by Rumelhart. (Up until this point, there was no obvious way to automatically train an NN with sample data.)
Some Resources
- http://jeremykun.com/2011/08/11/the-perceptron-and-all-the-things-it-cant-perceive/
- http://www.cogsci.rpi.edu/~rsun/sun-zhang-jcsr2004-f.pdf
- http://www-cs-faculty.stanford.edu/~eroberts/courses/soco/projects/neural-networks/History/history1.html
- Elements of Artificial Neural Networks. Kishan Mehrotra, Chilukuri K. Mohan and Sanjay Ranka