Reed
Connectionist Dave approach to AI
neural networks, neuron model perceptrons threshold logic, perceptron training, convergence theorem single layer vs. multi-layer backpropagation stepwise vs. continuous activation function associative memory Hopfield networks, parallel relaxation
Symbolic vs. sub-symbolic AI
recall: Good Old-Fashioned AI is inherently symbolic Physical Symbol System Hypothesis: A necessary and sufficient condition for intelligence is the representation and manipulation of symbols. alternatives to symbolic AI
connectionist models based on a brain metaphor model individual neurons and their connections properties: parallel, distributed, sub-symbolic examples: neural nets, associative memories emergent models based on an evolution metaphor potential solutions compete and evolve properties: massively parallel, complex behavior evolves out of simple behavior examples: genetic algorithms, cellular automata, artificial life
Connectionist models (neural nets)
humans lack the speed & memory of computers yet humans are capable of complex reasoning/action
maybe our brain architecture is well-suited for certain tasks
general brain architecture:
many (relatively) slow neurons, interconnected dendrites serve as input devices (receive electrical impulses from other neurons) cell body "sums" inputs from the dendrites (possibly inhibiting or exciting) if sum exceeds some threshold, the neuron fires an output impulse along axon
Brain metaphor
connectionist models are based on the brain metaphor
large number of simple, neuron-like processing elements large number of weighted connections between neurons note: the weights encode information, not symbols! parallel, distributed control emphasis on learning
brief history of neural nets
1940's 1950's & 1960's 1970's 1980's & 1990's theoretical birth of neural networks McCulloch & Pitts (1943), Hebb (1949) optimistic development using computer models Minsky (50's), Rosenblatt (60's) DEAD Minsky & Papert showed serious limitations REBIRTH new models, new techniques Backpropagation, Hopfield nets
Artificial neurons
McCulloch & Pitts (1943) described an artificial neuron
inputs are either excitatory (+1) or inhibitory (-1) each input has a weight associated with it the activation function multiplies each input value by its weight if the sum of the weighted inputs >= ,
then the neuron fires (returns 1), else doesn't fire (returns 1)
w
1
if wixi >= , output = 1
w
2 n
w
if wixi < , output -1
...
x1
x2
xn
Computation via activation function
can view an artificial neuron as a computational element
accepts or classifies an input if the output fires
INPUT: x1 = 1, x2 = 1 .75*1 + .75*1 = 1.5 >= 1 OUTPUT: 1
1 .75 .75
INPUT: x1 = 1, x2 = -1 .75*1 + .75*-1 = 0 < 1 INPUT: x1 = -1, x2 = 1 OUTPUT: -1
x1
x2
.75*-1 + .75*1 = 0 < 1 INPUT: x1 = -1, x2 = -1
OUTPUT: -1
.75*-1 + .75*-1 = -1.5 < 1 OUTPUT: -1
this neuron computes the AND function
In-class exercise
specify weights and thresholds to compute OR
INPUT: x1 = 1, x2 = 1 w1*1 + w2*1 >= OUTPUT: 1
w
1
INPUT: x1 = 1, x2 = -1
w
2
w1*1 + w2*-1 >= INPUT: x1 = -1, x2 = 1
OUTPUT: 1
x1
x2
w1*-1 + w2*1 >= INPUT: x1 = -1, x2 = -1 w1*-1 + w2*-1 <
OUTPUT: 1
OUTPUT: -1
Normalizing thresholds
to make life more uniform, can normalize the threshold to 0 simply add an additional input x0 = 1, w0 = -
w
1
0
w
2 n
- w
1
w
w
w
2
n
...
...
x1
x2
xn
1
x1
x2
xn
advantage: threshold = 0 for all neurons wixi >=
- *1 +
wixi >= 0
Perceptrons
Rosenblatt (1958) devised a learning algorithm for artificial neurons given a training set (example inputs & corresponding desired outputs)
1. start with some initial weights 2. iterate through the training set, collect incorrect examples 3. if all examples correct, then DONE 4. otherwise, update the weights for each incorrect example
if x1, ...,xn should have fired but didn't,
<= n)
wi += xi (0 <= i
if x1, ...,xn shouldn't have fired but did,
<= n)
wi -= xi (0 <= i
1. GO TO 2
Example: perceptron learning
Suppose we want to train a perceptron to compute AND training set:
x1 = 1, x2 = 1 x1 = 1, x2 = -1 -1 x1 = -1, x2 = 1 -1 x1 = -1, x2 = -1 -1 1
0
-0.9 0.6 0.2
randomly, let: w0 = -0.9, w1 = 0.6, w2 = 0.2 using these weights:
x1 = 1, x2 = 1: x1 = 1, x2 = -1: x1 = -1, x2 = 1: x1 = -1, x2 = -1: -0.9*1 + 0.6*1 + 0.2*1 -0.9*1 + 0.6*1 + 0.2*-1 -0.9*1 + 0.6*-1 + 0.2*1 -0.9*1 + 0.6*-1 + 0.2*-1 = = = = -0.1 -1 -0.5 -1 -1.3 -1 -1.7 -1 WRONG OK OK OK
1
x1
x2
new weights: w0 = -0.9 + 1 = 0.1
w1 = 0.6 + 1 = 1.6 w2 = 0.2 + 1 = 1.2
Example: perceptron learning (cont.)
0
0.1 1.6 1.2
using these updated weights:
x1 = 1, x2 = 1: x1 = 1, x2 = -1: x1 = -1, x2 = 1: x1 = -1, x2 = -1: 0.1*1 + 1.6*1 + 1.2*1 0.1*1 + 1.6*1 + 1.2*-1 0.1*1 + 1.6*-1 + 1.2*1 0.1*1 + 1.6*-1 + 1.2*-1 = = = = 2.9 1 0.5 1 -0.3 -1 -2.7 -1 OK WRONG OK OK
1
x1
x2
new weights: w0 = 0.1 1 = -0.9
w1 = 1.6 1 = 0.6 w2 = 1.2 + 1 = 2.2
0
-0.9 0.6 2.2
using these updated weights:
x1 = 1, x2 = 1: x1 = 1, x2 = -1: x1 = -1, x2 = 1: x1 = -1, x2 = -1: -0.9*1 + 0.6*1 + 2.2*1 -0.9*1 + 0.6*1 + 2.2*-1 -0.9*1 + 0.6*-1 + 2.2*1 -0.9*1 + 0.6*-1 + 2.2*-1 = = = = 1.9 1 -2.5 -1 0.7 1 -3.7 -1 OK OK WRONG OK
1
x1
x2
new weights: w0 = -0.9 1 = -1.9
w1 = 0.6 + 1 = 1.6 w2 = 2.2 1 = 1.2
Example: perceptron learning (cont.)
0
-1.9 1.6 1.2
using these updated weights:
x1 = 1, x2 = 1: x1 = 1, x2 = -1: x1 = -1, x2 = 1: x1 = -1, x2 = -1: -1.9*1 + 1.6*1 + 1.2*1 -1.9*1 + 1.6*1 + 1.2*-1 -1.9*1 + 1.6*-1 + 1.2*1 -1.9*1 + 1.6*-1 + 1.2*-1 = = = = 0.9 1 -1.5 -1 -2.3 -1 -4.7 -1 OK OK OK OK
1
x1
x2
DONE!
EXERCISE: train a perceptron to compute OR
key reason for interest in perceptrons:
Convergence
Perceptron Convergence Theorem The perceptron learning algorithm will always find weights to classify the inputs if such a set of weights exists.
Minsky & Papert showed such weights exist if and only if the problem is linearly separable intuition: consider the case with 2 inputs, x1 and x2
-1 -1
if you can draw a line and separate the accepting & non-accepting examples, then linearly separable
x1
-1 1 1 1 1
the intuition generalizes: for n inputs, must be able to separate with an (n-1)-dimensional plane.
x2
Linearly separable
x1
1
-1
AND function
1
x1
1
1
OR function
1
-1
-1
-1
1
1
x2
1
x2
why does this make sense? firing depends on w0 + w1x1 + w2x2 >= 0 border case is when i.e., a line the training algorithm simply shifts the line around (by changing w0 + w1x1 + w2x2 = 0 the equation of x2 = x1 (-w1/w2) + (-w0 /w2)
Inadequacy of perceptrons
inadequacy of perceptrons is due to the fact that many simple problems are not linearly separable
x1 1
1
XOR function
-1
-1
1
x2
-0.1
however, can compute XOR by introducing a new, hidden unit
-3.5 1.5 1
1.5
1.5 1
x1
x2
Hidden units
the addition of hidden units allows the network to develop complex feature detectors (i.e., internal representations)
e.g., Optical Character Recognition (OCR) perhaps one hidden unit
"looks for" a horizontal bar
another hidden unit
"looks for" a diagonal
the combination of specific
hidden units indicates a 7
Building multi-layer nets
smaller example: can combine perceptrons to perform more complex computations (or classifications) 3-layer neural net 2 input nodes 1 hidden node 2 output nodes
1.5
1
.75
-0.1
.75
1.5
1.5
-3.5
RESULT?
1
1
x1
x2
HINT: left output node is AND right output node is XOR FULL ADDER
Hidden units & learning
every classification problem has a perceptron solution if enough hidden layers are used
i.e., multi-layer networks can compute anything
(recall: can simulate AND, OR, NOT gates) expressiveness is not the problem learning is!
it is not known how to systematically find solutions the Perceptron Learning Algorithm can't adjust weights between levels
Minsky & Papert's results about the "inadequacy" of perceptrons pretty much killed neural net research in the 1970's rebirth in the 1980's due to several developments
faster, more parallel computers new learning algorithms e.g., backpropagation new architectures e.g., Hopfield nets
Backpropagation nets
backpropagation nets are multi-layer networks
normalize inputs between 0 (inhibit) and 1 (excite) utilize a continuous activation function
perceptrons utilize a stepwise activation function output = 1 if sum >= 0 0 if sum < 0
backpropagation nets utilize a continuous activation function output = 1/(1 + e-sum)
Backpropagation example (XOR)
x1 = 1, x2 = 1
sum(H1) = -2.2 + 5.7 + 5.7 = 9.2, output(H1) = 0.99 sum(H2) = -4.8 + 3.2 + 3.2 = 1.6, output(H2) = 0.83 sum = -2.8 + (0.99*6.4) + (0.83*-7) = -2.28, output = 0.09
2.8
6.4
2.2
-7 H 5.7 1 1
1
x1 = 1, x2 = 0
sum(H1) = -2.2 + 5.7 + 0 = 3.5, output(H1) = 0.97 sum(H2) = -4.8 + 3.2 + 0 = -1.6, output(H2) = 0.17 sum = -2.8 + (0.97*6.4) + (0.17*-7) = 2.22, output = 0.90
3.2
H 3.2
2
4.8
5.7
x1 = 0, x2 = 1
sum(H1) = -2.2 + 0 + 5.7 = 3.5, output(H1) = 0.97 sum(H2) = -4.8 + 0 + 3.2 = -1.6, output(H2) = 0.17 sum = -2.8 + (0.97*6.4) + (0.17*-7) = 2.22, output = 0.90
x1
x2
x1 = 0, x2 = 0
sum(H1) = -2.2 + 0 + 0 = -2.2, output(H1) = 0.10 sum(H2) = -4.8 + 0 + 0 = -4.8, output(H2) = 0.01 sum = -2.8 + (0.10*6.4) + (0.01*-7) = -2.23, output = 0.10
Backpropagation learning
there exists a systematic method for adjusting weights, but no global convergence theorem (as was the case for perceptrons) backpropagation (backward propagation of error) vaguely stated
select arbitrary weights pick the first test case make a forward pass, from inputs to output compute an error estimate and make a backward pass, adjusting weights to reduce the error repeat for the next test case
testing & propagating for all training cases is known as an epoch despite the lack of a convergence theorem, backpropagation works well in practice
however, many epochs may be required for convergence
Problems/challenges in neural nets research
learning problem
can the network be trained to solve a given problem? if not linearly separable, no guarantee (but backprop effective in practice)
architecture problem
are there useful architectures for solving a given problem? most applications use a 3-layer (input, hidden, output), fully-connected net
scaling problem
how can training time be minimized? difficult/complex problems may require thousands of epochs
generalization problem
how know if the trained network will behave "reasonably" on new inputs? cross-validation oft...