Unsupervised Learning
Unsupervised Learning
So far, we have only looked at supervised learning, in
which an external teacher improves network
performance by comparing desired and actual outputs
and modifying the synaptic weights accordingly.
Applications
12/9/2010
A Hierarchical Neural Model for the Detection of
Motion Patterns in Optical Flow Fields
Data Flow Diagram
of Visual Areas in
Macaque Brain
Blue:
motion perception
pathway
Overview
1. The Structure of the Visual Cortex
2. Using Selective Tuning t
12/8/2010
Adaptive Resonance Theory (ART)
Adaptive Resonance Theory (ART)
Adaptive Resonance Theory (ART) networks perform
completely unsupervised learning.
Their competitive learning algorithm is similar to the
first (unsupervised) phase of CPN learning.
Exemplar Analysis
Ensuring Coverage
When building a neural network application, we must
make sure that we choose an appropriate set of
exemplars (training data):
The entire problem space must be covered.
There must be no inconsistencies (contradictions)
K-Class Classification Problem
K-Class Classification Problem
Let us denote the kk-th class by Ck, with nk exemplars or
training samples, forming the sets Tk for k = 1, , K:
cfw_(
)
Tk = x kp , d kp | p = 1,., nk
The complete training set is T = T1TK.
Th
Associative Networks
Interpolative Associative Memory
Associative networks are able to store a set of
patterns and output the one that is associated with the
current input.
For heterohetero-association, we can use a simple two
two-layer
network of the fol
Adaptive Networks
Adaptive Networks
As you know, there is no equation that would tell you
the ideal number of neurons in a multimulti-layer network.
So far, we have determined the number of hiddenhiddenlayer units in BPNs by trial and error.
Ideally, we w
About Assignment #3
About Assignment #3
Two approaches to backpropagation learning:
1. PerPer-pattern learning:
Update weights after every exemplar presentation.
2. PerPer
Per-epoch
epoch (batch
(batch-mode) learning:
Update weights after every epoch. Dur
Creating Data Representations
Creating Data Representations
On the other hand, sets of orthogonal vectors (such
as 100, 010, 001) can be processed by the network
more easily.
This becomes clear when we consider that a neurons
net input signal is computed
Radial Basis Functions
Radial Basis Functions
If we are using such linear interpolation, then our
radial basis function (RBF) 0 that weights an input
vector based on its distance to a neurons reference
(weight) vector is 0(D) = D-1.
Since it is difficult
Cascade Correlation
Cascade Correlation
Weights to each new hidden node are trained to
maximize the covariance of the nodes output with the
current network error.
wi =
Covariance:
K
P
S(wnew ) = ( xnew, p xnew )( Ek , p Ek )
k =1 p =1
wnew : vector of we
Example I: Predicting the Weather
Example I: Predicting the Weather
We decide (or experimentally determine) to use a
hidden layer with 42 sigmoidal neurons.
The next thing we need to do is collecting the
training exemplars.
exemplars.
In summary, our netw
The Hopfield Network
Asynchronous Hopfield Network
The nodes of a Hopfield network can be updated
synchronously or asynchronously.
Current network state O, attractors (stored patterns)
X and Y:
Synchronous updating means that at time step (t+1)
every neur
Assignment #3 Question 2
Assignment #3 Question 2
Regarding your cascade correlation projects, here are
a few tips to make your life easier.
This will require more training, but it may find a better
(=lower error) overall solution for the weight vectors.
Sigmoidal Neurons
f i (net i (t ) =
Sigmoidal Neurons
1
1 + e ( net i ( t ) ) /
fi(neti(t)
1
This leads to a simplified form of the sigmoid function:
S (net ) =
= 0.1
=1
We do not need a modifiable threshold , because we
will use dummy inputs as we did f
Adaline Schematic
The Adaline uses gradient descent to determine the
weight vector that leads to minimal error.
i1
i2
in
The Adaline Learning Algorithm
w0 + w1i1 + + wnin
Adjust
weights
Output
The idea is to pick samples in random order and
perform (slow)
Capabilities of Threshold Neurons
Capabilities of Threshold Neurons
By choosing appropriate weights wi and threshold
we can place the line dividing the input space into
regions of output 0 and output 1in any position and
orientation.
orientation
Therefor
Visual Illusions
Visual Illusions
Visual Illusions demonstrate how we perceive an interpreted version of
the incoming light pattern rather that the exact pattern itself.
September 14, 2010
Neural Networks
Lecture 3: Models of Neurons and Neural Networks
H
Supervised Function Approximation
Supervised Function Approximation
In supervised learning, we train an ANN with a set of
vector pairs, soso-called exemplars
exemplars.
Each pair (x
(x, y) consists of an input vector x and a
corresponding output vector y.
Refresher: Perceptron Training Algorithm
Algorithm Perceptron
Perceptron;
Another Refresher: Linear Algebra
How can we visualize a straight line defined by an
equation such as w0 + w1i1 + w2i2 = 0?
Start with a randomly chosen weight vector w0;
Let k = 1;
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Assignment #1
Posted on September 26 due by October 7 at 4pm
Question 1: Perceptron Learning
The chart below shows a set of two-dimensional input samples from two classes:
1
Class1
0.9
Class1
0.8
0
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Assignment #3
Posted on November 7
Due by November 23 (Question 1)
and December 9 (Question 2)
Question 1: A Backpropagation Network for Digit Recognition
Now let us try a backpropagation network t
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Assignment #2
Posted on October 10 due by October 19, 4pm
Question 1: A Perceptron Network for Digit Recognition
There is an excellent online database of handwritten digits that is well-suited for
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Practice Midterm Exam
Duration: 75 minutes
No calculators, no books, and no notes are allowed (in the actual exam).
Question 1: _ out of _ points
Question 2: _ out of _ points
Question 3: _ out of
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Practice Exam
Duration: 2:30 hours
No calculators, no books, and no notes are allowed (in the actual exam).
Question 1: _ out of _ points
Question 2: _ out of _ points
Question 3: _ out of _ points
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Practice Exam Sample Solutions
Duration: 2:30 hours
No calculators, no books, and no notes are allowed (in the actual exam).
Question 1: _ out of _ points
Question 2: _ out of _ points
Question 3:
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Assignment #4
Sample Solutions
Question 2: A Hopfield Network for Beginners
(a) Determine the weight matrix for an auto-associative, discrete Hopfield Network (as
discussed in class) that has four
CS 672 Neural Networks Fall 2010
Instructor: Marc Pomplun
Assignment #1 Sample Solutions
Question1:
a) w0 = 0.2, w1 = 1, w2 = -1
Let = 0.5.
The input coordinates:
For Class 1:
cfw_(0.08,0.72), (0.26,0.58), (0.45,0.15), (0.60,0.30)
For Class -1: cfw_(0.10,