This preview shows page 1. Sign up to view the full content.
Unformatted text preview: Pattern Recognition
ECE‐8443 Chapter 4
Nonparametric Classification Techniques
(Sections 4.34.5 of the textbook) Electrical and Computer Engineering Department,
Mississippi State University. 1 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Outline
Parzen Window (cont.)
Kn –Nearest Neighbor Estimation
The Nearest‐Neighbor Rule 2 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows (cont.)
• Parzen Windows – Probabilistic Neural Networks
• Compute a Parzen estimate based on n patterns . – Patterns with d features sampled from c classes
– The input unit is connected to n patterns 3 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department . Parzen Windows (cont.) 4 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows (cont.)
• Training the network
– Algorithm
1. Normalize each pattern x of the training set to 1
2. Place the first training pattern on the input units
3. Set the weights linking the input units and the first pattern units such that: w1 = x1
4. Make a single connection from the first pattern unit to the category unit corresponding to the known class of that pattern
5. Repeat the process for all remaining training patterns by setting the weights such that wk = xk (k = 1, 2, …, n)
We finally obtain the following network 5 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows (cont.) 6 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows (cont.)
• Testing the network
– Algorithm
1. Normalize the test pattern x and place it at the input units
2. Each pattern unit computes the inner product in order to yield the net activation t
net k = w k .x ⎡ net k − 1 ⎤
2
⎥
⎣σ
⎦ and emit a nonlinear function f ( net k ) = exp ⎢ 3. Each output unit sums the contributions from all pattern units connected to it
n Pn ( x  ω j ) = ∑ ϕ i ∝ P ( ω j  x )
i =1 4. Classify by selecting the maximum value of Pn(x  ωj) (j = 1, …, c)
7 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
• Kn ‐ Nearest neighbor estimation
– Goal: a solution for the problem of the unknown “best” window function
• Let the cell volume be a function of the training data
• Center a cell about x and let it grows until it captures kn samples (kn = f(n))
• kn are called the kn nearest‐neighbors of x 2 possibilities can occur:
• Density is high near x; therefore the cell will be small which provides a good resolution
• Density is low; therefore the cell will grow large and stop until higher density regions are reached We can obtain a family of estimates by setting kn=k1√n and choosing different values for k1
8 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
Illustration
For kn = √n = 1 ; the estimate becomes:
Pn(x) = kn / n.Vn = 1 / V1 =1 / 2x‐x1 9 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation 10 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation 11 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
– Estimation of a‐posteriori probabilities
• Goal: estimate P(ωi  x) from a set of n labeled samples
– Let’s place a cell of volume V around x and capture k samples
– ki samples amongst k turned out to be labeled ωi then: pn(x, ωi) = ki /n.V
An estimate for pn(ωi x) is: pn ( ω i  x ) = p n ( x ,ω i )
j =c ∑ p ( x ,ω
j =1 12 Chapter 4 Saurabh Prasad n Pattern Recognition j ) ki
=
k Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
– ki/k is the fraction of the samples within the cell that are labeled ωi
– For minimum error rate, the most frequently represented category within the cell is selected
– If k is large and the cell sufficiently small, the performance will approach the best possible 13 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
• The nearest–neighbor rule
– Let Dn = {x1, x2, …, xn} be a set of n labeled prototypes
– Let x’ ∈ Dn be the closest prototype to a test point x then the nearest‐
neighbor rule for classifying x is to assign it the label associated with x’
– The nearest‐neighbor rule leads to an error rate greater than the minimum possible: the Bayes rate
– If the number of prototype is large (unlimited), the error rate of the nearest‐neighbor classifier is never worse than twice the Bayes rate (it can be demonstrated!)
– If n → ∞, it is always possible to find x’ sufficiently close so that:
P(ωi  x’) ≅ P(ωi  x) 14 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
– If P(ωm  x) ≅ 1, then the nearest neighbor selection is almost always the same as the Bayes selection 15 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation 16 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
– The k – nearest‐neighbor rule
• Goal: Classify x by assigning it the label most frequently represented among the k nearest samples and use a voting scheme 17 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation 18 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department K‐Nearest Neighbor Estimation
Example:
k = 3 (odd value) and x = (0.10, 0.25)t Prototypes
(0.15,
(0.10,
(0.09,
(0.12, 0.35)
0.28)
0.30)
0.20) Labels ω1
ω2
ω5
ω2 Closest vectors to x with their labels are:
{(0.10, 0.28, ω2); (0.12, 0.20, ω2); (0.15, 0.35,ω1)}
One voting scheme assigns the label ω2 to x since ω2 is the most frequently represented
19 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department ...
View Full
Document
 Spring '10
 Staff

Click to edit the document details