This preview shows page 1. Sign up to view the full content.
Unformatted text preview: Pattern Recognition
ECE‐8443 Chapter 4
Nonparametric Classification Techniques
Electrical and Computer Engineering Department,
Mississippi State University. 1 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Outline
• Introduction
• Density Estimation
• Parzen Windows 2 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Introduction
• All Parametric densities are unimodal (have a single local maximum), whereas many practical problems involve multi‐
modal densities
– Exception: GMMs • Nonparametric procedures can be used with arbitrary distributions and without the assumption that the forms of the underlying densities are known
• There are two types of nonparametric methods:
– Estimating P(x  ωj ) – Bypass probability and go directly to a‐posteriori probability estimation
3 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Density Estimation
– Basic idea:
– Probability that a vector x will fall in region R is: P = ∫ p( x' )dx' (1) ℜ – P is a smoothed (or averaged) version of the density function p(x). If we have a sample of size n, the probability that k points fall in R is then:
⎛ n⎞ k
Pk = ⎜ ⎟ P ( 1 − P )n − k
⎜k⎟
⎝⎠ (2) and the expected value for k is:
E(k) = nP
4 Chapter 4 Saurabh Prasad Pattern Recognition
4 (3)
Electrical and Computer Engineering Department Density Estimation
ML estimation of P = θ
max( Pk  θ ) is reached for
θ θˆ = k
≅P
n Therefore, the ratio k/n is a good estimate for the probability P and hence for the density function p. p(x) is continuous and that the region R is so small that p does not vary significantly within it, we can write: ∫ p( x' )dx' ≅ p( x)V (4) ℜ where is a point within R and V the volume enclosed by R.
5 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Density Estimation
• Justification of equation (4) ∫ p( x' )dx' ≅ p( x )V (4) ℜ We assume that p(x) is continuous and that region R is so small that p does not vary significantly within R. Since p(x) = constant, it is not a part of the sum/integral. 6 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Density Estimation
∫ p( x' )dx' = p( x' )∫ dx' = p( x' )∫ 1 ℜ ℜ ℜ ( x )dx' = p( x' )μ ( ℜ ) ℜ Where: μ(R) is: a surface in the Euclidean space R2
a volume in the Euclidean space R3
a hypervolume in the Euclidean space Rn
Since p(x) ≅ p(x’) = constant, therefore in the Euclidean space R3: ∫ p( x' )dx' ≅ p( x ).V ℜ k
and p( x ) ≅
nV
7 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Density Estimation
Combining equation (1) , (3) and (4) yields: p( x ) ≅ 8 Chapter 4 Saurabh Prasad Pattern Recognition k/n
V Electrical and Computer Engineering Department Density Estimation
– Condition for convergence
The fraction k/(nV) is a space averaged value of p(x).
p(x) is obtained only if V approaches zero. lim p( x ) = 0 (if n = fixed) V →0 , k = 0 This is the case where no samples are included in R: it is an uninteresting case! lim p( x ) = ∞ V →0 , k ≠ 0 In this case, the estimate diverges: it is an uninteresting case!
9 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Density Estimation
• The volume V needs to approach 0 anyway if we want to use this estimation
• Practically, V cannot be allowed to become small since the number of samples is always limited
• One will have to accept a certain amount of variance in the ratio k/n
• Theoretically, if an unlimited number of samples is available, we can circumvent this difficulty
To estimate the density of x, we form a sequence of regions
R1, R2,…containing x: the first region contains one sample, the second two samples and so on.
Let Vn be the volume of Rn, kn the number of samples falling in Rn and pn(x)
be the nth estimate for p(x):
pn(x) = (kn/n)/Vn 10 Chapter 4 Saurabh Prasad Pattern Recognition (7)
Electrical and Computer Engineering Department Density Estimation
Three necessary conditions should apply if we want pn(x) to converge to p(x): 1 ) lim Vn = 0
n→ ∞ 2 ) lim k n = ∞
n→∞ 3 ) lim k n / n = 0
n→∞ There are two different ways of obtaining sequences of regions that satisfy these conditions:
(a) Shrink an initial region where Vn = 1/√n and show that pn ( x ) → p( x )
n→∞ This is called “the Parzen‐window estimation method”
(b) Specify kn as some function of n, such as kn = √n; the volume Vn is
grown until it encloses kn neighbors of x. This is called “the kn‐nearest neighbor estimation method”
11 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Density Estimation 12 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows
– Parzen‐window approach to estimate densities assume that the region Rn is a d‐dimensional hypercube
d
Vn = hn (hn : length of the edge of ℜ n ) Let ϕ (u) be the following window function :
1
⎧
j = 1,... , d
⎪1 u j ≤
ϕ (u) = ⎨
2
⎪0 otherwise
⎩
– ϕ((x‐xi)/hn) is equal to unity if xi falls within the hypercube of volume Vn centered at x and equal to zero otherwise.
13 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows
– The number of samples in this hypercube is:
⎛ x − xi
kn = ∑ ϕ ⎜
⎜h
i =1
n
⎝
i=n ⎞
⎟
⎟
⎠ By substituting kn in equation (7), we obtain the following estimate:
1 i = n 1 ⎛ x − xi
ϕ⎜
pn ( x) = ∑
n i =1 V n ⎜ h n
⎝ ⎞
⎟
⎟
⎠ Pn(x) estimates p(x) as an average of functions of x and the samples (xi) (i = 1,… ,n). These functions ϕ can be general! 14 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows
1 ⎛x⎞
δ n ( x) =
ϕ⎜ ⎟
V n ⎜ hn ⎟
⎝⎠ 15 Chapter 4 Saurabh Prasad Pattern Recognition Window functions, in some sense are helping in interpolating between the observed samples to estimate the overall density
Broad hn: Smooth, out of focus estimates
Narrow hn : Noisy estimates Electrical and Computer Engineering Department Parzen Windows 16 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows
– Illustration
• The behavior of the Parzen‐window method
– Case where p(x) N(0,1) Let ϕ(u) = 1/√(2π) exp(‐u2/2) and hn = h1/√n (n>1)
(h1: known parameter) Thus: 1 i = n 1 ⎛ x − xi
pn ( x ) = ∑ ϕ ⎜
n i = 1 hn ⎜ hn
⎝ ⎞
⎟
⎟
⎠ is an average of normal densities centered at the samples xi.
17 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows
– Numerical results: For n = 1 and h1=1 1 −1 / 2
e
( x − x1 )2 → N ( x1 ,1 )
p1 ( x ) = ϕ ( x − x1 ) =
2π For n = 10 and h = 0.1, the contributions of the individual samples are clearly observable (C.f. next slide) !
18 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows 19 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Parzen Windows 20 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Analogous results are also obtained in two dimensions as illustrated: 21 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Analogous results are also obtained in two dimensions as illustrated: 22 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Case where p(x) = λ1.U(a,b) + λ2.T(c,d) (unknown density) (mixture of a uniform and a triangle density) 23 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Case where p(x) = λ1.U(a,b) + λ2.T(c,d) (unknown density) (mixture of a uniform and a triangle density) 24 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Classification using Parzen Windows
– Classification example
In classifiers based on Parzen‐window estimation:
• We estimate the densities for each category and classify a test point by the label corresponding to the maximum posterior
• The decision region for a Parzen‐window classifier depends upon the choice of window function as illustrated in the following figure. 25 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department Classification using Parzen Windows 26 Chapter 4 Saurabh Prasad Pattern Recognition Electrical and Computer Engineering Department ...
View
Full
Document
This note was uploaded on 02/20/2012 for the course ECE 8443 taught by Professor Staff during the Spring '10 term at University of Houston.
 Spring '10
 Staff

Click to edit the document details