10: Lecture Density estimation II
g g g g g g
Parzen windows Smooth kernels Bandwidth selection for univariate data Multivariate density estimation Product kernels Nave Bayes classifier
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
1
KDE: Parzen windows (1)
g In the previous lecture we found out that the non-parametric density estimate was
V is the volume surrounding x k where N is the total number of examples P(x) NV k is the number of examples inside V
g Suppose that the region that encloses the k examples is a hypercube with sides of length h centered at the estimation point x
n Then its volume is given by V=hD, where D is the number of dimensions
g To find the number of examples that fall within this region we define a kernel function K(u)
1 u j < 1/ 2 K (u) = 0 otherwise j = 1,.., D
n This kernel, which corresponds to a unit hypercube centered at the origin , is known as a Parzen window or the nave estimator
g The total number of points inside the hypercube is then
x x(n k = K h n =1
N
x x4 x3 x2 V x1
1/V
K(x-x1) x1 K(x-x2) x2 K(x-x3) x3 K(x-x4)
n K((x-x(n)/h) is equal to unity if and only if the point x(n falls inside a hypercube of side h centered at x
g Substituting back into the expression for the density estimate
PKDE ( x ) = 1 NhD
K
n =1
N
xx h
(n
g Notice that the Parzen window density estimate resembles the histogram, except that the cell locations are determined by the data points
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
x4
2
KDE: Parzen windows (2)
g To understand the role of the kernel function we compute the expectation of the probability estimate P(x)
1 E[PKDE (x )] = NhD x x ( n EK h = n =1 1 x x( n = = D E K h h 1 x x = D K P( x )dx h h
N
n where we have assumed that the vectors x(n are drawn independently from the true density P(x)
g We can see that the expectation of the estimated density PKDE(x) is a convolution of the true density P(x) with the kernel function
n The width w of the kernel plays the role of a smoothing parameter: the wider the kernel function, the smoother the estimate PKDE(x)
g For h0, the kernel approaches a delta function and PKDE(x) approaches the true density
n However, in practice we have a finite number of points, so h cannot be made arbitrarily small, since the density estimate PKDE(x) approaches a set of delta functions centered at the data points
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
3
KDE: smooth kernels
g The Parzen window has several drawbacks
n Yields density estimates that have discontinuities n Weights equally all the points xi, regardless of their distance to the estimation point x
g It is easy to to overcome some of these difficulties by generalizing the Parzen window with a smooth kernel function K(u) which satisfies the condition
RD
K (x )dx = 1
K (x ) = 1 exp x T x D/2 (2 ) 2 1
n Usually, but not not always, K(u) will be a radially symmetric, unimodal probability density function, such as the multivariate Gaussian density function
n where the expression of the density estimate remains the same as with Parzen windows
1 PKDE ( x ) = NhD
x x( n K h n =1
N
0.045 0.04 0.035 0.03 PKDE(x); h=3 0.025 0.02
Density estimate
g Just as the Parzen window estimate can be considered a sum of boxes centered at the observations, the smooth kernel estimate is a sum of bumps placed at the observations
n The kernel function determines the shape of the bumps n The parameter h, also called the smoothing parameter or bandwidth, determines their width
Kernel functions
0.015 0.01 0.005 0 -10
-5
0
5
10
15 x
20
25
30
35
40
Data points
4
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
Choosing the bandwidth: univariate case (1)
g The problem of choosing the bandwidth is crucial in density estimation
n A large bandwidth will over-smooth the density and mask the structure in the data n A small bandwidth will yield a density estimate that is spiky and very hard to interpret
0.09 0.08 0.07 0.06 PKDE(x); h=1.0 0.05 0.04 0.03 0.02 0.01 0 -10 0.05 0.045 0.04 0.035 PKDE(x); h=2.5 -5 0 5 10 15 x 20 25 30 35 40 0.03
0.025 0.02
0.015 0.01 0.005 0 -10 -5 0 5 10 15 x 20 25 30 35 40
0.035
0.03
0.03
0.025
0.025 0.02 0.02 PKDE(x); h=10.0 -5 0 5 10 15 x 20 25 30 35 40 PKDE(x); h=5.0
0.015
0.015
0.01 0.01 0.005
0.005
0 -10
0 -10
-5
0
5
10
15 x
20
25
30
35
40
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
5
Choosing the bandwidth: univariate case (2)
g We would like to find a value of the smoothing parameter that minimizes the error between the estimated density and the true density
n A natural measure is the mean square error at the estimation point x, defined by
MSE x (PKDE ) = E (PKDE (x ) P(x )) = {E[PKDE (x ) P(x )] } + var (PKDE (x ))
2 2 bias variance
[
] 144 2444 14243 4 3 4 4
g This expression is an example of the bias-variance dilemma of statistics: the bias can be reduced at the expense of the variance, and vice versa
n The bias of an estimate is the systematic error incurred in the estimation n The variance of an estimate is the random error incurred in the estimation
g The bias-variance dilemma applied to bandwidth selection simply means that
n A large bandwidth will reduce the differences among the estimates of PKDE(x) for different data sets (the variance) but it will increase the bias of PKDE(x) with respect to the true density P(x) n A small bandwidth will reduce the bias of PKDE(x), at the expense of a larger variance in the estimates PKDE(x)
BIAS
0.4 0.35 0.3 0.5 0 . 2 = h ; ) x (E P 0.25 0.2 1 0.4 . 0 = h ; ) x ( E 0.3 P
D K
VARIANCE
0.7
0.6
D K 0.15
0.1 0.05 0 -3
0.2
0.1
-2
-1
0 x
1
2
3
0 -3
-2
-1
0 x
1
2
3
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
6
Bandwidth selection methods, univariate case (1)
g Subjective choice
n The natural way for choosing the smoothing parameter is to plot out several curves and choose the estimate that is most in accordance with ones prior (subjective) ideas n However, this method is not practical in pattern recognition since we typically have highdimensional data
g Reference to a standard distribution
n Assume a standard density function and find the value of the bandwidth minimizes that the integral of the square error (MISE)
hopt = arg min{MISE(PKDE (x )) } = arg min E (PKDE (x ) P(x )) dx
2 h h
{[
]}
n If we assume that the true distribution is a Gaussian density and we use a Gaussian kernel, it can be shown that the optimal value of the bandwidth becomes [Silverman]
hopt = 1.06 N1/ 5
g where is the sample variance and N is the number of training examples
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
7
Bandwidth selection methods, univariate case
n Better results can be obtained if we use a robust measure of the spread instead of the sample variance and we reduce the coefficient 1.06 to better cope with multimodal densities [Silverman]. With this in mind, the optimal bandwidth becomes
hopt = 0.9 AN1/ 5 IQR where A = min , 1.34
g IQR is the interquartile range, a robust estimate of the spread. It is computed as one half the difference between the 75th percentile (Q3) and the 25th percentile (Q1). The formula for semiinterquartile range is therefore: (Q3-Q1)/2
n
A percentile rank is the proportion of examples in a distribution that a specific example is greater than or equal to
g Likelihood cross-validation
n The ML estimate of h is degenerate since it yields hML=0, a density estimate with delta functions at each training data point n An practical alternative is to maximize the pseudo-likelihood computed using crossvalidation
1 N hMLCV = arg max log fi x( n h N n=1
( )
where fi x
( )
(m
1 = (N 1)h
x( m x (n m K h n =1,n
N
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
8
Multivariate density estimation
g The derived expression of the estimate PKDE(x) for multiple dimensions was
1 PKDE ( x ) = NhD x x( n K h n =1
N
n Notice that the bandwidth h is the same for all the axes, so this density estimate will be weight all the axis equally
g However, if the spread of the data is much greater in one of the coordinate directions than the others, we should use a vector of smoothing parameters or even a full covariance matrix, which complicates the procedure g There are two basic alternatives to solve the scaling problem without having to use a more general kernel density estimate
n Pre-scale each axis (normalize to unit variance, for instance)
n Pre-whiten the data (linearly transform to have unit covariance matrix), estimate the density,
and then transform back [Fukunaga]
g The whitening transform is simply y=-1/2MTx,
where and M are the eigenvalue and eigenvector matrices of the sample covariance of x g Fukunagas method is equivalent to using a hyper-ellipsoidal kernel
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University
9
Product kernels
g A very common method of performing multivariate density estimation is the product kernel, defined as
PPKDE (x ) = 1 N K x, x (n ,h1,..., hD N i=1
(
) )
D x( d) x (n ( d) 1 Kd h1 hD d=1 hd
where K x, x( n , h1,..., hD =
(
n The product kernel consists of the product of one-dimensional kernels
g Typically the same kernel function is used in each dimension ( Kd(x)=K(x) ), and only the bandwidths are allowed to differ
n Bandwidth selection can then be performed with any of the methods presented for univariate density estimation
g It is important to notice that although the expression of K(x,x(n,h1,hD) uses kernel independence, this does not imply that any type of feature independence is being assumed
n A density estimation method that assumed feature independence would have the following expression D x(d) x (n (d) 1 N PFEAT IND (x ) = Kd Nh hd d=1 d i=1 n Notice how the order of the summation and product are reversed compared to the product kernel
Introduction to Pattern Recognition Ricardo Gutierrez-Osuna Wright State University 10
Product kernel, example...