lecture2-annotated - Machine Learning 10-701/15-781 Fall...

Info iconThis preview shows pages 1–6. Sign up to view the full content.

View Full Document Right Arrow Icon
1 © Eric Xing @ CMU, 2006-2008 Machine Learning Machine Learning 10 10 -701/15 701/15 -781, Fall 2008 781, Fall 2008 Theory of Classification Theory of Classification and and Nonparametric Classifier Nonparametric Classifier Eric Xing Eric Xing Lecture 2, September 10, 2008 Reading: Chap. 2.5 CB and handouts © Eric Xing @ CMU, 2006-2008 Miscellaneous z TA introduction! z Recitation time/location to be changed, now 6-7pm here! z Did you receive our announcements? z Logistic questions?
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 © Eric Xing @ CMU, 2006-2008 Classification z Representing data: z Hypothesis (classifier) = K X X X X M 2 1 K n n n x x x M 2 1 © Eric Xing @ CMU, 2006-2008 Outline z What is theoretically the best classifier z Probabilistic theory of classification z Discrete density estimation and Bayesian theorem z Bayesian decision rule for Minimum Error z Nonparametric Classifier (Instance-based learning learning ) z Nonparametric density estimation z K-nearest-neighbor classifier z Optimality of kNN z Problem of kNN
Background image of page 2
3 © Eric Xing @ CMU, 2006-2008 Decision-making as dividing a high-dimensional space z Distributions of samples from normal and abnormal cells © Eric Xing @ CMU, 2006-2008 Density Estimation z A Density Estimator learns a mapping from a set of attributes to a Probability Probability z Often know as parameter estimation if the distribution form is specified z Binomial, Gaussian … z Four important issues: z Nature of the data (iid, correlated, …) z Objective function (MLE, MAP, …) z Algorithm (simple algebra, gradient methods, EM, …) z Evaluation scheme (likelihood on test data, predictability, consistency, …)
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
4 © Eric Xing @ CMU, 2006-2008 Density Estimation Schemes Data ) , , ( 1 1 1 n x x K ) , , ( 1 n M M x x K K ) , , ( 2 1 2 n x x K Objective functions Maximum likelihood Bayesian Conditional likelihood Margin Score param 5 10 15 10 K 3 10 Algorithm Analytical Gradient EM Sampling iid correlated © Eric Xing @ CMU, 2006-2008 Parameter Learning from iid Data z Goal: estimate distribution parameters θ from a dataset of N independent , identically distributed ( iid ), fully observed , training cases D = { x 1 , . . . , x N } z Maximum likelihood estimation (MLE) 1. One of the most common estimators 2. With iid and full-observability assumption, write L ( ) as the likelihood of the data: 3. pick the setting of parameters most likely to have generated the data we saw: ) ; , , ( ) ( , N x x x P L K 2 1 = = = = N i i N x P x P x P x P 1 2 1 ) ; ( ) ; ( , ), ; ( ) ; ( K ) ( max arg * L = ) ( log max arg L =
Background image of page 4
5 © Eric Xing @ CMU, 2006-2008 Example: Bernoulli model z Data: z We observed N iid coin tossing: D ={1, 0, 1, …, 0} z Representation: Binary r.v: z Model: z How to write the likelihood of a single observation x i ? z The likelihood of dataset ={ x 1 , …,x N }: i i x x i x P = 1 1 ) ( ) ( θ () = = = = N i x x N i i N i i x P x x x P 1 1 1 2 1 1 ) ( ) | ( ) | ,..., , ( } , { 1 0 = n x tails # head # ) ( ) ( = = = = 1 1 1 1 1 N i i N i i x x x x x P = 1 1 ) ( ) ( = = = 1 for 0 for 1 ) ( x x x P © Eric Xing @ CMU, 2006-2008 Maximum Likelihood Estimation z Objective function: z We need to maximize this w.r.t.
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 6
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 01/26/2010 for the course MACHINE LE 10701 taught by Professor Ericp.xing during the Fall '08 term at Carnegie Mellon.

Page1 / 28

lecture2-annotated - Machine Learning 10-701/15-781 Fall...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online