lecture9-annotated - Machine Learning 10-701/15-781 Fall...

Info iconThis preview shows pages 1–7. Sign up to view the full content.

View Full Document Right Arrow Icon
1 © Eric Xing @ CMU, 2006-2008 1 Machine Learning Machine Learning 10 10 -701/15 701/15 -781, Fall 2008 781, Fall 2008 Ensemble methods Ensemble methods Boosting from Weak Learners Boosting from Weak Learners Eric Xing Eric Xing Lecture 9, October 6, 2008 Reading: Chap. 14.3 C.B book © Eric Xing @ CMU, 2006-2008 2 z Project proposal due this Wed z Mid term
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
2 © Eric Xing @ CMU, 2006-2008 3 The essence of kernel z Feature mapping, but “without paying a cost” z E.g., polynomial kernel z How many dimensions we’ve got in the new space? z How many operations it takes to compute K()? z Kernel design, any principle? z K(x,z) can be thought of as a similarity function between x and z z This intuition can be well reflected in the following “Gaussian” function (Similarly one can easily come up with other K() in the same spirit) z Is this necessarily lead to a “legal” kernel? (in the above particular case, K() is a legal one, do you know how many dimension φ (x) is? © Eric Xing @ CMU, 2006-2008 4 (3) Structured Prediction “Do you want sugar in it?” <verb pron verb noun prep pron> z Unstructured prediction z Structured prediction z Part of speech tagging z Image segmentation
Background image of page 2
3 © Eric Xing @ CMU, 2006-2008 5 OCR example brace Sequential structure xy a-z a-z a-z a-z a-z y x © Eric Xing @ CMU, 2006-2008 6 z Inputs: z a set of training samples , where and z Outputs: z a predictive function : z Examples: z SVM: z Logistic Regression: where Classical Classification Models
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
4 © Eric Xing @ CMU, 2006-2008 7 z Assumptions: z Linear combination of features z Sum of partial scores: index p represents a part in the structure z Random fields or Markov network features: Structured Models space of feasible outputs discriminant function © Eric Xing @ CMU, 2006-2008 8 Discriminative Learning Strategies z Max Conditional Likelihood z We predict based on: z And we learn based on: z Max Margin: z We predict based on: z And we learn based on: = = c c c c f w Z p ) , ( exp ) , ( 1 ) | ( max arg | * y x x w x y x y w y {} = = i c i i c c i i i i i i f w Z p ) , ( exp ) , ( 1 ) | ( max arg , | * y x x w x y x y w w w ) , ( max arg ) , ( max arg | * y x w y x x y y f f w T y c c c c = = ( ) = ) , ( ) , ( min max arg , | , * i i i T i i i f f i x y x y w x y w y y w
Background image of page 4
5 © Eric Xing @ CMU, 2006-2008 9 E.g. Max-Margin Markov Networks z Convex Optimization Problem: z Feasible subspace of weights: z Predictive Function: © Eric Xing @ CMU, 2006-2008 10 OCR Example z We want: argmax word w T f (, word ) = “brace” z Equivalently: w T f “brace” ) > w T f ( , “aaaaa”) w T f “brace” ) > w T f ( , “aaaab”) w T f “brace” ) > w T f ( , “zzzzz”) a lot!
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
6 © Eric Xing @ CMU, 2006-2008 11 z Brute force enumeration of constraints: z The constraints are exponential in the size of the structure z Alternative: min-max formulation z add only the most violated constraint z Handles more general loss functions z Only polynomial # of constraints needed z Several algorithms exist … Min-max Formulation © Eric Xing @ CMU, 2006-2008 12 Results: Handwriting Recognition Length: ~8 chars Letter: 16x8 pixels 10-fold Train/Test 5000/50000 letters 600/6000 words Models: M 3 nets *Crammer & Singer 01 0 5 15
Background image of page 6
Image of page 7
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

Page1 / 23

lecture9-annotated - Machine Learning 10-701/15-781 Fall...

This preview shows document pages 1 - 7. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online