Overview Course and Basics
CSci 5525: Machine Learning Instructor: Arindam Banerjee
September 3, 2008
Instructor: Arindam Banerjee
Course Overview and Basics
General Information
Course Number: CSci 5525 Class: Mon Wed 04:00-05:15 pm Location: EE/CS 3-111 Instructor: Arindam Banerjee TA: Amrudin Agovic Office Hours:
Arindam: EE/CS 6-213 Mon Wed 5:15 pm - 6:15 pm Amrudin: EE/CS 2-209 Wed 10am-12noon
Web page: http://www-users.itlabs.umn.edu/classes/Fall-2008/csci5525 Email:
[e-mail] [e-mail]
Instructor: Arindam Banerjee Course Overview and Basics
Course Work: Homeworks
There will be 4 homeworks All submissions in pdf, all programming in Matlab (MALT) Ok to discuss with others, you have to write on your own, put the name(s) of people you discussed with Dates:
HW HW HW HW 1: 2: 3: 4: Sept 17 (Wed), due Sept 26 (Fri) at Noon Oct 01 (Wed), due Oct 10 (Fri) at Noon Nov 03 (Mon), due Nov 14 (Fri) at Noon Nov 24 (Mon), due Dec 05 (Fri) at Noon
Late submission policy in class webpage
Instructor: Arindam Banerjee
Course Overview and Basics
Course Work: Exams, etc.
Mid-Term: Oct 20 (Mon) in class Final: [Dec 15 (Mon), 4:00-6:00 pm] Closed book, Closed notes exam Allowed 1 sheet for mid-term, 2 sheets for final Participation:
Ask questions Participate in discussions
Web Links:
Online Submission for homeworks Bulletin Board for discussions
Instructor: Arindam Banerjee
Course Overview and Basics
Grading
Homework: 50 % = 4 12.5 % Mid-Term: 20 % Final: 25 % Participation: 5 %
Grading is absolute: A = 90-100, A- = 80-90, B+ = 75-80, B = 65-75, B- = 60-65, C+ = 55-60, C = 45-55, C- = 40-45, D+ = 30-40, D = 20-30, F = less than 20.
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines Kernel Methods
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines Kernel Methods Learning Theory
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines Kernel Methods Learning Theory Online Learning
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines Kernel Methods Learning Theory Online Learning Boosting
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines Kernel Methods Learning Theory Online Learning Boosting Clustering, Dimensionality Reduction
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines Kernel Methods Learning Theory Online Learning Boosting Clustering, Dimensionality Reduction Semi-supervised Learning
Instructor: Arindam Banerjee
Course Overview and Basics
Topics
Linear discriminants Probabilistic models: Generative, Discriminative Perceptrons Neural Networks Decision Trees Support Vector Machines Kernel Methods Learning Theory Online Learning Boosting Clustering, Dimensionality Reduction Semi-supervised Learning Structured Prediction
Instructor: Arindam Banerjee Course Overview and Basics
Overview
Key Concepts:
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Algorithms:
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Algorithms:
Linear Models, Layered Linear Models
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Algorithms:
Linear Models, Layered Linear Models Ensemble Models, Online Learning
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Algorithms:
Linear Models, Layered Linear Models Ensemble Models, Online Learning Advances: Semi-supervised learning, structured prediction
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Algorithms:
Linear Models, Layered Linear Models Ensemble Models, Online Learning Advances: Semi-supervised learning, structured prediction
Theory:
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Algorithms:
Linear Models, Layered Linear Models Ensemble Models, Online Learning Advances: Semi-supervised learning, structured prediction
Theory:
Basics, Risk Minimization
Instructor: Arindam Banerjee
Course Overview and Basics
Overview
Key Concepts:
Model Selection Over-fitting, Regularization
Main Trade-offs/Issues:
Generative vs Discriminative Max Likelihood vs Max Margin
Algorithms:
Linear Models, Layered Linear Models Ensemble Models, Online Learning Advances: Semi-supervised learning, structured prediction
Theory:
Basics, Risk Minimization Error-rate Bounds, Regret Bounds
Instructor: Arindam Course Banerjee Overview and Basics
Key Concepts
Model selection
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection Trade-off prior belief with learning from observations
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection Trade-off prior belief with learning from observations Similar to Bayesian priors and Bayesian conditionals
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection Trade-off prior belief with learning from observations Similar to Bayesian priors and Bayesian conditionals Being conservative is a good idea
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection Trade-off prior belief with learning from observations Similar to Bayesian priors and Bayesian conditionals Being conservative is a good idea
Overfitting
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection Trade-off prior belief with learning from observations Similar to Bayesian priors and Bayesian conditionals Being conservative is a good idea
Overfitting
Predict well on training set, poorly on test set/future data
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection Trade-off prior belief with learning from observations Similar to Bayesian priors and Bayesian conditionals Being conservative is a good idea
Overfitting
Predict well on training set, poorly on test set/future data Result of greedy/non-conservative learning
Instructor: Arindam Banerjee
Course Overview and Basics
Key Concepts
Model selection
"Bias" manual model selection "Learning" algorithmic model selection
Regularization
Guides model selection Trade-off prior belief with learning from observations Similar to Bayesian priors and Bayesian conditionals Being conservative is a good idea
Overfitting
Predict well on training set, poorly on test set/future data Result of greedy/non-conservative learning To be avoided using regularization
Instructor: Arindam Banerjee
Course Overview and Basics
Classification
Assume: A fixed (unknown) distribution on Rd {-1, +1} Given: A set T = {(x1 , y1 ), . . . , (xn , yn )} of n samples from the distribution. Problem: Find a function f : Rd {-1, +1} that has "low" error rate, i.e., L(f ) = P(f (x) = y ) is low. Let C be the set of functions over which f is searched for
"Bias" determines the set C A learning algorithm is the search algorithm in C
For Multiclass problems, (x, y ) Rd {1, . . . , c} For Regression problems, (x, y ) Rd R For Multi-dimensional Regression problems, (x, y) Rd Rk
Instructor: Arindam Banerjee Course Overview and Basics
Generative Vs Discriminative
Generative:
Assume a (parametric) model for p(x|y ) Training Estimating parameters of the model Prediction using Bayes rule p(y |x) = p(x|y )p(y ) p(x)
Example: Linear Discriminant Analysis
Discriminative:
Do not assume a model for p(x|y ), and hence p(x)
Assume a model for p(y |x) Direct formulation in terms of loss
Example: Logistic ...