MSE Bias 2 Variance The tradeoff depends on several factors Intrinsic

Mse bias 2 variance the tradeoff depends on several

This preview shows page 81 - 92 out of 124 pages.

MSE = Bias 2 + Variance The tradeoff depends on several factors Intrinsic complexity of the phenomenon to be predicted, Size of the training set: the larger the better, Size of the feature vectors: larger or smaller? 68
Image of page 81

Subscribe to view the full document.

Machine learning – Overfitting and Generalization Curse of dimensionality Is there a (hyper)plane that perfectly separates dogs from cats? No perfect separation No perfect separation Linearly separable case Looks like the more features we have, the better it is. But. . . (Source: Vincent Spruyt) 69
Image of page 82
Machine learning – Overfitting and Generalization Curse of dimensionality Is there a (hyper)plane that perfectly separates dogs from cats? Yes, but overfitting No, but better on unseen data (Source: Vincent Spruyt) 70
Image of page 83

Subscribe to view the full document.

Machine learning – Overfitting and Generalization Curse of dimensionality Is there a (hyper)plane that perfectly separates dogs from cats? Yes, but overfitting No, but better on unseen data Why is that? (Source: Vincent Spruyt) 70
Image of page 84
Machine learning – Overfitting and Generalization Curse of dimensionality The amount of training data needed to cover 20% of the feature range grows exponentially with the number of dimensions. Reducing the feature dimension is often favorable. “Many algorithms that work fine in low dimensions become intractable when the input is high-dimensional.” Bellman, 1961. (Source: Vincent Spruyt) 71
Image of page 85

Subscribe to view the full document.

Machine learning – Feature engineering Feature engineering Feature selection: choice of distinct traits used to describe each sample in a quantitative manner. Ex: fruit acidity, bitterness, size, weight, number of seeds, . . . Correlations between features: weight vs size, seeds vs bitterness, . . . . Information is redundant and can be summarized with less but more relevant features. Feature extraction: extract/generate new features from the initial set of features intended to be informative, non-redundant and facilitating the subsequent task. Common procedure: Principal Component Analysis (PCA) 72
Image of page 86
Machine learning – Principal Component Analysis (PCA) Principal Component Analysis (PCA) In most applications examples are not spread uniformly throughout the example space, but are concentrated on or near a low-dimensional subspace/manifold. No correlations Both features are informative, No dimensionality reductions. Strong correlation Features “influence” each other, Dimensionality reductions possible. 73
Image of page 87

Subscribe to view the full document.

Machine learning – Principal Component Analysis (PCA) Principal Component Analysis (PCA) 74
Image of page 88
Machine learning – Principal Component Analysis (PCA) Principal Component Analysis (PCA) Find the principal axes (eigenvectors of the covariance matrix) , 74
Image of page 89

Subscribe to view the full document.

Machine learning – Principal Component Analysis (PCA) Principal Component Analysis (PCA) Find the principal axes (eigenvectors of the covariance matrix), Keep the ones with largest variations (largest eigenvalues) , 74
Image of page 90
Machine learning – Principal Component Analysis (PCA) Principal Component Analysis (PCA) Find the principal axes (eigenvectors of the covariance matrix), Keep the ones with largest variations (largest eigenvalues), Project the data on this low-dimensional space , 74
Image of page 91

Subscribe to view the full document.

Image of page 92

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern

Ask Expert Tutors You can ask 0 bonus questions You can ask 0 questions (0 expire soon) You can ask 0 questions (will expire )
Answers in as fast as 15 minutes