This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: Journal of Machine Learning Research 3 (2003) 1229-1243 Submitted 5/02; Published 3/03 Dimensionality Reduction via Sparse Support Vector Machines Jinbo Bi [email protected] Kristin P. Bennett [email protected] Department of Mathematical Sciences Rensselaer Polytechnic Institute Troy, NY 12180, USA Mark Embrechts [email protected] Department of Decision Science and Engineering Systems Rensselaer Polytechnic Institute Troy, NY 12180, USA Curt M. Breneman [email protected] Minghu Song [email protected] Department of Chemistry Rensselaer Polytechnic Institute Troy, NY 12180, USA Editors: Isabelle Guyon and Andre Elisseeff Abstract We describe a methodology for performing variable ranking and selection using support vector machines (SVMs). The method constructs a series of sparse linear SVMs to generate linear models that can generalize well, and uses a subset of nonzero weighted variables found by the linear models to produce a final nonlinear model. The method exploits the fact that a linear SVM (no kernels) with 1-norm regularization inherently performs variable selection as a side-effect of minimizing capacity of the SVM model. The distribution of the linear model weights provides a mechanism for ranking and interpreting the effects of variables. Starplots are used to visualize the magnitude and variance of the weights for each variable. We illustrate the effectiveness of the methodology on synthetic data, benchmark problems, and challenging regression problems in drug design. This method can dramatically reduce the number of variables and outperforms SVMs trained using all attributes and using the attributes selected according to correlation coefficients. The visualization of the resulting models is useful for understanding the role of underlying variables. Keywords: Variable Selection, Dimensionality Reduction, Support Vector Machines, Regression, Pattern Search, Bootstrap Aggregation, Model Visualization 1. Introduction Variable selection refers to the problem of selecting input variables that are most predictive of a given outcome. Appropriate variable selection can enhance the effectiveness and domain inter- pretability of an inference model. Variable selection problems are found in many supervised and unsupervised machine learning tasks including classification, regression, time series prediction, clustering, etc. We shall focus on supervised regression tasks, but the general methodology can c 2003 Jinbo Bi, Kristin Bennett, Mark Embrechts, Curt Breneman and Minghu Song. BI, BENNETT, EMBRECHTS, BRENEMAN AND SONG be extended to any inference task that can be formulated as an 1-norm SVM, such as classifica- tion and novelty detection (Campbell and Bennett, 2000, Bennett and Bredensteiner, 1997). The objective of variable selection is two-fold: improving prediction performance (Kittler, 1986) and enhancing understanding of the underlying concepts in the induction model....
View Full Document
This note was uploaded on 11/09/2011 for the course CIS 6930 taught by Professor Staff during the Fall '08 term at University of Florida.
- Fall '08
- Machine Learning