NonlinearRegression - CSC 411 / CSC D11 Nonlinear...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
CSC 411 / CSC D11 Nonlinear Regression 3 Nonlinear Regression Sometimes linear models are not sufficient to capture the real-world phenomena, and thus nonlinear models are necessary. In regression, all such models will have the same basic form, i.e., y = f ( x ) (1) In linear regression, we have f ( x ) = Wx + b ; the parameters W and b must be fit to data. What nonlinear function do we choose? In principle, f ( x ) could be anything: it could involve linear functions, sines and cosines, summations, and so on. However, the form we choose will make a big difference on the effectiveness of the regression: a more general model will require more data to fit, and different models are more appropriate for different problems. Ideally, the form of the model would be matched exactly to the underlying phenomenon. If we’re modeling a linear process, we’d use a linear regression; if we were modeling a physical process, we could, in principle, model f ( x ) by the equations of physics. In many situations, we do not know much about the underlying nature of the process being modeled, or else modeling it precisely is too difficult. In these cases, we typically turn to a few models in machine learning that are widely-used and quite effective for many problems. These methods include basis function regression (including Radial Basis Functions), Artificial Neural Networks, and k -Nearest Neighbors. There is one other important choice to be made, namely, the choice of objective function for learning, or, equivalently, the underlying noise model. In this section we extend the LS estimators introduced in the previous chapter to include one or more terms to encourage smoothness in the estimated models. It is hoped that smoother models will tend to overfit the training data less and therefore generalize somewhat better. 3.1 Basis function regression A common choice for the function f ( x ) is a basis function representation 1 : y = f ( x ) = s k w k b k ( x ) (2) for the 1D case. The functions b k ( x ) are called basis functions. Often it will be convenient to express this model in vector form, for which we define b ( x ) = [ b 1 ( x ) , . . . , b M ( x )] T and w = [ w 1 , . . . , w M ] T where M is the number of basis functions. We can then rewrite the model as y = f ( x ) = b ( x ) T w (3) Two common choices of basis functions are polynomials and Radial Basis Functions (RBF) . A simple, common basis for polynomials are the monomials , i.e., b 0 ( x ) = 1 , b 1 ( x ) = x, b 2 ( x ) = x 2 , b 3 ( x ) = x 3 , ... (4) 1 In the machine learning and statistics literature, these representations are often referred to as linear regression, since they are linear functions of the “features” b k ( x ) Copyright c c 2009 Aaron Hertzmann and David Fleet 9
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
CSC 411 / CSC D11 Nonlinear Regression -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -8 -6 -4 -2 0 2 4 6 8 x Polynomial basis functions x 0 x 1 x 2 x 3 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 -1 -0.5 0 0.5 1 1.5 2 x Radial Basis Functions Figure 1: The first three basis functions of a polynomial basis, and Radial Basis Functions
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 11/09/2010 for the course CS CSCD11 taught by Professor Davidfleet during the Spring '10 term at University of Toronto- Toronto.

Page1 / 8

NonlinearRegression - CSC 411 / CSC D11 Nonlinear...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online