This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: STA 414/2104 Mar 9, 2010 Notes I Sample test questions posted I Review and/or questions on Thursday this week I Test will have 3 questions: one from Sample test, one specific to 414/2104 I Extra Office Hour Monday, March 15, 34 I Watch web site for late breaking announcement re MidTerm 1 / 20 STA 414/2104 Mar 9, 2010 Neural Networks I “feed forward single layer neural network” I Y k = g k { β k + M X m = 1 β km σ ( α m + p X ‘ = 1 α ‘ m X ‘ ) } = f k ( X ‘ ) I σ ( x ) = 1 1 + e x tanh ( x ) = e x e x e x + e x , maps to ( 1 , + 1 ) 2 / 20 STA 414/2104 Mar 9, 2010 ... neural networks I Y k = g k { β k + M X m = 1 β km σ ( α m + p X ‘ = 1 α ‘ m X ‘ ) } = f k ( X ‘ ) I θ = ( α m ,α m ,β k ,β k ) I R ( θ ) = ∑ N i = 1 ∑ K k = 1 { y ik f k ( x i ) } 2 , or I R ( θ ) = ∑ N i = 1 ∑ K k = 1 y ik log f k ( x i ) I dim ( θ ) = M ( p + 1 ) + K ( M + 1 )→ regularization/shrinkage, also called weight decay I minimize R ( θ ) + λ J ( θ ) = R ( θ ) + λ X km β 2 km + X m ‘ α 2 m ‘ ! I standardize inputs to mean 0, variance 1 for regularization I backfitting algorithm for minimizing R ( θ ) described in § 11.4; extension to R ( θ ) + λ J ( θ ) in § 11.5.2 3 / 20 STA 414/2104 Mar 9, 2010 ... neural networks I nnet in MASS library: recommend λ ∈ ( 10 4 , 10 2 ) for squared error loss; λ ∈ ( . 01 ,. 1 ) for loglikelihood I compare Figure 11.4 top/bottom I results very sensitive to starting values: R ( θ ) has many local maxima I recommendation (Ripley): take average predictions over several nnet fits I weight decay seems to be more important than number of hidden units I See § 11.7, 8, 9 for interesting examples where neural nets work well 4 / 20 STA 414/2104 Mar 9, 2010...
View
Full
Document
This document was uploaded on 08/12/2010.
 Spring '09

Click to edit the document details