This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Stability of Tikhonov Regularization 9.520 Class 15, 05 April, 2006 Sasha Rakhlin Plan Review of Stability Bounds • Stability of Tikhonov Regularization Algorithms • Uniform Stability Review notation : S = { z 1 , ..., z n } ; S i,z = { z 1 , ..., z i − 1 , z, z i +1 , ..., z n } An algorithm A has uniform stability β if ∀ ( S, z ) ∈ Z n +1 , ∀ i, sup V ( f S , u ) − V ( f S i,z , u ) ≤ β.   u ∈Z Last class : Uniform stability of β = O 1 implies good n generalization bounds. This class : Tikhonov Regularization has uniform stability = O 1 of β . n Reminder : The Tikhonov Regularization algorithm: n 1 f S = arg min V ( f ( x i ) , y i ) + λ f 2 K n i =1 f ∈H Generalization Bounds Via Uniform Stability If β = k for some k , we have the following bounds from n the last lecture: k n 2 P I [ f S ] − I S [ f S ] + ≤ 2 exp .   ≥ n − 2( k + M ) 2 Equivalently, with probability 1 − δ , k 2 ln(2 /δ ) I [ f S ] ≤ I S [ f S ] + + (2 k + M ) . n n Lipschitz Loss Functions, I We say that a loss function (over a possibly bounded do main X ) is Lipschitz with Lipschitz constant L if ∀ y 1 ,y 2 ,y , V ( y 1 ,y ) − V ( y 2 ,y ) . ≤ L y 1 − y 2 ∈ Y     Put differently, if we have two functions f 1 and f 2 , under an LLipschitz loss function, sup V ( f 1 ( x ) ,y ) − V ( f 2 ( x ) ,y ) . ≤ L f 1 − f 2 ( x ,y )     ∞ Yet another way to write it: V ( f 1 , · ) − V ( f 2 , · ) f 1 ( · ) − f 2 ( · )   ∞ ≤ L   ∞ Lipschitz Loss Functions, II If a loss function is LLipschitz, then closeness of two func tions (in L ∞ norm) implies that they are close in loss....
View
Full
Document
This note was uploaded on 11/11/2011 for the course BIO 9.07 taught by Professor Ruthrosenholtz during the Spring '04 term at MIT.
 Spring '04
 RuthRosenholtz

Click to edit the document details