This preview shows pages 1–7. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: EE 649 Pattern Recognition Error Estimation Ulisses BragaNeto ECE Department Texas A&M University EE 649 Pattern Recognition – p.1/62 Main Ideas How does one estimate the classification error (the generalization error) of a given designed classifier? We do not know the true distribution of the data, but are given only training data, or (rarely) testing data. Error estimation is involved in classifier design itself (implicitly) and in feature selection. Error estimation has to do with the epistemological question: is scientific knowledge possible? EE 649 Pattern Recognition – p.2/62 Measures of Error in Classification Given any classifier ψ : R d → { , 1 } , its error is: ǫ [ ψ ] = P ( Y negationslash = ψ ( X )) = E [  Y − ψ ( X )  ] Bayes error (minimum classification error): ǫ ∗ = ǫ [ ψ ∗ ] = E [  Y − ψ ∗ ( X )  ] Designed classifier error for classification rule Ψ n : ǫ n = ǫ [ ψ n ] = E [  Y − ψ n ( X )  ] = E bracketleftbig  Y − Ψ n ( X ; S n )  vextendsingle vextendsingle S n bracketrightbig Expected classification error: μ n = E [ ǫ n ] = E bracketleftbig E bracketleftbig  Y − Ψ n ( X ; S n )  vextendsingle vextendsingle S n bracketrightbigbracketrightbig EE 649 Pattern Recognition – p.3/62 Some Observations The Bayes error ǫ ∗ provides a universal bound on classification performance, but it is usually very difficult to estimate with any accuracy. The designed classifier error ǫ n is the most important one for practical purposes; it provides the generalization error of the classifier at hand. The expected classification error E [ ǫ n ] is of limited practical use; it is used to compare the performance of classification rules or to answer theoretical questions about a classification rule (such as consistency). However, the expected error can become important in practice if ǫ n ≈ E [ ǫ n ] , that is, if the error variance Var ( ǫ n ) is very small. EE 649 Pattern Recognition – p.4/62 Error Estimators An error estimator is a mapping ˆ ǫ n : (Ψ n ,S n ,ξ ) mapsto→ [0 , 1] where ξ specifies internal random factors unrelated to the classification rule or the training data. The error estimator ˆ ǫ n is a random variable, through S n and ξ . The error estimate is the value of ˆ ǫ n given realizations of S n and ξ and is thus a real number. Unless otherwise stated, ˆ ǫ n is meant to be an approximation to the designed classifier error ǫ n = E [  Y − ψ n ( X )  ] . EE 649 Pattern Recognition – p.5/62 Randomization An error estimator can be: Nonrandomized : Given the training data S n , the estimator ˆ ǫ n is fixed (there are no internal random factors ξ ). Randomized: Given the training data S n , the estimator ˆ ǫ n is not a fixed quantity. It is still a random variable through ξ ....
View
Full
Document
This note was uploaded on 02/08/2010 for the course ECEN 649 taught by Professor Staff during the Spring '08 term at Texas A&M.
 Spring '08
 Staff

Click to edit the document details