to Introduction Boosting
Hojung Cho Topics for Bioinformatics Oct 10 2006
Boosting
Underlying principle
While building a highly accurate prediction rule is not an easy task, it is not hard to come up with very rough rules of thumb ("weak learners") that are only moderately accurate and to combine these into a highly accurate classifier.
Outline The boosting framework Choice of AdaBoost LogiBoost References
The Rules for Boosting
1) set all weights of training examples equal 2) train a weak learner on the weighted examples 3) see how well the weak learner performs on data and give it a weight based on how well it did 4) re-weight training examples and repeat 5) when done, predict by voting by majority
Weak learner : "rough and moderately inaccurate" predictor, but one that can predict better than chance (1/2) - > Boosting shows the strength of weak learnability
Two fundamental questions for designing the Boosting algorithm 1) How should each distribution or weighting (subset of examples) be chosen on each round? 2) place the most weight on the examples most often misclassified by the preceding weak rules forcing the weak learner to focus on the "hardest " examples
How should the weak learners be combined into a single rule? take a weighted majority vote of their predictions choice of : analytically or numerically
A Boosting approach
Given: ( x1, y1 ),...,( xn, yn ) where xi X , yi = {-1, +1} Y Initialize D1 (i) = 1/ n. For m = 1,..., M : (i) Train base learner using distribution Dm . (ii) Get base classifier (weak learner) h m : X (iii)Choose m (iv) Update: D (i) exp(- m yi hm ( xi )) Dm+1 (i ) = m Zm where Z mis a normalization factor (chosen so that Dm+1 will be a distribution). R R
1 1- m m = ln 2 m
Binary classification
:AdaBoost
if hm ( xi yi
D( ) m+1) (i ) =
Dm (i ) exp( m ), g exp(- m ), Zm
if hm ( xi ) = yi
Since exp( m ) > 1, it emphasizes the i-th example of it was misclassified.
Output the final classifier:
M H ( x) = sign m hm ( x) m =1
Simple example
Example Round 1 Round 2
Round 3
Final Hypothesis
Choice of
Schapire and Singer proved that the training error is bounded by From the theorem above, We can derive
Proof
AdaBoost
Boosting and additive logistic regression
(Friedman et al, 2000)
Boosting: an approximation to additive modeling on the logistic scale using maximum Bernoulli (binomial in multiclass case) likelihood as a criterion. Propose more direct approximations that exhibit nearly identical results to boosting (AdaBoost). Reduce computation.
The probability of y =+1 when the f(x) is the weighted average of the basic classifiers in AdaBoost is represented by p(x),
. Note than the close connection between the log loss (negative log likelihood)of a model above, and the function we attempt to minimize in AdaBoost,
For any distribution over pairs(x,y), both the expectations are...