Figure 813 data with two features and two classes

•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
• •
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
• •
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
• •
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
••
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
• •
•
•
•
•
•
•
•
•
•
•
• •
• •
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
FIGURE 8.13. Data with two features and two
classes (blue and orange), displaying a pure interaction.
The left panel shows the partition found by three splits
of a standard, greedy, tree-growing algorithm. The ver-
tical grey line near the left edge is the first split, and
the broken lines are the two subsequent splits. The al-
gorithm has no idea where to make a good initial split,
and makes a poor choice. The right panel shows the
near-optimal splits found by bumping the tree-growing
algorithm 20 times.

Bagging and Bumping Bagging and Random Forests Boosting Methods Boosting Trees
Bagging (Revisited)
The bagging estimate is defined as
ˆ
f
bag
( x ) =
1
B
B
X
b = 1
ˆ
f
* b
( x )
for regression and
ˆ
f
bag
( x ) = max
k
B
X
b = 1
I
{
ˆ
f * b ( x )= k }
for classification where k ∈ K are the possible classes.
c 2019 The Trustees of the Stevens Institute of Technology

Bagging and Bumping Bagging and Random Forests Boosting Methods Boosting Trees
Out-of-Bag Error
1. Predict response for the i
th
observation using the trees that
did not utilize this observation in their construction
2. Take the average of these responses (or majority vote) to
get a single OOB prediction for each of the n observations
3. Use these to calculate the OOB MSE or the OOB
Classification error
c 2019 The Trustees of the Stevens Institute of Technology

Bagging and Bumping Bagging and Random Forests Boosting Methods Boosting Trees
Random Forests
1. Create a bootstrapped sample of the observations
2. Build a tree, but at each split, only consider m ≤ p
predictors
3. Use the trees built in this way to calculate your prediction
like in bagging.