This preview shows pages 1–4. Sign up to view the full content.
4/15/2009
1
COMPUTER SCIENCE 51
Spring 2009
cs51.seas.harvard.edu
Prof. Greg Morrisett
Prof. Ramin Zabih
Bad line representation?
10
5
0
5
10
10
5
0
5
10
x
2*x
3*x
4*x
5*x
6*x
7*x
8*x
LS fitting and residuals
0
1
2
3
4
5
6
0
2
4
6
8
10
12
Time
Mileage
( )
2
Err(
, )
i
i
i
m b
y
mx
b
=

+
∑
Nice example graph
0
100
200
300
400
500
600
14.6
14.8
15
15.2
15.4
15.6
15.8
16
Code for sample data
(
define
data
(list '(15.85 235) '(15.69 280) '(15.38 360)
'(15.3 442) '(14.84 528)))
(
define
(dataerr m b data)
(
let*
([predict (
lambda
(x) (+ (* m x) b))]
[residual (
lambda
(pt) ( (predict (car pt)) (cadr pt)))]
[sse (map square (map residual data))])
(foldr + 0 sse)))
Greedy algorithm
•
To find the LS squares line:
•
Start with some initial (
m
,
b
)
•
Make a small change to (
m
,
b
)
–
See if Err(
m
,
b
) goes down
–
If so, make that your new (
m
,
b
)
•
Done when small changes don’t
reduce
Err(
m
,
b
)
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document 4/15/2009
2
Hillclimbing
•
Sometimes called hillclimbing
–
In our case, we want to find the bottom
of the hill not the top
•
Can do a series of 1D optimizations
–
Find best slope and fix it
–
Find best intercept and fix it
–
Repeat until done
•
There are many alternatives
Why least squares?
•
There are lots of sensible ways to
compute goodness of fit
•
Why do we always use least squares?
•
This is a deep topic
–
But we will point out two things that are
special about least squares
Fix m=2, change b
1
0.5
0
0.5
1
2
4
6
8
10
12
14
16
b
sum of squared erors
Fix b=0.5, change m
1.8
1.9
2
2.1
2.2
2.5
3
3.5
4
4.5
5
5.5
m
Some error functions are easy
•
You will notice that the squared error
has a
single
minimum value
–
The plots I just showed you demonstrate
this is true in 1D
•
But it’s true in 2D also
0
50
100
0
20
40
60
80
10
0
50
100
150
200
250
300
350
400
450
500
Consequences
•
For error functions like this, if we get an
answer where a small change doesn’t
improve it, we know we are right!
–
Some issue with what “small” means
•
Our LShillclimbingmethod will always
converge to the right answer
–
By slowly rolling downhill
–
Hard to analyze how long it will take
4/15/2009
3
Easy and hard error measures
•
What makes this error function easy?
–
It has a single minimum, and we can get
to it from anywhere by rolling downhill
–
Not all error functions have this property
•
But, least squares does
–
Need to
think
before coding!
Why is an error function hard?
•
An error function where we can get
stuck if we roll downhill is a hard one
–
Where we get stuck depends on where
we start (i.e., initial guess/conditions)
–
An error function is easy if the area
“above it” has a certain shape
•
No nooks and crannies
–
Easy: convex
–
Hard: nonconvex
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
This is the end of the preview. Sign up
to
access the rest of the document.
This note was uploaded on 07/26/2009 for the course COMPUTERSC CS51 taught by Professor Gregmorrisett during the Spring '09 term at Harvard.
 Spring '09
 GREGMORRISETT

Click to edit the document details