1. So far we have discussed solving Ax=b for square
n
n
matrices
A
. For more general
n
m
matrices, there
are a variety of scenarios.
a. When
m < n
, the problem is
underdetermined
since there is not enough information to determine a
unique solution for all the variables. Usually m<n implies that there are
infinite solutions
. However, in
some cases, there may be contradictory equations leading to the absence of any solutions.
b. When
m > n
, the problem is
overdetermined
although this in itself doesn’t tell us everything about
the nature of the solution. For example, if enough equations are linear combinations of each other,
there can still be a unique solution or infinite solutions.
i. We can use the rank of the matrix to enumerate the possibilities. Recall that the
rank
of a
matrix is the number of linearly independent columns that it has. Thus a
n
m
matrix has at
most a rank of n.
ii. If the rank <
n
some columns are linear combinations of others and we say that the matrix is
rank-deficient
and there may be an infinite number of solutions.
iii. On the other hand, if the rank =
n
, i.e. all the columns are linearly independent and we are
guaranteed either a unique solution or no solution. In the case of no solution, there is the notion
of the “closest fit” in a
least squares
sense. That is, the least squares solution finds the closest
possible solution.
2. When solving systems of equations, we want
Ax=b
.
We define the
residual
as
r=b-Ax
, and note that the
residual gives us some measure of the error.
Of course the goal is to attain
r=
0.
3. The
least squares method
finds the
best-fit
or best approximation to the solution in the sense that the least
squares solution
x
minimizes the
2
L
norm of the residual, i.e. it minimizes
2
r
.
4. An example, consider interpolating a given a set of m data points,
( , )
i
i
t y
, with a straight line
1
2
y
x
x t
.
Here, each data point leads to a new equation, for example with three data points (
m
=3) we obtain
1
1
1
2
2
2
3
3
1
1
1
t
y
x
t
y
x
t
y
.
a. Here, if one gave the exact same pair
)
,
(
i
i
y
t
three times, then there is really only one point and there
are an infinite number of lines that go through it.
b. If the three points all line on the same line, then that line is a unique solution to the problem.
c. If the three points do not all lie on a line, the problem is overdetermined and there is no solution, i.e.
no straight line that passes through the points. In this case, we can look for the best least squares
solution, or the line that passes as close as possible to the three points.
t