This is a different presentation of the least squares problem from the one given in Chap-
ter 3, but it is totally equivalent since plugging the expression for
w
(
w
=
Ax
−
b
) into
the objective function gives the well-known formulation of the problem as one consisting
of minimizing the function
Ax
−
b
2
over
n
. The least squares problem essentially
assumes that the right-hand side is unknown and is subjected to noise and that the matrix
A
is known and fixed. However, in many applications the matrix
A
is not exactly known
and is also subjected to noise. In these cases it is more logical to consider a different prob-
lem, which is called
the total least squares problem
, in which one seeks to find a minimal
norm perturbation to both the right-hand-side vector and the matrix so that the resulting
perturbed system is consistent:
(TLS)
min
E
,
w
,
x
E
2
F
+
w
2
s.t.
(
A
+
E
)
x
=
b
+
w
,
E
∈
m
×
n
,
w
∈
m
.
Note that we use here the Frobenius norm as a matrix norm. Problem (TLS) is not a
convex problem since the constraints are quadratic
equality
constraints. However, despite
Downloaded 01/06/17 to 69.91.157.97. Redistribution subject to SIAM license or copyright; see

11.7. Total Least Squares
231
the nonconvexity of the problem we can use the KKT conditions in order to simplify it
considerably and eventually even solve it. The trick is to fix
x
and solve the problem with
respect to the variables
E
and
w
, giving rise to the problem
(
P
x
)
min
E
,
w
E
2
F
+
w
2
s.t.
(
A
+
E
)
x
=
b
+
w
.
Problem
(
P
x
)
is a linearly constrained convex problem and hence the KKT conditions
are necessary and sufficient (Theorem 10.7). The Lagrangian of problem
(
P
x
)
is given by
L
(
E
,
w
,
λ
) =
E
2
F
+
w
2
+
2
λ
T
[(
A
+
E
)
x
−
b
−
w
]
.
By the KKT conditions, (
E
,
w
) is an optimal solution of
(
P
x
)
if and only if there exists
λ
∈
m
such that
2
E
+
2
λ
x
T
=
0
(
∇
E
L
=
0
)
,
(11.39)
2
w
−
2
λ
=
0
(
∇
w
L
=
0
)
,
(11.40)
(
A
+
E
)
x
=
b
+
w
(
feasibility
)
.
(11.41)
From (11.40) we have
λ
=
w
. Substituting this in (11.39) we obtain
E
=
−
wx
T
.
(11.42)
Combining (11.42) with (11.41) we have
(
A
−
wx
T
)
x
=
b
+
w
, so that
w
=
Ax
−
b
x
2
+
1
,
(11.43)
and consequently, by plugging the above into (11.42) we have
E
=
−
(
Ax
−
b
)
x
T
x
2
+
1
.
(11.44)
Finally, by substituting (11.43) and (11.44) into the objective function of problem
(
P
x
)
we
obtain that the value of problem
(
P
x
)
is equal to
Ax
−
b
2
x
2
+
1
. Consequently, the TLS problem
reduces to
(
TLS
)
min
x
∈
n
Ax
−
b
2
x
2
+
1
.
We have thus proven the following result.
Theorem 11.22.
x
is an optimal solution of
(
TLS
)
if and only if
(
x
,
E
,
w
)
is an optimal
solution of
(TLS)
where
E
=
−
(
Ax
−
b
)
x
T
x
2
+
1
and
w
=
Ax
−
b
x
2
+
1
.
The new formulation
(
TLS
)
is much simpler than the original one. However, the
objective function is still nonconvex, and the question that still remains is whether we
can efficiently find an optimal solution of this simplified formulation. Using the special
structure of the problem, we will show that the problem can actually be solved efficiently
by using a
homogenization
argument. Indeed, problem
(
TLS
)
is equivalent to
min
x
∈
n
,
t
∈
Ax
−
t
b
2
x
2
+
t
2
:
t
=
1
,
Downloaded 01/06/17 to 69.91.157.97. Redistribution subject to SIAM license or copyright; see