# This is a different presentation of the least squares

• 29

This preview shows page 24 - 26 out of 29 pages.

This is a different presentation of the least squares problem from the one given in Chap- ter 3, but it is totally equivalent since plugging the expression for w ( w = Ax b ) into the objective function gives the well-known formulation of the problem as one consisting of minimizing the function Ax b 2 over n . The least squares problem essentially assumes that the right-hand side is unknown and is subjected to noise and that the matrix A is known and fixed. However, in many applications the matrix A is not exactly known and is also subjected to noise. In these cases it is more logical to consider a different prob- lem, which is called the total least squares problem , in which one seeks to find a minimal norm perturbation to both the right-hand-side vector and the matrix so that the resulting perturbed system is consistent: (TLS) min E , w , x E 2 F + w 2 s.t. ( A + E ) x = b + w , E m × n , w m . Note that we use here the Frobenius norm as a matrix norm. Problem (TLS) is not a convex problem since the constraints are quadratic equality constraints. However, despite Downloaded 01/06/17 to 69.91.157.97. Redistribution subject to SIAM license or copyright; see
11.7. Total Least Squares 231 the nonconvexity of the problem we can use the KKT conditions in order to simplify it considerably and eventually even solve it. The trick is to fix x and solve the problem with respect to the variables E and w , giving rise to the problem ( P x ) min E , w E 2 F + w 2 s.t. ( A + E ) x = b + w . Problem ( P x ) is a linearly constrained convex problem and hence the KKT conditions are necessary and sufficient (Theorem 10.7). The Lagrangian of problem ( P x ) is given by L ( E , w , λ ) = E 2 F + w 2 + 2 λ T [( A + E ) x b w ] . By the KKT conditions, ( E , w ) is an optimal solution of ( P x ) if and only if there exists λ m such that 2 E + 2 λ x T = 0 ( E L = 0 ) , (11.39) 2 w 2 λ = 0 ( w L = 0 ) , (11.40) ( A + E ) x = b + w ( feasibility ) . (11.41) From (11.40) we have λ = w . Substituting this in (11.39) we obtain E = wx T . (11.42) Combining (11.42) with (11.41) we have ( A wx T ) x = b + w , so that w = Ax b x 2 + 1 , (11.43) and consequently, by plugging the above into (11.42) we have E = ( Ax b ) x T x 2 + 1 . (11.44) Finally, by substituting (11.43) and (11.44) into the objective function of problem ( P x ) we obtain that the value of problem ( P x ) is equal to Ax b 2 x 2 + 1 . Consequently, the TLS problem reduces to ( TLS ) min x n Ax b 2 x 2 + 1 . We have thus proven the following result. Theorem 11.22. x is an optimal solution of ( TLS ) if and only if ( x , E , w ) is an optimal solution of (TLS) where E = ( Ax b ) x T x 2 + 1 and w = Ax b x 2 + 1 . The new formulation ( TLS ) is much simpler than the original one. However, the objective function is still nonconvex, and the question that still remains is whether we can efficiently find an optimal solution of this simplified formulation. Using the special structure of the problem, we will show that the problem can actually be solved efficiently by using a homogenization argument. Indeed, problem ( TLS ) is equivalent to min x n , t Ax t b 2 x 2 + t 2 : t = 1 , Downloaded 01/06/17 to 69.91.157.97. Redistribution subject to SIAM license or copyright; see
• • • 