of Department Statistics ST02: Multivariate Data Analysis and Chemometrics
Bent Jrgensen and Yuri Goegebeur
Module 2: Matrix algebra
2.1 2.2 2.3 2.4 2.5 2.6 2.7 Matrices, vectors and scalars . . . . . . . . . . Special matrices . . . . . . . . . . . . . . . . . Matrix and vector operations . . . . . . . . . . Matrix multiplication . . . . . . . . . . . . . . . Norm and trace . . . . . . . . . . . . . . . . . . Orthogonal vectors, matrices and projection . . Eigenvalues, determinant and condition number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 . 2 . 3 . 5 . 8 . 8 . 10
2.1
Matrices, vectors and scalars
Ugarte: You despise me, dont you? Rick Blaine: If I gave you any thought I probably would. [Casablanca, 1942]
The following quote serves as a suitable introduction to this module: Linear algebra is the language of chemometrics. One cannot expect to truly understand most chemometric techniques without a basic understanding of linear algebra. This [module] reviews the basics of linear algebra and provides the reader with the foundations required for understanding most chemometrics literature. It is presented in a rather dense fashion. [...] The goal has been to condense into as few pages as possible the aspects of linear algebra used in most chemometrics methods. [Wise and Gallagher, 1998] Note that in this module, we use a mixture of the general notation introduced in Module 1, and a local notation, where letters are used freely to denote dierent vectors and matrices etc. A single number is usually called a scalar, and is represented by math italics, e.g. c. A matrix is a two dimensional array of numbers and is represented by bold upper case math italics e.g. X. For example X= 12 3 8 2.3 4 1
is a 23 matrix, having 2 rows and 3 columns. The dimensions of a matrix are normally presented with the number of rows rst and the number of columns second. We use square brackets to represent a matrix, in order to emphasize its rectangular shape. http://statmaster.sdu.dk/courses/ST02 February 7, 2006
2.2 Special matrices
2
Indices: The entries of a matrix are denoted by the same letter as the matrix, but in lower case math italics with two indices, e.g. X = {xij } for i = 1, . . . , 2 (rows) and j = 1, . . . , 3 (columns). For example, in the matrix X above, the (2,1) element is x21 = 2.3. A vector consists of a row or column of numbers and is represented by bold lower case italics e.g. x. Vectors can be considered as matrices with one dimension equal to 1. For example z = 3 14 9.4 0 is a 1 4 row vector (sometimes the entries are separated by commas), and 5.6 w = 2.8 1.9 a 3 1 column vector.
2.2
Special matrices
A square matrix is one that has the same number of rows and columns, e.g. 7 4 2 A = 11 3 4.1 3 4 12 is a 3 3 square matrix. The main diagonal of a square matrix consists of the numbers aii with equal row and column indices, and is usually represented as a vector. So 7 3 12 is the row vector representing the main diagonal of the matrix A from above. A square matrix is called diagonal when all the non-diagonal entries are zero, e.g. A= 1 0 0 4 .
We let diag{a1 , . . ., an } denote the diagonal matrix with main diagonal entries a1 , . . ., an . We let diagA denote the row vector containing the diagonal of A. An identity matrix (sometimes called a unit matrix) is a diagonal matrix whose main diagonal elements are equal to 1. It is denoted by I. For example I= 1 0 0 1
is an identity matrix. A subscript may be added to emphasize the dimension of the matrix, so I n denotes the n n identity matrix. http://statmaster.sdu.dk/courses/ST02 February 7, 2006
2.3 Matrix and vector operations
3
A zero matrix or vector is any matrix or vector whose entries are all zero. For example 0 0 0= 0 0 0 0
is the 3 2 zero matrix, which may also be denoted 032 to emphasize its dimensions. In particular, we use 0 to denote the origin of a vector space. vector whose entries are all 1, e.g. 1 1 1
A One matrix or vector is any matrix or 1 1= 1 1
Again its dimensions may be added as a subscript for emphasis, e.g. 132 .
2.3
Matrix and vector operations
Addition and subtraction: Two matrices can be added or subtracted componentwise if they have identical dimensions (are conformable for addition). Hence 9 7 0 7 9 0 8 4 + 11 3 = 19 7 2 4 5 6 3 10 and 9 7 0 7 9 14 8 4 11 3 = 3 1 2 4 5 6 7 2
Scalar multiplication: A matrix can be multiplied by a constant, which is done componentwise. For example, if X= then the scalar multiple 3X is 3X = 36 9 24 6.9 12 3 12 3 8 2.3 4 1
The transpose X of a matrix X is the matrix formed by letting columns become rows (or rows columns), and is denoted by the superscript . So, for example, the 4 3 matrix 3.1 0.2 6.1 4.8 5 13.8 X = 20 1 2.2 5.1 3.8 3.94 http://statmaster.sdu.dk/courses/ST02 February 7, 2006
2.3 Matrix and vector operations
4
becomes a 3 4 matrix after transposition, 3.1 4.8 20 5.1 X = 0.2 5 1 3.8 . 6.1 13.8 2.2 3.94 Some authors use the superscript instead of for transpose. A square matrix A is called symmetric if A = A , e.g. A= 1 2 2 4 .
Linear combinations: If a1 , . . . , ak are vectors of common dimension, and c1 , . . . , ck are constants, then the vector c1 a1 + + ck ak is called a linear combination of a1 , . . . , ak with coecients c1 , . . . , ck . The linear subspace spanned by a1 , . . . , ak is the set of all linear combinations of a1 , . . . , ak with arbitrary c1 , . . . , ck . Vectors a1 , . . . , ak are said to be linearly independent if c1 a1 + + ck ak = 0 implies c1 = = ck = 0. The rank of a matrix is the maximum number of linearly independent columns of the matrix. For example 1 2 = 2, rank 3 4 but rank because the vectors 1 2 3 6 = 1.
1 2 and are linearly dependent. The rank is also equal to 3 6 the maximum number of linearly independent rows of the matrix. Geometric interpretation: consider for example the three columns x1 , x2 , x3 of the above matrix X. They are 4-vectors. If they all point in the same direction (up to a sign), then the rank of X is 1. If this is not the case, but they all lie within a plane through 0, then X has rank 2. If this is not the case, then the rank is 3. The maximum possible rank for a matrix is the smallest of its column and row dimensions. In case of maximum rank the matrix is said to have full rank, or be nonsingular. A singular matrix is also called rank-decient, or is said to have a problem of multicollinearity. http://statmaster.sdu.dk/courses/ST02 February 7, 2006
2.4 Matrix multiplication
5
2.4
Matrix multiplication
If matrix A = {aij } has dimensions k m and matrix B = {bij } has dimensions m n (having the second dimension of A and the rst dimension of B in common), they are said to be comformable for multiplication. Then the matrix product C = {ci } is a matrix of dimensions k n with elements dened by
m
ci =
j=1
aij bj
and we write C = AB. Example 1 7 9 3 2 5 6 17 67 38 = 54 93 123 42 12 25 62 31
6 10 11 3 0 1 8 5
Example: Consider the Beer-Lambert law for m constituents and k wavelengths plus noise:
m
xj =
=1
y a j + ej ,
for wavelengths numbered j = 1, . . . , k. This equation may be written in matrix notation as follows: x = yA + e, where x = [x1 . . . xk ] is 1 k,
A = {a j } is m k,
y = [y1 . . . ym ] is 1 m,
e = [e1 . . . ek ] is 1 k.
In particular, the matrix product of two square matrices of common dimension is a square matrix of that dimension. Multiplication of matrices is not commutative, that is, generally AB = BA, even if the second product is allowable. Associative rule for matrix product:. (AB)C = A(BC) = ABC where the last form may be used due to the equality of the two rst. Distributive rule of matrix multiplication and addition: A(B + C) = AB + AC http://statmaster.sdu.dk/courses/ST02 7, February 2006
2.4 Matrix multiplication
6
Let a and b be column vectors of common dimension. The inner product (dot product) of a and b is dened by ab=a b = b a The result is a scalar. Note that each element in the matrix AB is the result of an 9 be inner product between a row of A and a column of B. For example, let a = 3 11 the second row of the rst matrix above (represented as a column) and b = be 8 the third column of the second matrix above. Then the (2,3) entry of the product is a b = 9 11 + 3 8 = 123 An n 1 one vector 1 is also known as a summing vector, because 1 x is the sum of the elements of x. Let a and b be column vectors of the dimensions m and n, respectively. The outer product of a and b is the m n matrix dened by ab = {ai bj } It may be thought of as the matrix product of a column vector and a row vector, which are always conformable for multiplication, because they have the dimension 1 in common. Inverse matrix: A non-singular square matrix A has a unique inverse matrix A1 (square with the same dimension as A), which satises AA1 = A1 A = I. A is then said to be invertible. See Examples for various illustrations of how the inverse of a matrix is calculated. Inverse of a product: For A and B invertible square matrices of the same size: (AB)1 = B 1 A1 Transpose of product: For A and B conformable for multiplication (AB) = B A Matrix inverse and transpose commute: A
1
= A1
= A
where the last notation may be used for convenience to represent either of the two forms. http://statmaster.sdu.dk/courses/ST02 February 7, 2006
2.4 Matrix multiplication
7
Linear equations: For an invertible square matrix A and given vector b, the equation Ax = b with x unknown, has unique solution x = A1 b. For example, for a 3 3 matrix, this equation is equivalent to the following system of linear equations in x1 , x2 , x3 : a11 x1 + a12 x2 + a13 x3 = b1 a21 x1 + a22 x2 + a23 x3 = b2 a31 x1 + a32 x2 + a33 x3 = b3 If A is square, but not invertible, then any matrix A that satises AA A = A is called a generalized inverse of A. There are in general many such matrices A . The Moore-Penrose inverse A (unique) is further assumed to satisfy A AA = A AA A A = AA = A A
Let A be an n k matrix of rank k n. Then the k n matrix A+ = A A
1
A
is called the left pseudo-inverse of A. Note that the rank condition on A guarantees that the k k matrix A A is invertible. A+ satises the condition A+ A = I k Similarly, let A be an n k matrix of rank n k. Then the k n matrix A =A AA
1
is called the right pseudo-inverse of A. Then A satises the condition AA = I n
http://statmaster.sdu.dk/courses/ST02
February 7, 2006
2.5 Norm and trace
8
2.5
Norm and trace
Vector norm (length): Consider the n-vector x1 . x = . . . xn The norm of x is dened by x = = x x x2 + + x2 . n 1
The norm of x is the length of the line going from 0 to the point in n-space indicated by x. A vector of norm 1 is called a unit vector. The unit vector in the direction of x is dened by standardization x . e= x The trace tr of a square matrix is the sum of its diagonal elements, e.g. tr 1 2 3 4 = 1 + 4 = 5.
Trace operator satises a kind of commutative property: tr(AB) = tr(BA) for matrices A and B such that AB and BA both exist (i.e. A and B dimension). have common
2.6
Orthogonal vectors, matrices and projection
The two n-vectors x and y are said to be orthogonal if their inner product is zero, x y = 0. For example, the matrix has orthogonal columns. A set of vectors is said to be an orthogonal set if the vectors are pairwise orthogonal. An orthogonal set consisting of unit vectors, is called an orthonormal set. http://statmaster.sdu.dk/courses/ST02 February 7, 2006 1 1 0 5 1 1
2.6 Orthogonal vectors, matrices and projection
9
If
X Y =0
the columns of X are all orthogonal to those of Y . A matrix A is called orthogonal if the columns of A are orthonormal, that is, A A=I It would have been more logical to call A an orthonormal matrix, but the above terminology is standard in linear algebra. For example, the matrix 0.7071 0.1925 0 0.9623 0.7071 0.1925 is orthogonal. If a square matrix A is orthogonal, then so is A , and A1 = A . The orthogonal projection of the n-vector y onto the linear subspace spanned by the columns of the n k matrix X is Hy, where H=X X X which requires that rank X = k. Note the following useful properties of rank: rank X = rank X X = rank XX .
1
X
Matrix H is a projection matrix, also known in statistics as a hat matrix. It is symmetric and idempotent, that is, it satises H 2 = HH = H. Its trace is tr(H) = k. In the simplest case, the projection of y on x is e y e, where e =
x x
is the unit vector in the direction of x.
http://statmaster.sdu.dk/courses/ST02
February 7, 2006
2.7 Eigenvalues, determin...