ECE153
Prof. UCSD Young-Han Kim
Handout #32
Thursday, May 26, 2011
Solutions to Homework Set #6
(Prepared by TA Lele Wang)
1. Covariance matrices. Which of the following matrices can be a covariance matrix? Justify
your answer either by constructing a random vector X, as a function of the i.i.d zero mean unit
variance random variables Z1 , Z2 , and Z3 , with the given covariance matrix, or by establishing
a contradiction.
112
111
12
21
(a)
(d) 1 2 3
(b)
(c) 1 2 2
02
12
233
123
Solution:
(a) This cannot be a covariance matrix because it is not symmetric.
(b) This is a covariance matrix for X1 = Z1 + Z2 and X2 = Z1 + Z3 .
(c) This is a covariance matrix for X1 = Z1 , X2 = Z1 + Z2 , and X3 = Z1 + Z2 + Z3 .
2
(d) This cannot be a covariance matrix. Suppose it is, then 23 = 9 > 22 33 = 6, which
contradicts the Schwartz inequality. You can also verify this by showing that the matrix
is not positive semidenite. For example, the determinant is 2. Also one of the
eigenvalues is negative (1 = 0.8056).
Alternatively, we can directly show that this matrix does not satisfy the denition of
positive semideniteness by
2
112
2 0 1 1 2 3 0 = 1 < 0.
2 3 3 1
2. Gaussian random vector. Given a Gaussian random vector X N (, ), where = (1 5 2)T
and
110
= 1 4 0 .
009
(a) Find the pdfs of
i.
ii.
iii.
iv.
v.
X1 ,
X2 + X3 ,
2X1 + X2 + X3 ,
X3 given (X1 , X2 ), and
(X2 , X3 ) given X1 .
(b) What is P{2X1 + X2 X3 < 0}? Express your answer using the Q function.
1
(c) Find the joint pdf on Y = AX, where
A=
211
.
1 1 1
Solution:
(a)
i. The marginal pdfs of a jointly Gaussian pdf are Gaussian. Therefore X1 N (1, 1).
ii. Since X2 and X3 are independent (23 = 0), the variance of the sum is the sum
of the variances. Also the sum of two jointly Gaussian random variables is also
Gaussian. Therefore X2 + X3 N (7, 13).
iii. Since 2X1 + X2 + X3 is a linear transformation of a Gaussian random vector,
X1
X2 ,
2X1 + X2 + X3 = 2 1 1
X3
it is a Gaussian random vector with mean and variance
110 2
1
= 2 1 1 5 = 9 and 2 = 2 1 1 1 4 0 1 = 21 .
009 1
2
Thus 2X1 + X2 + X3 N (9, 21).
iv. Since 13 = 0, X3 and X1 are uncorrelated and hence independent since they are
jointly Gaussian; similarly, since 23 = 0, X3 and X2 are independent. Therefore
the conditional pdf of X3 given (X1 , X2 ) is the same as the pdf of X3 , which is
N (2, 9).
v. We use the general formula for the conditional Gaussian pdf:
X2 | {X1 = x1 } N 21 1 (x 1 ) + 2 , 22 21 1 12
11
11
In the case of (X2 , X3 ) | X1 ,
11 = 1 ,
21 =
1
,
0
22 =
40
.
09
Therefore the mean and variance of (X2 , X3 ) given X1 = x1 are
(X2 ,X3 )|X1 =
1
0
(X2 ,X3 )|X1 =
40
1
09
0
1
1
x1 1 +
x +4
5
,
=1
2
2
1 0=
40
10
30
=
.
09
00
09
Thus X2 and X3 are conditionally independent given X1 . The conditional densities
are X2 | {X1 = x1 } N (x1 + 4, 3) and X3 | {X1 = x} N (2, 9).
2
(b) Let Y = 2X1 + X2 X3 . Similarly as part (a)iii., 2X1 + X2 X3 is a linear transformation
of a Gaussian random vector,
X1
2X1 + X2 X3 = 2 1 1 X2 ,
X3
it is a Gaussian random vector with mean and variance
1
= 2 1 1 5 = 5 and 2 = 2 1 1
2
2
110
1 4 0 1 = 21 .
0 0 9 1
Thus 2X1 + X2 X3 N (5, 21), i.e., Y N (5, 21). Thus
(Y 5)
(0 5)
<
21
21
P{Y < 0} = P
=Q
5
21
.
(c) In general, AX N (AX , AX AT ). For this problem,
1
2 1 1
9
5=
Y = AX =
,
1 1 1
2
2
110 2 1
21 6
2 1 1
1 4 0 1 1 =
.
Y = AX AT =
6 12
1 1 1
009 1 1
Thus Y N
9
21 6
,
2
6 12
.
3. Gaussian Markov chain. Let X, Y, and Z be jointly Gaussian random variables with zero
mean and unit variance, i.e., E(X ) = E(Y ) = E(Z ) = 0 and E(X 2 ) = E(Y 2 ) = E(Z 2 ) = 1. Let
X,Y denote the correlation coecient between X and Y , and let Y,Z denote the correlation
coecient between Y and Z . Suppose that X and Z are conditionally independent given Y .
(a) Find X,Z in terms of X,Y and Y,Z .
(b) Find the MMSE estimate of Z given (X, Y ) and the corresponding MSE.
Solution:
(a) From the denition of X,Z , we have
X,Z =
Cov(X, Z )
,
X Z
where,
Cov(X, Z ) = E(XZ ) E(X )E(Z ) = E(XZ ) 0 = E(XZ ),
X = E(X 2 ) E(X )2 = 1 0 = 1,
Y = E(Y 2 ) E(Y )2 = 1 0 = 1.
3
Thus, X,Z = E(XZ ).
Moreover, since X and Z are conditionally independent given Y ,
E(XZ ) = E(E(XZ |Y )) = E[E(X |Y )E(Z |Y )].
Now E(X |Y ) can be easily calculated from the bivariate Gaussian conditional density
E( X | Y ) = E( X ) +
X,Y X
(Y E(Y )) = X,Y Y.
Y
Similarly, we have
E(Z |Y ) = Y,Z Y.
Therefore, combining the above,
X,Z = E(XZ )
= E[E(X |Y )E(Z |Y )]
= E(X,Y Y,Z Y 2 )
= X,Y Y,Z E(Y 2 )
= X,Y Y,Z .
(b) X , Y and Z are jointly Gaussian random variables. Thus, the minimum MSE estimate
of Z given (X, Y ) is linear.
(X,Y )T =
(X,Y )T Z =
1
X,Y
X,Y
,
1
E(XZ )
= X,Z ,
E( Y Z )
Y,Z
Z (X,Y )T = X,Z
Y,Z .
Therefore,
X
Z = Z (X,Y )T 1 )T
(X,Y
Y
= X,Z
Y,Z
= X,Z
Y,Z
=
1
X,Y
X,Y
1
1
1
1
X,Y
1 2
X,Y
X
Y
X,Y
1
1
0 2 Y,Z + Y,Z
X,Y
1 2
X,Y
X
,
Y
where the last equality follows from the result of (a). Thus,
Z = 0 Y,Z
4
X
= Y,Z Y.
Y
X
Y
The corresponding MSE is
MSE = 2 Z (X,Y )T 1 )T (X,Y )T Z
Z
(X,Y
= 1 X,Z
Y,Z
= 1 X,Z
1
Y,Z
X,Y
1
X,Y
1
1
1
X,Y
1 2
X,Y
X,Z
Y,Z
X,Y
1
X,Z
Y,Z
X,Z
Y,Z
= 1 0 Y,Z
= 1 2 .
Y,Z
4. Prediction of an autoregressive process. Let X be a random vector with zero mean and
covariance matrix
1
2 n 1
1
2 1
X =
.
..
.
.
.
n 1
1
for || < 1. X1 , X2 , . . . , Xn1 are observed, nd the best linear MSE estimate (predictor) of
Xn . Compute its MSE.
Solution: We have
1
.
..
.
.
X = .
n 2
n 1
By dening Y = X1
Xn 1
T
n 2 n 1
.
.
.
.
.
. .
1
1
, we have
Y
1
.
..
.
= .
.
n 2
YX = n1
X Y =
n 1
2
x = 1.
5
n 2
. ,
.
.
1
T
,
,
Therefore,
Xn = X Y 1 Y
Y
= n 1
= hT Y
= 0
1
.
..
.
.
.
n 2
1
n 2
.Y
.
.
1
(where hT = X Y 1 )
Y
0 Y
(since hT Y = X Y )
= Xn1 ;
and
MSE = x 2 X Y 1 YX
Y
= 1 hT YX
= 1 0
= 1 2 .
n 1
.
0 .
.
5. Noise cancellation. A classical problem in statistical signal processing involves estimating a
weak signal (e.g., the heart beat of a fetus) in the presence of a strong interference (the heart
beat of its mother) by making two observations; one with the weak signal present and one
without (by placing one microphone on the mothers belly and another close to her heart).
The observations can then be combined to estimate the weak signal by cancelling out the
interference. The following is a simple version of this application.
Let the weak signal X be a random variable with mean and variance P , and the observations
be Y1 = X + Z1 (Z1 being the strong interference), and Y2 = Z1 + Z2 (Z2 is a measurement
noise), where Z1 and Z2 are zero mean with variances N1 and N2 , respectively. Assume that
X , Z1 and Z2 are uncorrelated. Find the best linear MSE estimate of X given Y1 and Y2 and
its MSE. Interprete the results.
Solution: This is a vector linear MSE problem. Since Z1 and Z2 are zero mean, X = Y1 =
and Y2 = 0. We rst normalize the random variables by subtracting o their means to get
X = X , and
Y
Y= 1
.
Y2
Now using the orthogonality principle we can nd the best linear MSE estimate X of X . To
do so we rst nd
Y =
P + N1
N1
N1
N1 + N2
and
6
YX =
P
.
0
Thus,
X = T X 1 Y
Y
Y
1
N1 + N2
N1
Y
N1
P + N1
P (N1 + N2 ) + N1 N2
P
(N1 + N2 ) N1 Y .
=
P (N1 + N2 ) + N1 N2
=P
0
The best linear MSE estimate is X = X + . Thus,
P
((N1 + N2 )(Y1 ) N1 Y2 ) +
P (N1 + N2 ) + N1 N2
1
=
(P ((N1 + N2 )Y1 N1 Y2 )) + N1 N2 ) .
P (N1 + N2 ) + N1 N2
X=
The MSE can be calculated by
2
MSE = X T X 1 YX
Y
Y
=P
P
(N1 + N2 ) N1
P (N1 + N2 ) + N1 N2
P
0
P 2 (N1 + N2 )
P (N1 + N2 ) + N1 N2
P N1 N2
=
.
P (N1 + N2 ) + N1 N2
=P
The equation for the MSE makes perfect sense. First, note that if N1 and N2 are held constant
N
but P goes to innity, the MSE tends to N11 N22 . Next, note that if both N1 and N2 go to
+N
2
innity, the MSE goes to X , i.e., the estimate becomes worthless. Finally, note that if either
N1 or N2 goes to 0, the MSE also goes to 0. This is because the estimator will then use the
measurement with zero noise variance (that is, the one with no noise) and ignore the other
measurement.
6. Iocane or Sennari: Return of the chemistry professor. An absent-minded chemistry professor
forgets to label two identically looking bottles. One contains a chemical named Iocane and
the other contains a chemical named Sennari. It is well known that the radioactivity level
of Iocane has the Unif[0, 1] distribution, while the radioactivity level of Sennari has the
Exp(1) distribution.
In the previous homework, we found the optimal rule to decide which bottle is which, by
measuring the radioactivity level of one of the bottles. The chemistry professor got smarter
this time; she now measures both bottles.
(a) Let X be the radioactivity level measured from one bottle, and let Y be the radioactivity
level measured from the bottle. other What is the optimal decision rule (based on the
measurement (X, Y )) that maximizes the chance of correctly identifying the contents?
Assume that the radioactivity level of one chemical is independent of the level of the
other bottle (conditioned on which bottle contains which).
(b) What is the associated probability of error?
7
Solution: This is a vector linear MSE problem. Since Z1 and Z2 are zero mean, X = Y1 =
and Y2 = 0. We rst normalize the random variables by subtracting o their means to get
X = X , and
Y
Y= 1
.
Y2
Now using the orthogonality principle we can nd the best linear MSE estimate X of X . To
do so we rst nd
Y =
P + N1
N1
N1
N1 + N2
and
YX =
P
.
0
Thus,
X = T X 1 Y
Y
Y
1
N1 + N2
N1
Y
N1
P + N1
P (N1 + N2 ) + N1 N2
P
(N1 + N2 ) N1 Y .
=
P (N1 + N2 ) + N1 N2
=P
0
The best linear MSE estimate is X = X + . Thus,
P
((N1 + N2 )(Y1 ) N1 Y2 ) +
P (N1 + N2 ) + N1 N2
1
=
(P ((N1 + N2 )Y1 N1 Y2 )) + N1 N2 ) .
P (N1 + N2 ) + N1 N2
X=
The MSE can be calculated by
2
MSE = X T X 1 YX
Y
Y
=P
P
(N1 + N2 ) N1
P (N1 + N2 ) + N1 N2
P
0
P 2 (N1 + N2 )
P (N1 + N2 ) + N1 N2
P N1 N2
=
.
P (N1 + N2 ) + N1 N2
=P
The equation for the MSE makes perfect sense. First, note that if N1 and N2 are held constant
N
but P goes to innity, the MSE tends to N11 N22 . Next, note that if both N1 and N2 go to
+N
2
innity, the MSE goes to X , i.e., the estimate becomes worthless. Finally, note that if either
N1 or N2 goes to 0, the MSE also goes to 0. This is because the estimator will then use the
measurement with zero noise variance (that is, the one with no noise) and ignore the other
measurement.
8
7. Iocane or Sennari: Return of the chemistry professor. An absent-minded chemistry professor
forgets to label two identically looking bottles. One contains a chemical named Iocane and
the other contains a chemical named Sennari. It is well known that the radioactivity level
of Iocane has the Unif[0, 1] distribution, while the radioactivity level of Sennari has the
Exp(1) distribution.
In the previous homework, we found the optimal rule to decide which bottle is which, by
measuring the radioactivity level of one of the bottles. The chemistry professor got smarter
this time; she now measures both bottles.
(a) Let X be the radioactivity level measured from one bottle, and let Y be the radioactivity
level measured from the other bottle. What is the optimal decision rule (based on the
measurement (X, Y )) that maximizes the chance of correctly identifying the contents?
Assume that the radioactivity level of one chemical is independent of the level of the
other bottle (conditioned on which bottle contains which).
(b) What is the associated probability of error?
Solution: Let = 0 denote the case in which the rst bottle (measurement X ) is Iocane
and the second bottle (measurement Y ) is Sennari. Let = 1 denote the other case.
(a) The optimal MAP rule is equivalent to the ML rule
fX,Y | (x, y |0) > fX,Y | (x, y |1),
otherwise.
0,
1,
D(x, y ) =
Since fX,Y | (x, y |0) = 10x1 ey and fX,Y | (x, y |1) = ex 10y1 ,
D(x, y ) =
(x < 1, y > 1) or (0 y < x 1),
otherwise.
0,
1,
(b) The probability of error is given by
1
1
P( = D(X, Y )) = P(0 X Y 1| = 0) + P(0 Y < X 1| = 1)
2
2
= P(0 X Y 1| = 0)
1
y
=
0
ey dxdy
0
= 1 2e1 ,
1
which is less than the error probability 2 (1 e1 ) from the single measurement.
9
Solutions to Additional Exercises
1. Markov chain. Suppose X1 and X3 are independent given X2 . Show that
f (x1 , x2 , x3 ) = f (x1 )f (x2 | x1 )f (x3 | x2 ) = f (x3 )f (x2 | x 3 )f (x1 | x2 ).
In other words, if X1 X2 X3 forms a Markov Chain, then so does X3 X2 X1 .
Solution: By denition of conditional independence,
f (x1 , x3 | x2 ) = f (x1 | x2 )f (x3 | x2 ) .
Therefore, using the denition of conditional density,
f (x3 | x1 , x2 ) =
f (x1 , x2 , x3 )
f (x1 , x3 | x2 )f (x2 )
f (x1 | x2 )f (x3 | x2 )
=
=
= f (x3 | x2 ) .
f (x1 , x2 )
f (x1 | x 2 )f (x2 )
f (x1 | x2 )
We are given that X1 and X3 are independent given X2 . Then
f (x1 , x2 , x3 ) = f (x1 )f (x2 | x1 )f (x3 | x1 , x2 ) = f (x1 )f (x2 | x1 )f (x3 | x2 ) ,
In this case X1 X2 X3 is said to form a Markov chain. Similarly,
f (x1 , x2 , x3 ) = f (x3 )f (x2 | x3 )f (x1 | x2 , x3 ) = f (x3 )f (x2 | x3 )f (x1 | x2 ),
This shows that if X1 X2 X3 is a Markov chain, then X3 X2 X1 is also a Markov
chain.
2. Proof of Property 4. In Lecture Notes #6 it was stated that conditionals of a Gaussian
random vector are Gaussian. In this problem you will prove that fact.
If
Y
2
is a zero-mean GRV then X |{Y = y} N X Y 1 y, X X Y 1 YX .
Y
Y
X
Justify each of the following steps of the proof.
(a) Let X be the best MSE linear estimate of X given Y. Then X and X X are individually
zero-mean Gaussians. Find their variances.
(b) X and X X are independent.
(c) Now write X = X + (X X ). If Y = y then X = X Y 1 y + (X X ).
Y
(d) Now complete the proof.
Remark: This proof can be extended to vector X.
Solution:
(a) Let X be the best MSE linear estimate of X given Y. In the MSE vector case section of
Lecture Notes #6 it was shown that X and X X are individually zero-mean Gaussian
1
2
random variables with variances X Y Y YX and X X Y 1 YX , respectively.
Y
10
(b) The random variables X and X X are jointly Gaussian since they are obtained by
a linear transformation of the GRV [ Y X ]T . By orthogonality, X and X X are
and Y are
uncorrelated, so they are also independent. By the same reasoning, X X
independent.
(c) Now write X = X + (X X ). Then given Y = y
X = X Y 1 y + (X X ) ,
Y
since X X is independent of Y.
2
(d) Thus X | {Y = y} is Gaussian with mean X Y 1 y and variance X X Y 1 YX .
Y
Y
3. Additive nonwhite Gaussian noise channel. Let Yi = X + Zi for i = 1, 2, . . . , n be n observations of a signal X N(0, P ). The additive noise random variables Z1 , Z2 , . . . , Zn are
zero mean jointly Gaussian random variables that are independent of X and have correlation
E(Zi Zj ) = N 2|ij | for 1 i, j n.
(a) Find the best MSE estimate of X given Y1 , Y2 , . . . , Yn .
(b) Find the MSE of the estimate in part (a).
Hint: the coecients for the best estimate are of the form hT = [ a b b b b a ].
Solution:
(a) The best estimate of X is of the form
n
X=
hi Yi .
i=1
We apply the orthogonality condition E(XYj ) = E(XYj ) for 1 j n:
n
hi E(Yi Yj )
P=
i=1
n
=
hi E((X + Zi )(X + Zj ))
i=1
n
=
i=1
hi (P + N 2|ij | ) .
There are n equations with n unknowns:
P +N
P + N/2
P
P P + N/2
P +N
.
.
.
.
.
.=
.
.
.
n2 P + N/2n3
P P + N/2
P
P + N/2n1 P + N/2n2
11
..
.
P + N/2n2 P + N/2n1
h1
P + N/2n3 P + N/2n2 h2
.
.
.
.
.
. .
.
.
.
P +N
P + N/2 hn1
hn
P + N/2
P +N
By the hint, there are only 2 degrees of freedom given, a and b. Solving this equation
using the rst 2 rows of the matrix, we obtain
h1
2
h2
1
P
.
.
. =
. 3N + (n + 2)P . .
.
hn1
1
hn
2
(b) The minimum mean square error is
MSE = E(X X )X
n
=P P
hi Yi
i=1
(n + 2)P
3N + (n + 2)P
3P N
=
.
3N + (n + 2)P
1
=P
4. Sucient statistic. The bias of a coin is a random variable P U[0, 1]. Let Z1 , Z2 , . . . , Z10
be the outcomes of 10 coin ips. Thus Zi B(P ) and {Z1 , Z2 , . . . , Z10 } are conditionally
independent given P . If X is the total number of heads, then X |{P = p} Binom(10, p).
Assuming that the total number of heads is 9, show that
fP |Z1 ,Z2 ,...,Z10 (p|z1 , z2 , . . . , z10 ) = fP |X (p|9)
is independent of the order of the outcomes.
Solution: The tosses of the coin are conditionally independent given the bias, that is,
pZ1 ,...,Z10 |P (z1 , . . . , z10 | p) = pZ1 |P (z1 |p)pZ2 |P (z2 |p) pZ10 |P (z10 |p) .
Suppose that the order of the outcomes is nine heads followed by one tail. Then
pZ1 ,...,Z10 |P (H, H, . . . , H, T | p) = p9 (1 p) .
We use Bayes rule to nd the conditional pdf of P .
fP |Z1 ,...,Z10 (p | H, . . . , H, T ) =
pZ1 ,Z2 ,...,Z10 |P (H, . . . , H, T | p)
fP ( p) .
pZ1 ,Z2 ,...,Z10 (H, H, . . . , H, T )
This expression is zero when p < 0 or p > 1 since the a priori pdf fP (p) is zero. For 0 p 1:
fP |Z1 ,...,Z10 (p|H, . . . , H, T ) =
=
=
p9 (1 p)
1
0 fZ1 ,Z2 ,...,Z10 |P (H, . . . , H, T | p)fP (p) dp
p9 (1 p)
19
0 p (1
p9 (1 p)
p) dp
1/110
= 110p9 (1 p).
12
Note that the result is independent of the order of heads and tails. This is due to the fact that
the tosses are conditionally independent, and therefore the conditional pmf of (Z1 , . . . , Z10 )
given P is a function only of the number of heads.
5. Gambling. Let X1 , X2 , X3 , . . . be independent random variables with the same mean > 0
1
and the same variance 2 . Find the limit of P{ n n Xi < /2} as n .
i=1
1
Solution: By the weak law of large numbers, the sample mean n n Xi converges to the
i=1
mean E(X ) in probability, so P(|Sn | < ) 1 as n . But if Sn > /2 then
P (Sn < /2) 0.
13