31
A.2. SOLUTIONS FOR CHAPTER 2
d) Find the density of the interarrival time for cfw_NA (t); t
0. Note: This part is quite messy and is done
most easily via Laplace transforms.
Solution: The process cfw_NA (t); t > 0 is not a Poisson process, but, perhaps
30
APPENDIX A. SOLUTIONS TO EXERCISES
where
Z
m(t) =
t
[1
G( )] d =
0
Z
t
F(a/ ) d.
0
Finally, note that the answer is the same if V is discrete or mixed.
Exercise 2.23: Let cfw_N1 (t); t > 0 be a Poisson counting process of rate . Assume that the arrival
29
A.2. SOLUTIONS FOR CHAPTER 2
1
X
n=i
f(si |n)p(n) =
=
=
1
si 1 X (t si )n
i
(i i)!
(n i)!
i si 1 e
i
(i
n=i
si
i)!
i si 1 e
i
(i
1
X
i
n
n i (t
i)!
t
si )n i e
(n i)!
n=i
si
e
(t si )
.
This is the Erlang distribution, and it follows because the preced
28
APPENDIX A. SOLUTIONS TO EXERCISES
PMF of the number waiting at time t and take the limit of this PMF as t ! 1. For very
large t, the number M of combined arrivals before t is large with high probability. Given
M = m, the geometric distribution above i
A.2. SOLUTIONS FOR CHAPTER 2
27
in (A.12). In other words, Nm is independent of the arrival time of bus m 1. From the
formula in (A.12), the PMF of the number entering a bus is also independent of m. Thus
the desired PMF is that on the right side of (A.12
26
APPENDIX A. SOLUTIONS TO EXERCISES
Solution: Let the sample value of N1 + N2 be n12 . By the same argument in (c),
n
pN1 +N2 |N (n12 |n) =
(p1 + p2 )n12 (1 p1 p2 )n n12 .
n12
e) Find the conditional PMF for N given that N1 = n1 .
Solution: Since N is
23
A.2. SOLUTIONS FOR CHAPTER 2
Solution: There are 2t binary n-tuples and each has a probability, so the joint PMF can
be viewed as a vector of 2t numbers. The binomial probability pN ( ) (k) = q k (1 q) k
k
species the sum of the probabilities of the n-
24
APPENDIX A. SOLUTIONS TO EXERCISES
Taking the limit as m ! 1, each of the n terms in the product approaches 1, so the limit
is 1/n!, verifying the rst equality in (c). For the second,
1
t
m
m
n
t
= exp (m n) ln 1
= exp (m
m
n t
= exp
t+
+ (m n)o(1/m) .
25
A.2. SOLUTIONS FOR CHAPTER 2
e
Solution: Again expressing N (t+s) = N (t) + N (t, t+s),
h
i
e
E [N (t) N (t+s)] = E N 2 (t) + E N (t)N (t, t+s)
= E N 2 (t) + E [N (t)] E [N (s)]
=
t+
t + t s.
2 2
In the nal step, we have used the fact (from Table 1.2)
22
APPENDIX A. SOLUTIONS TO EXERCISES
Your solution should contain four 3-tuples with probability 1/8 each, two 3-tuples with probability 1/4 each,
and two 3-tuples with probability 0. Note that by making the subsequent arrivals IID and equiprobable, you
18
APPENDIX A. SOLUTIONS TO EXERCISES
Gaussian with mean 0 and standard deviation 103 . Since one or more occurrences of 1012
occur only with probability 10 4 , this can be neglected in the sketch, so the CDF is approximately Gaussian with 3 sigma points
19
A.1. SOLUTIONS FOR CHAPTER 1
Exercise 1.48: Let cfw_Yn ; n
cfw_Yn ; n
1 be a sequence of rvs and assume that limn!1 E [|Yn |] = 0. Show that
1 converges to 0 in probability. Hint 1: Look for the easy way. Hint 2: The easy way uses the
Markov inequality
17
A.1. SOLUTIONS FOR CHAPTER 1
Exercise 1.44: Let X1 , X2 . . . be a sequence of IID rvs each with mean 0 and variance
p
Sn = X1 + + Xn for all n and consider the random variable Sn / n
2
. Let
p
S2n / 2n. Find the limiting
CDF for this sequence of rv 0
15
A.1. SOLUTIONS FOR CHAPTER 1
Solution:
n
=
k+1
n!
n!
=
(k + 1)!(n k 1)!
k!(k + 1)(n k)!/(n
n n k
n n k
=
.
k k+1
k
k
b) Extend (a) to show that, for all ` n
k,
!
n
k+`
!
n
n k
k
k
Solution: Using k + ` in place of k + 1 in (A.5),
n
n
n k (` 1)
k+`
16
APPENDIX A. SOLUTIONS TO EXERCISES
follows. Replacing the upper bound in (A.8) with the asymptotic inequality in (A.9), we
get Prcfw_Sn k ppqp pSn (k).
Exercise 1.39: Let cfw_Xi ; i
1 be IID binary rvs. Let Prcfw_Xi = 1 = , Prcfw_Xi = 0 = 1
. Let
Sn =
14
APPENDIX A. SOLUTIONS TO EXERCISES
Solution: The mean value of Ij (x) is FX (x) and the variance (after a short calculation) is
FX (x)Fc (x). This is nite (and in fact at most 1/4), so Theorem 1.7.1 applies and
X
8
9
n
< 1X
=
lim Pr
Ij (x) FX (x) > = 0
12
0 as z
nite.
APPENDIX A. SOLUTIONS TO EXERCISES
2
as z ! 1, so the integral is nite. Thus r+ belongs to the region where gZ (r) is
The whole point of this is that the random variables for which r+ = are those for which
the density or PMF go to 0 with i
9
A.1. SOLUTIONS FOR CHAPTER 1
Only one or two lines of explanation are needed.
Solution: We have shown that X1 + X2 is a rv, so (X1 + X2 ) + X3 is a rv, etc.
Exercise 1.15: (Stieltjes integration) a) Let h(x) = u(x) and FX (x) = u(x) where u(x) is the un
2
A.1
APPENDIX A. SOLUTIONS TO EXERCISES
Solutions for Chapter 1
Exercise 1.2: This exercise derives the probability of an arbitrary (non-disjoint) union of events, derives
the union bound, and derives some useful limit expressions.
a) For 2 arbitrary eve
59
A.4. SOLUTIONS FOR CHAPTER 4
Solution: The ergodicity of k and k 0 assures the inherently reachable assumption of Theorem 4.6.8, and thus we know from the fact that k satises the termination condition of
the policy improvement algorithm that g g 0 . Th
57
A.4. SOLUTIONS FOR CHAPTER 4
of nal cost going from 1 to 0. If George is in the drive state (2) and uses the try policy,
r2 = 0 and with probability 1/2, he parks and has one unit nal cost getting to the theatre.
With probability 1/2, he doesnt park an
60
A.5
APPENDIX A. SOLUTIONS TO EXERCISES
Solutions for Chapter 5
Exercise 5.1: The purpose of this exercise is to show that for an arbitrary renewal process, N (t), the
number of renewals in (0, t] is a (non-defective) random variable.
a) Let X1 , X2 , .
58
APPENDIX A. SOLUTIONS TO EXERCISES
(k0 )
(k )
Solution: For all i 2 R0 , Pij i = Pij i . The steady state vector is determined solely by
0
0
the transition probabilities in the recurrent class, so i = i for all i 2 R0 . Since ri = ri for
0 , it also fo
55
A.4. SOLUTIONS FOR CHAPTER 4
Solution:
p1
: n
0 H
Y
p0
0
01
010
p1
p0
z
X n
z
X n
z
X n 1
1
2
3 X
y
p1
p0
The solution for v0 and v1 in terms of v2 is the same as (a). The basic equation for v2 in
terms of its outward transitions is
v2 = 1 + p0 v0 +
54
APPENDIX A. SOLUTIONS TO EXERCISES
Finally, state 2 is the trapping state, so v2 = 0 and v0 = 1/p0 p1 .
b) For parts b) to d), let (a1 , a2 , a3 , . . . , ak ) = (0, 1, 1, . . . , 1), i.e., zero followed by k
1 ones. Draw the
corresponding Markov chain
56
APPENDIX A. SOLUTIONS TO EXERCISES
c) Let r be an arbitrary reward vector and consider the equation
w + g (1) (1) + g (2) (2) = r + [P ]w .
(A.19)
Determine what values g (1) and g (2) must have in order for (A.19) to have a solution. Argue that with t
52
APPENDIX A. SOLUTIONS TO EXERCISES
Solution: With arbitrary P12 , the steady state probabilities become
1 =
0.01
;
P12 + 0.01
2 =
P12
.
P12 + 0.01
The steady state gain, g = r , then becomes g = P12 /(P12 + 0.01). Solving for w as before,
we get
w1 =
P
51
A.4. SOLUTIONS FOR CHAPTER 4
b) Suppose one has found the expected rst-passage-times vj for states j = 2 to 4 (or in general from 2 to
M). Find an expression for v1 , the expected rst recurrence time for state 1 in terms of v2 , v3 , . . . vM and
P12 ,
47
A.4. SOLUTIONS FOR CHAPTER 4
From the rst equation, 2 = 0 and from the third 3 = 0, so 2 = 1 is the right eigenvector,
unique within a scale factor.
d) Use (c) to show that there is no diagonal matrix [] and no invertible matrix [U ] for which [P ][U ]
53
A.4. SOLUTIONS FOR CHAPTER 4
Solution: We have already seen that v1 (n) = 0 for all n, and thus, since P23 = 0, we have
v2 (n) = 1 + (1/2)v2 (n 1). Since v2 (0) = 0, this iterates to
v2 (n) = 1 +
1
1 + v2 (n 2)
2
= 1+
1 1
+ 1 + v2 (n 3)
2 4
= = 2
2
(n