This preview shows page 1. Sign up to view the full content.
Unformatted text preview: s no cycles, and
thus contains one or more nodes with no outgoing edges. Show that each such node is in a
recurrent class. Note: this result is not true for Markov chains with countably inﬁnite state
spaces.
Exercise 4.2. Consider a ﬁnitestate Markov chain in which some given state, say state 1,
is accessible from every other state. Show that the chain has at most one recurrent class of
states. (Note that, combined with Exercise 4.1, there is exactly one recurrent class and the
chain is then a unichain.)
Exercise 4.3. Show how to generalize the graph in Figure 4.4 to an arbitrary number of
states M ≥ 3 with one cycle of M nodes and one of M − 1 nodes. For M = 4, let node 1
be the node not in the cycle of M − 1 nodes. List the set of states accessible from node 1
in n steps for each n ≤ 12 and show that the bound in Theorem 4.5 is met with equality.
Explain why the same result holds for all larger M. 184 CHAPTER 4. FINITESTATE MARKOV CHAINS Exercise 4.4. Consider a Markov chain with one ergodic class of m states, say {1, 2, . . . , m}
n
and M − m other states that are all transient. Show that Pij > 0 for all j ≤ m and
n ≥ (m − 1)2 + 1 + M − m.
Exercise 4.5. a) Let τ be the number of states in the smallest cycle of an arbitrary ergodic
n
Markov chain of M ≥ 3 states. Show that Pij > 0 for all n ≥ (M − 2)τ + M. Hint: Look at
the last part of the proof of Theorem 4.4.
b) For τ = 1, draw the graph of an ergodic Markov chain (generalized for arbitrary M ≥ 3)
n
for which there is an i, j for which Pij = 0 for n = 2M − 3. Hint: Look at Figure 4.4.
d) For arbitrary τ < M − 1, draw the graph of an ergodic Markov chain (generalized for
n
arbitrary M) for which there is an i, j for which Pij = 0 for n = (M − 2)τ + M − 1.
Exercise 4.6. A transition probability matrix P is said to be doubly stochastic if
X Pij = 1 for all i; j X Pij = 1 for all j. i That is, the row sum and the column sum each equal 1. If a doubly stochastic chain has
M states and is ergodic (i.e., has a single class of states and is aperiodic), calculate its
steadystate probabilities.
Exercise 4.7. a) Find the steadystate probabilities π0 , . . . , πk−1 for the Markov chain
below. Express your answer in terms of the ratio ρ = p/q . Pay particular attention to the
special case ρ = 1.
b) Sketch π0 , . . . , πk−1 . Give one sketch for ρ = 1/2, one for ρ = 1, and one for ρ = 2.
c) Find the limit of π0 as k approaches 1; give separate answers for ρ < 1, ρ = 1, and
ρ > 1. Find limiting values of πk−1 for the same cases.
✿♥
✘0
② 1−p p
1−p ③
♥
1
② p
1−p ③
♥
2
② p
1−p ③ ... ♥
k−2
② p
1−p ③♥
k−1
② p Exercise 4.8. a) Find the steadystate probabilities for each of the Markov chains in
Figure 4.2 of section 4.1. Assume that all clockwise probabilities in the ﬁrst graph are the
same, say p, and assume that P4,5 = P4,1 in the second graph.
b) Find the matrices [P ]2 for the same chains. Draw the graphs for the Markov chains
represented by [P ]2 , i.e., the graph of two step transitions for the original chains. Find
the steadystate probabilities for these two step chains. Explain why your steadystate
probabilities are not unique.
c) Find limn→1 [P ]2n for each of the chains. 4.8. EXERCISES 185 Exercise 4.9. Answer each of the following questions for each of the following nonnegative
matrices [A] i) ∑ 10
11 ∏ ii) 1
0
0 1/2 1/2 0 .
0 1/2 1/2 a) Find [A]n in closed form for arbitrary n > 1.
b) Find all eigenvalues and all right eigenvectors of [A].
c) Use (b) to show that there is no diagonal matrix [Λ] and no invertible matrix [Q] for
which [A][Q] = [Q][Λ].
d) Rederive the result of part (c) using the result of (a) rather than (b).
Exercise 4.10. a) Show that g (x ), as given in (4.21), is a continuous function of x for
x ≥ 0 , x 6= 0 .
b) Show that g (x ) = g (β x ) for all β > 0. Show that this implies that the supremum of
P
g (x ) over x ≥ 0 , x 6= 0 is the same as the supremum over x ≥ 0 , i xi = 1. Note that
this shows that the supremum must be achieved, since it is a supremum of a continuous
function over a closed and bounded space.
Exercise 4.11. a) Show that if x1 and x2 are real or complex numbers, then x1 + x2  =
x1  + x2  implies that for some β , β x1 and β x2 are both real and nonnegative.
b) Show from this that if the inequality in (4.25) is satisﬁed with equality, then there is
some β for which β xi = xi  for all i.
Exercise 4.12. a) Let ∏ be an eigenvalue of a matrix [A], and let ∫ and π be right and
left eigenvectors respectively of ∏, normalized so that π ∫ = 1. Show that
[[A] − ∏∫ π ]2 = [A]2 − ∏2∫ π .
b) Show that [[A]n − ∏n∫ π ][[A] − ∏∫ π ] = [A]n+1 − ∏n+1∫ π .
c) Use induction to show that [[A] − ∏∫ π ]n = [A]n − ∏n∫ π . Exercise 4.13. Let [P ] be the transition matrix for a Markov unichain with M recurrent
states, numbered 1 to M, and K transient states, J + 1 to J + K . Thus [P...
View
Full
Document
This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.
 Spring '09
 R.Srikant

Click to edit the document details