Discrete-time stochastic processes

# At the beginning of the nth time interval the queue

This preview shows page 1. Sign up to view the full content.

This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: s no cycles, and thus contains one or more nodes with no outgoing edges. Show that each such node is in a recurrent class. Note: this result is not true for Markov chains with countably inﬁnite state spaces. Exercise 4.2. Consider a ﬁnite-state Markov chain in which some given state, say state 1, is accessible from every other state. Show that the chain has at most one recurrent class of states. (Note that, combined with Exercise 4.1, there is exactly one recurrent class and the chain is then a unichain.) Exercise 4.3. Show how to generalize the graph in Figure 4.4 to an arbitrary number of states M ≥ 3 with one cycle of M nodes and one of M − 1 nodes. For M = 4, let node 1 be the node not in the cycle of M − 1 nodes. List the set of states accessible from node 1 in n steps for each n ≤ 12 and show that the bound in Theorem 4.5 is met with equality. Explain why the same result holds for all larger M. 184 CHAPTER 4. FINITE-STATE MARKOV CHAINS Exercise 4.4. Consider a Markov chain with one ergodic class of m states, say {1, 2, . . . , m} n and M − m other states that are all transient. Show that Pij > 0 for all j ≤ m and n ≥ (m − 1)2 + 1 + M − m. Exercise 4.5. a) Let τ be the number of states in the smallest cycle of an arbitrary ergodic n Markov chain of M ≥ 3 states. Show that Pij > 0 for all n ≥ (M − 2)τ + M. Hint: Look at the last part of the proof of Theorem 4.4. b) For τ = 1, draw the graph of an ergodic Markov chain (generalized for arbitrary M ≥ 3) n for which there is an i, j for which Pij = 0 for n = 2M − 3. Hint: Look at Figure 4.4. d) For arbitrary τ < M − 1, draw the graph of an ergodic Markov chain (generalized for n arbitrary M) for which there is an i, j for which Pij = 0 for n = (M − 2)τ + M − 1. Exercise 4.6. A transition probability matrix P is said to be doubly stochastic if X Pij = 1 for all i; j X Pij = 1 for all j. i That is, the row sum and the column sum each equal 1. If a doubly stochastic chain has M states and is ergodic (i.e., has a single class of states and is aperiodic), calculate its steady-state probabilities. Exercise 4.7. a) Find the steady-state probabilities π0 , . . . , πk−1 for the Markov chain below. Express your answer in terms of the ratio ρ = p/q . Pay particular attention to the special case ρ = 1. b) Sketch π0 , . . . , πk−1 . Give one sketch for ρ = 1/2, one for ρ = 1, and one for ρ = 2. c) Find the limit of π0 as k approaches 1; give separate answers for ρ < 1, ρ = 1, and ρ > 1. Find limiting values of πk−1 for the same cases. ✿♥ ✘0 ② 1−p p 1−p ③ ♥ 1 ② p 1−p ③ ♥ 2 ② p 1−p ③ ... ♥ k−2 ② p 1−p ③♥ k−1 ② p Exercise 4.8. a) Find the steady-state probabilities for each of the Markov chains in Figure 4.2 of section 4.1. Assume that all clockwise probabilities in the ﬁrst graph are the same, say p, and assume that P4,5 = P4,1 in the second graph. b) Find the matrices [P ]2 for the same chains. Draw the graphs for the Markov chains represented by [P ]2 , i.e., the graph of two step transitions for the original chains. Find the steady-state probabilities for these two step chains. Explain why your steady-state probabilities are not unique. c) Find limn→1 [P ]2n for each of the chains. 4.8. EXERCISES 185 Exercise 4.9. Answer each of the following questions for each of the following non-negative matrices [A] i) ∑ 10 11 ∏ ii) 1 0 0 1/2 1/2 0 . 0 1/2 1/2 a) Find [A]n in closed form for arbitrary n > 1. b) Find all eigenvalues and all right eigenvectors of [A]. c) Use (b) to show that there is no diagonal matrix [Λ] and no invertible matrix [Q] for which [A][Q] = [Q][Λ]. d) Rederive the result of part (c) using the result of (a) rather than (b). Exercise 4.10. a) Show that g (x ), as given in (4.21), is a continuous function of x for x ≥ 0 , x 6= 0 . b) Show that g (x ) = g (β x ) for all β > 0. Show that this implies that the supremum of P g (x ) over x ≥ 0 , x 6= 0 is the same as the supremum over x ≥ 0 , i xi = 1. Note that this shows that the supremum must be achieved, since it is a supremum of a continuous function over a closed and bounded space. Exercise 4.11. a) Show that if x1 and x2 are real or complex numbers, then |x1 + x2 | = |x1 | + |x2 | implies that for some β , β x1 and β x2 are both real and non-negative. b) Show from this that if the inequality in (4.25) is satisﬁed with equality, then there is some β for which β xi = |xi | for all i. Exercise 4.12. a) Let ∏ be an eigenvalue of a matrix [A], and let ∫ and π be right and left eigenvectors respectively of ∏, normalized so that π ∫ = 1. Show that [[A] − ∏∫ π ]2 = [A]2 − ∏2∫ π . b) Show that [[A]n − ∏n∫ π ][[A] − ∏∫ π ] = [A]n+1 − ∏n+1∫ π . c) Use induction to show that [[A] − ∏∫ π ]n = [A]n − ∏n∫ π . Exercise 4.13. Let [P ] be the transition matrix for a Markov unichain with M recurrent states, numbered 1 to M, and K transient states, J + 1 to J + K . Thus [P...
View Full Document

## This note was uploaded on 09/27/2010 for the course EE 229 taught by Professor R.srikant during the Spring '09 term at University of Illinois, Urbana Champaign.

Ask a homework question - tutors are online