This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: 6.262 Discrete Stochastic Processes, Spring 2010 Problem Set 8 — Solutions due: Friday, April 14, 2010 Problem 1 (Exercise 5.1) Let { P ij  i,j ∈ N } be the set of transition probabilities for a countable state Markov chain. For each i,j , let F ij ( n ) be the probability that state j occurs sometime between time 1 and n inclusive, given X = i . Given j ∈ N , suppose that { x k  k ≥ } is a set of nonnegative numbers satisfying x i = P ij + X k 6 = j P ik x k . ( ? ) The goal is to show that x i ≥ F i,j ( n ) for all i ∈ N . First off, why are we doing this? Equation 5.8 in the notes and the surrounding discussion tells us that the set of probabilities F i,j ( n ) for i,j ∈ N satisfy F i,j ( n ) = P ij + X k 6 = j P ik F k,j ( n 1) . Next, Equation 5.9 states that F ij ( ∞ ), defined as the large n limit of F ij ( n ) satisfies ( ? ), that is, F i,j ( ∞ ) = P i,j + X k 6 = j P ik F k,j ( ∞ ) . However, the text also warns us that the equation in question may not determine F i,j ( ∞ ) uniquely. In particular, note that letting x i = 1 for all i ∈ N constitutes a valid solution to ( ? ). However, F i,j ( ∞ ) = 1 for all i,j if and only if the chain contains no transient states. Now suppose we can show that for any given j and any solution { x 1 ,x 2 ,... } to the system of equations given by x i = P ij + ∑ k 6 = j P ik x k , we must have x i ≥ F i,j ( n ) for any n . Taking limits, it will follow that x i ≥ lim n →∞ F i,j ( n ) = F i,j ( ∞ ), from which we will immediately conclude that F i,j ( ∞ ) is the minimal nonnegative solution to ( ? ). And that’s the point. This type of argument is best approached by induction. For n = 1, the claim follows immedi ately, as F i,j (1) = P i,j ≤ P i,j + X k 6 = j P i,k x k = x j for all i ∈ N , where the first inequality follows from the fact that x i ≥ 0 for all i , and the last equality follows from the definition of { x i } (i.e. as solutions to ( ? )). As a sanity check (and since we were asked to), let’s also do n = 2. F i,j (2) = P i,j + X k 6 = j P i,k F k,j (1) ≤ P i,j + X k 6 = j P i,k x k = x j for all i ∈ N , where the inequality follows by the previously shown fact that F i,j (1) ≤ x j for all i ∈ N . More generally, let us show that if the claim holds for some n , it will also hold for n + 1. (And since we have already shown that it holds for n = 1, it will follow that the statement is true for all 1 n ∈ N .) Specifically, suppose that for some n ∈ N , we have that x i ≥ F i,j ( n ) for all i ∈ N . Recalling that F i,j ( n + 1) = P i,j + X k 6 = j P i,k F k,j ( n ) , and observing that by the induction hypothesis, x i ≥ F i,j ( n ), we obtain F i,j ( n + 1) ≤ P i,j + X k 6 = j P i,k x k = x i , where the last equality follows from the fact that x 1 ,x 2 ,... is a solution to ( ? ). The inductive step is therefore verified, and the result follows....
View
Full
Document
This note was uploaded on 10/21/2010 for the course EE 5581 taught by Professor Moon,j during the Spring '08 term at Minnesota.
 Spring '08
 Moon,J

Click to edit the document details