This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: 6.262 Discrete Stochastic Processes, Spring 2010 Problem Set 8 — Solutions due: Friday, April 14, 2010 Problem 1 (Exercise 5.1) Let { P ij  i,j ∈ N } be the set of transition probabilities for a countable state Markov chain. For each i,j , let F ij ( n ) be the probability that state j occurs sometime between time 1 and n inclusive, given X = i . Given j ∈ N , suppose that { x k  k ≥ } is a set of nonnegative numbers satisfying x i = P ij + X k 6 = j P ik x k . ( ? ) The goal is to show that x i ≥ F i,j ( n ) for all i ∈ N . First off, why are we doing this? Equation 5.8 in the notes and the surrounding discussion tells us that the set of probabilities F i,j ( n ) for i,j ∈ N satisfy F i,j ( n ) = P ij + X k 6 = j P ik F k,j ( n 1) . Next, Equation 5.9 states that F ij ( ∞ ), defined as the large n limit of F ij ( n ) satisfies ( ? ), that is, F i,j ( ∞ ) = P i,j + X k 6 = j P ik F k,j ( ∞ ) . However, the text also warns us that the equation in question may not determine F i,j ( ∞ ) uniquely. In particular, note that letting x i = 1 for all i ∈ N constitutes a valid solution to ( ? ). However, F i,j ( ∞ ) = 1 for all i,j if and only if the chain contains no transient states. Now suppose we can show that for any given j and any solution { x 1 ,x 2 ,... } to the system of equations given by x i = P ij + ∑ k 6 = j P ik x k , we must have x i ≥ F i,j ( n ) for any n . Taking limits, it will follow that x i ≥ lim n →∞ F i,j ( n ) = F i,j ( ∞ ), from which we will immediately conclude that F i,j ( ∞ ) is the minimal nonnegative solution to ( ? ). And that’s the point. This type of argument is best approached by induction. For n = 1, the claim follows immedi ately, as F i,j (1) = P i,j ≤ P i,j + X k 6 = j P i,k x k = x j for all i ∈ N , where the first inequality follows from the fact that x i ≥ 0 for all i , and the last equality follows from the definition of { x i } (i.e. as solutions to ( ? )). As a sanity check (and since we were asked to), let’s also do n = 2. F i,j (2) = P i,j + X k 6 = j P i,k F k,j (1) ≤ P i,j + X k 6 = j P i,k x k = x j for all i ∈ N , where the inequality follows by the previously shown fact that F i,j (1) ≤ x j for all i ∈ N . More generally, let us show that if the claim holds for some n , it will also hold for n + 1. (And since we have already shown that it holds for n = 1, it will follow that the statement is true for all 1 n ∈ N .) Specifically, suppose that for some n ∈ N , we have that x i ≥ F i,j ( n ) for all i ∈ N . Recalling that F i,j ( n + 1) = P i,j + X k 6 = j P i,k F k,j ( n ) , and observing that by the induction hypothesis, x i ≥ F i,j ( n ), we obtain F i,j ( n + 1) ≤ P i,j + X k 6 = j P i,k x k = x i , where the last equality follows from the fact that x 1 ,x 2 ,... is a solution to ( ? ). The inductive step is therefore verified, and the result follows....
View
Full Document
 Spring '08
 Moon,J
 Probability theory, Xn

Click to edit the document details