Math 216, Fall 2010 Homework 1 1. Read Chapter 1 carefully. 2. Suppose X n is a Markov chain on the ﬁnite state space S with initial distribution φ and transition matrix P ij . Show that for any three states, i,j,k ∈ S and integers n ≥ 2, a,b ≥ 1. ( P n ) ik ≥ ( P a ) ij ( P b ) jk if a + b = n . There may be ways to go from i to k in n steps. Some of those paths may go through j along the way. So, the inequality says that probability ( P n ) ik is bounded below by the probabiliy of ﬁrst going from i to j in a steps, then from j to k in b steps. 3. Suppose X n is a Markov chain on the ﬁnite state space S with transition matrix P ij . Suppose a function f : S → R represents a payoﬀ or prize associated with each state: f ( y ) is the prize associated with state y . Let f k ( x ) be the expected payoﬀ at time k ≥ 0 when starting from state x : f k ( x ) = E x [ f ( X k )] , for each x ∈ S (0.1) Here E x indicates expectation where the initial state is x (the initial distribution is 1 at the x-state,
This is the end of the preview.
access the rest of the document.