# hw1 - Math 216 Fall 2010 Homework 1 1 Read Chapter 1...

This preview shows page 1. Sign up to view the full content.

Math 216, Fall 2010 Homework 1 1. Read Chapter 1 carefully. 2. Suppose X n is a Markov chain on the ﬁnite state space S with initial distribution φ and transition matrix P ij . Show that for any three states, i,j,k S and integers n 2, a,b 1. ( P n ) ik ( P a ) ij ( P b ) jk if a + b = n . There may be ways to go from i to k in n steps. Some of those paths may go through j along the way. So, the inequality says that probability ( P n ) ik is bounded below by the probabiliy of ﬁrst going from i to j in a steps, then from j to k in b steps. 3. Suppose X n is a Markov chain on the ﬁnite state space S with transition matrix P ij . Suppose a function f : S R represents a payoﬀ or prize associated with each state: f ( y ) is the prize associated with state y . Let f k ( x ) be the expected payoﬀ at time k 0 when starting from state x : f k ( x ) = E x [ f ( X k )] , for each x S (0.1) Here E x indicates expectation where the initial state is x (the initial distribution is 1 at the x-state,
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

Ask a homework question - tutors are online