Math 216, Fall 2010
Homework 1
1. Read Chapter 1 carefully.
2. Suppose
X
n
is a Markov chain on the ﬁnite state space
S
with initial distribution
φ
and transition
matrix
P
ij
. Show that for any three states,
i,j,k
∈
S
and integers
n
≥
2,
a,b
≥
1.
(
P
n
)
ik
≥
(
P
a
)
ij
(
P
b
)
jk
if
a
+
b
=
n
. There may be ways to go from
i
to
k
in
n
steps. Some of those paths may go through
j
along the way. So, the inequality says that probability (
P
n
)
ik
is bounded below by the probabiliy of
ﬁrst going from
i
to
j
in
a
steps, then from
j
to
k
in
b
steps.
3. Suppose
X
n
is a Markov chain on the ﬁnite state space
S
with transition matrix
P
ij
. Suppose a
function
f
:
S
→
R
represents a payoﬀ or prize associated with each state:
f
(
y
) is the prize associated
with state
y
. Let
f
k
(
x
) be the expected payoﬀ at time
k
≥
0 when starting from state
x
:
f
k
(
x
) =
E
x
[
f
(
X
k
)]
,
for each
x
∈
S
(0.1)
Here
E
x
indicates expectation where the initial state is
x
(the initial distribution is 1 at the xstate,
This is the end of the preview.
Sign up
to
access the rest of the document.
 Fall '08
 Mckinley,S
 Linear Algebra, Matrices, Integers, Lawler, finite state space, transition matrix Pij

Click to edit the document details