This preview shows page 1. Sign up to view the full content.
Math 216, Fall 2010
Homework 1
1. Read Chapter 1 carefully.
2. Suppose
X
n
is a Markov chain on the ﬁnite state space
S
with initial distribution
φ
and transition
matrix
P
ij
. Show that for any three states,
i,j,k
∈
S
and integers
n
≥
2,
a,b
≥
1.
(
P
n
)
ik
≥
(
P
a
)
ij
(
P
b
)
jk
if
a
+
b
=
n
. There may be ways to go from
i
to
k
in
n
steps. Some of those paths may go through
j
along the way. So, the inequality says that probability (
P
n
)
ik
is bounded below by the probabiliy of
ﬁrst going from
i
to
j
in
a
steps, then from
j
to
k
in
b
steps.
3. Suppose
X
n
is a Markov chain on the ﬁnite state space
S
with transition matrix
P
ij
. Suppose a
function
f
:
S
→
R
represents a payoﬀ or prize associated with each state:
f
(
y
) is the prize associated
with state
y
. Let
f
k
(
x
) be the expected payoﬀ at time
k
≥
0 when starting from state
x
:
f
k
(
x
) =
E
x
[
f
(
X
k
)]
,
for each
x
∈
S
(0.1)
Here
E
x
indicates expectation where the initial state is
x
(the initial distribution is 1 at the xstate,
This is the end of the preview. Sign up
to
access the rest of the document.
 Fall '08
 Mckinley,S
 Integers

Click to edit the document details