IEOR 4701
Assignment 2 Solution
Summer 2011
Question 1
: Suppose that (
X
n
:
n
≥
0) is a finite statespace Markov chain.
Suppose that
B
is a given subset of the statespace and define
T
B
= min
{
n
≥
0 :
X
n
∈
B
}
. We assume that
P
i
(
T
B
<
∞
) = 1 for all
i /
∈
B
.
Find systems of linear equations that are satisfied for the expectations in
a) and b) together with associated boundary conditions. Express your solution
using suitably defined vectors and matrices.
a)
g
(
i
) =
E
i
∑
T
B

1
j
=0
f
(
X
j
) for some function
f
(
·
).
b)
e
g
(
i
) =
E
i
exp

∑
T
B

1
k
=0
r
(
X
k
)
f
(
X
T
B
) , where
r
(
·
) is some given
nonnegative function and we set by convention that a sum over an empty set
is interpreted as zero. So, for instance, if
j
= 0 then
∑
j

1
k
=0
r
(
X
k
) = 0.
c) Now, define
τ
B
= min
{
n
≥
1 :
X
n
∈
B
}
(note that
τ
B
≥
1, whereas
T
B
≥
0). Also define
e
h
(
i
) =
E
i
exp

τ
B

1
X
k
=0
r
(
X
k
)
!
f
(
X
τ
B
)
!
.
Set up similar systems of linear equations as you did in a) and b). Do you have
clear boundary conditions this time? Can your solutions in a) and b) be of any
help to solve for the expectations in c) – explain how?
Solution.
a) First notice that for
i
∈
B
we have that
T
B
= 0 and hence
g
(
i
) = 0
.
If
i /
∈
B,
we can set up a recursion and use it to solve the problem. We have that
g
(
i
) =
E
i
[
∑
T
B

1
j
=0
f
(
X
j
)] =
f
(
i
) +
E
[
∑
T
B

1
j
=1
f
(
X
j
)]
.
Using the transition proba
bilities we have that
g
(
i
) =
f
(
i
) +
∑
k
∈
S
p
(
i, k
)
g
(
k
) =
f
(
i
) +
∑
k
∈
B
p
(
i, k
)
g
(
k
) +
∑
k /
∈
B
p
(
i, k
)
g
(
k
) (just split up the sum into two parts).
Now we have that
∑
k
∈
B
p
(
i, k
)
g
(
k
) = 0 and hence we get
g
(
i
) =
f
(
i
) +
∑
k /
∈
B
p
(
i, k
)
g
(
k
)
.
To
finish, define if
i /
∈
B
the vector
g
as the vector with components
g
(
i
) and simi
larly for the vector
f
.
Furthermore, if we define
Q
as the subset of the transition
matrix
P
with
p
(
i, j
) for
i, j /
∈
B
we have that
g
=
f
+
Q
g
.
b) We proceed as before.
If we start at a point
i
∈
B
then immediately
T
B
= 0 and hence he have the conditions ˜
g
(
i
) =
f
(
i
)
.
For
i /
∈
B,
we have the
recursion
˜
g
(
i
) =
E
i
[
e
∑
T
B

1
j
=0
r
(
X
j
)
f
(
X
T
B
)] =
e

r
(
i
)
E
i
[
e

∑
T
B

1
j
=1
r
(
X
j
)
f
(
X
T
B
)]
.
This last expression is equal to
e

r
(
i
)
∑
k
∈
S
p
(
i, k
)˜
g
(
k
) =
e

r
(
i
)
∑
k
∈
B
p
(
i, k
)˜
g
(
k
)+
e

r
(
i
)
∑
k /
∈
B
p
(
i, k
)˜
g
(
k
)
.
By the fact that ˜
g
(
i
) =
f
(
i
) if we have
i
∈
B
we get
that the last expression is actually
e

r
(
i
)
X
k
∈
B
p
(
i, k
)
f
(
k
) +
e

r
(
i
)
X
k /
∈
B
p
(
i, k
)˜
g
(
k
)
.
1
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
IEOR 4701
Assignment 2 Solution
Summer 2011
Hence, for
i /
∈
B,
we set ˜
g
as the vector whose components are ˜
g
(
i
)
,
f
as the
vector whose components are
f
(
i
) and
r
as the vector with components
e

r
(
i
)
.
This is the end of the preview.
Sign up
to
access the rest of the document.
 Summer '10
 KarlSigma
 Trigraph, Markov chain, Maximum likelihood, Boundary conditions

Click to edit the document details