ORIE3510
Introduction to Engineering Stochastic Processes
Spring 2010
Section 6
Problem 1
Let
{
X
n
}
be a DTMC with TPM
P
. Let
A
⊆
S
, and suppose we want to compute
P
(
X
enters
A
by time
m
) =
P
(
X
k
∈
A
,
for some
k
= 1
, . . . , m

X
0
=
i
) =:
β.
To determine
β
, we define an auxiliary MC
{
W
n
}
consisting of all states except those in
A
, plus
an additional state, call it
A
. Further, we make the state
A
absorbing. Let
N
= min
{
n
:
X
n
∈
A
}
and let
N
=
∞
if
X
n
/
∈
A
for all
n
. Hence,
N
is the first hitting time of set
A
for the chain
{
X
n
}
.
Define
W
n
=
X
n
,
if
n < N
A,
if
n
≥
N
.
Hence
X
and
W
follow the same path up until the point where
X
enters
A
, at which time
W
goes
to
A
and remains there forever. The TPM for
W
is given by
Q
ij
=
P
ij
,
if
i /
∈
A
, j /
∈
A
Q
iA
=
X
j
∈
A
P
ij
,
if
i /
∈
A
Q
AA
=
1
.
Finally, because
X
n
will have entered
A
my time
m
if and only if
W
m
=
A
, we have
β
=
P
(
X
k
∈
A
,
for some
k
= 1
, . . . , m

X
0
=
i
)
=
P
(
W
m
=
A

X
0
=
i
) =
P
(
W
m
=
A

W
0
=
i
) =
Q
m
iA
.
Next, suppose we want to compute
α
:=
P
(
X
m
=
j, X
k
/
∈
A
, k
= 1
, . . . , m

1

X
0
=
i
)
,
i, j /
∈
A
.
Noting that
[
X
m
=
j, X
k
/
∈
A
, k
= 1
, . . . , m

1] = [
W
m
=
j
]
,
it follows that
α
=
Q
m
ij
.
If
i /
∈
A
, but
j
∈
A
, we condition on the penultimate step, and get
α
=
X
r/
∈
A
P
(
X
m
=
j, X
m

1
=
r, X
k
/
∈
A
, k
= 1
, . . . , m

2

X
0
=
i
)
=
X
r/
∈
A
P
(
X
m
=
j

X
m

1
=
r, X
k
/
∈
A
, k
= 1
, . . . , m

2
, X
0
=
i
)
×
P
(
X
m

1
=
r, X
k
/
∈
A
, k
= 1
, . . . , m

2

X
0
=
i
)
=
X
r/
∈
A
P
rj
P
(
X
m

1
=
r, X
k
/
∈
A
, k
= 1
, . . . , m

2

X
0
=
i
)
=
X
r/
∈
A
P
rj
Q
m

1
ir
.
Similarly, if
i
∈
A
we can condition on the first transition to get
P
(
X
m
=
j, X
k
/
∈
A
, k
= 1
, . . . , m

1

X
0

i
) =
X
r/
∈
A
Q
m

1
rj
P
i
r,
j /
∈
A
.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Problem 2
Consider an irreducible, positive recurrent DTMC on a state space consisting of
n
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '09
 RESNIK
 Exponential distribution, jAc iA

Click to edit the document details