ORIE3510
Introduction to Engineering Stochastic Processes
Spring 2010
Section 4
Review
•
Stationary distribution interpretations
•
Computation of
π
•
Steadystate costs/rewards
•
Transient state analysis (expected number of visits to transient states & absorption proba
bilities)
Problem 4.67
Note that if
X
n
=
i
, then we can only move to
i

1 or
i
+ 1, or stay in
i
. Hence
P
i,i
+1
=
P
(
X
n
+1
=
i
+ 1

X
n
=
i, X
n

1
, . . . , X
0
) =
p
N

i
N
P
i,i
=
P
(
X
n
+1
=
i

X
n
=
i, X
n

1
, . . . , X
0
) =
p
i
N
+ (1

p
)
N

i
N
P
i,i

1
=
P
(
X
n
+1
=
i

1

X
n
=
i, X
n

1
, . . . , X
0
) = (1

p
)
i
N
.
Since the transition probabilities depend only on the fact that
X
n
=
i
, we see that this is a Markov
chain. Clearly, all classes communicate, so the chain is irreducible and hence recurrent. Further,
we see that the period of state 0 is 1 so the chain is aperiodic.
Suppose
N
= 2. Then we have
P
=
1

p
p
0
(1

p
)
1
2
1
2
p
1
2
0
1

p
p
and
π
0
= (1

p
)
π
0
+
1
2
(1

p
)
π
1
⇒
π
1
= 2
p
1

p
π
0
=
2
1
p
1
(1

p
)

1
π
0
and
π
2
=
1
2
pπ
1
+
pπ
2
⇒
π
2
=
p
1

p
2
π
0
=
2
2
p
2
(1

p
)

2
π
0
.
Since
∑
i
π
i
= 1 we have
π
0
=
1 +
2
X
i
=1
2
i
p
i
(1

p
)

i
!

1
=
1 + (1

p
)

2
2
X
i
=1
2
i
p
i
(1

p
)
2

i
!

1
=
(
1 + (1

p
)

2
(
1

(1

p
)
2
))

1
=
(1

p
)
2
so
π
0
= (1

p
)
2
. It then follows that
π
i
=
(
2
i
)
p
i
(1

p
)
2

i
for
i
= 1
,
2.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
For general
N
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '09
 RESNIK
 10%, Markov chain, 1 j, stationary distribution, 25 pt, 1 0 7 0 0 PT

Click to edit the document details