IEOR 4106: Introduction to Operations Research: Stochastic Models
Spring 2009, Professor Whitt
Class Lecture Notes: Tuesday, March 3.
ContinuousTime Markov Chains, Ross Chapter 6
Problems for Discussion and Solutions
1. Pooh Bear and the Three Honey Trees.
A bear of little brain named Pooh is fond of honey. Bees producing honey are located in
three trees: tree
A
, tree
B
and tree
C
. Tending to be somewhat forgetful, Pooh goes back
and forth among these three honey trees randomly (in a Markovian manner) as follows: From
A
, Pooh goes next to
B
or
C
with probability 1
/
2 each; from
B
, Pooh goes next to
A
with
probability 3
/
4, and to
C
with probability 1
/
4; from
C
, Pooh always goes next to
A
. Pooh
stays a random time at each tree. (Assume that the travel times can be ignored.) Pooh stays
at each tree an exponential length of time, with the mean being 5 hours at tree
A
or
B
, but
with mean 4 hours at tree
C
.
(a) Construct a CTMC enabling you to ﬁnd the limiting proportion of time that Pooh
spends at each honey tree.
(b) What is the average number of trips per day Pooh makes from tree
B
to tree
A
?
ANSWERS:
(a) Find the limiting fraction of time that Pooh spends at each tree.
———————————————————————
These problems are part of a longer set of lecture notes, which have been posted on the
web page. The focus here is on diﬀerent ways to model. We are thus focusing on Section 3 of
the notes.
For this problem formulation, it is natural to use the SMP (semiMarkov process) formu
lation of a CTMC (continuoustime Markov chain), involving the embedded DTMC and the
mean holding times in each state. (See Sections 6.2 and 7.6 of Ross.) We thus deﬁne the
embedded transition matrix
P
directly and the mean holding times 1
/ν
i
directly.
Note that this problem is formulated directly in terms of the DTMC, describing the random
motion at successive transitions, so it is natural to use this initial modelling approach. Here
the transition matrix for the DTMC is
P
=
A
B
C
0
1
/
2 1
/
2
3
/
4
0
1
/
4
1
0
0
.
In the displayed transition matrix
P
, we have only labelled the rows. The columns are assumed
to be labelled in the same order.
In general, the steadystate probability is
α
i
=
π
i
(1
/ν
i
)
∑
j
π
j
(1
/ν
j
)
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentIn this case, the steady state probability vector of the discretetime Markov chain is obtained
by solving
π
=
πP
, yielding
π
= (
8
17
,
4
17
,
5
17
)
.
Then the ﬁnal steadystate distribution, accounting for the random holding times is
α
= (
1
2
,
1
4
,
1
4
)
.
You could alternatively work with the inﬁnitesimal transition rate matrix
Q
. If we want
to deﬁne the inﬁnitesimal transition matrix
Q
(Ross uses lower case
q
), then we can do so by
setting
Q
i,j
=
ν
i
P
i,j
for
i
6
=
j .
This is the end of the preview.
Sign up
to
access the rest of the document.
 Spring '08
 Whitt
 Operations Research, Markov chain, longrun proportion, copier, CTMC, A. Pooh

Click to edit the document details