Substituting into the normalization equation
π
1
+
∑
`
i
=1
π
(2
,i
)
= 1, we obtain
1=
1+
b
`
X
i
=1
(1
−
r
)
i
−
1
!
π
1
=
b
(
1
−
(1
−
r
)
`
)
r
!
π
1
,
or
π
1
=
r
r
+
b
(
1
−
(1
−
r
)
`
)
.
Using the equation
π
(2
,i
)
=(1
−
r
)
i
−
1
bπ
1
, we can also obtain explicit formulas for the
π
(2
,i
)
.
Solution to Problem 7.11.
We use a Markov chain model with 3 states,
H
,
M
,
and
E
, where the state reﬂects the diﬃculty of the most recent exam. We are given
the transition probabilities
#
r
HH
r
HM
r
HE
r
MH
r
MM
r
ME
r
EH
r
EM
r
EE
$
=
#
0
.
5
.
5
.
25
.
5
.
25
.
25
.
25
.
5
$
.
It is easy to see that our Markov chain has a single recurrent class, which is aperiodic.
The balance equations take the form
π
1
=
1
4
(
π
2
+
π
3
)
,
π
2
=
1
2
(
π
1
+
π
2
)+
1
4
π
3
,
π
3
=
1
2
(
π
1
+
π
3
1
4
π
2
,
and solving these with the constraint
∑
i
π
i
=1g
ives
π
1
=
1
5
,π
2
=
π
3
=
2
5
.
Solution to Problem 7.12.
(a) This is a generalization of Example 7.6. We may
proceed as in that example and introduce a Markov chain with states 0
,
1
,...,n
, where
state
i
indicates that there
i
available rods at Alvin’s present location. However, that
Markov chain has a somewhat complex structure, and for this reason, we will proceed
diFerently.
We consider a Markov chain with states 0
,
1
, where state
i
indicates that
Alvin is oF the island and has
i
rods available. Thus, a transition in this Markov chain
reﬂects two trips (going to the island and returning). It is seen that this is a birth-death
process. This is because if there are
i
rods oF the island, then at the end of the round
trip, the number of rods can only be
i
−
1,
i
or
i
+1.
We now determine the transition probabilities. When
i>
0, the transition prob-
ability
p
i,i
−
1
is the probability that the weather is good on the way to the island, but
is bad on the way back, so that
p
i,i
−
1
=
p
(1
−
p
). When 0
<i<n
, the transition prob-
ability
p
i,i
+1
is the probability that the weather is bad on the way to the island, but is
81