This preview shows page 1. Sign up to view the full content.
Unformatted text preview: r al l transient.
Proof: Let j be a recurrent state. From Theorem 5.1, all states in a class are recurrent or
all are transient. Next suppose that j is positiverecurrent, so that 1/T j j > 0. Let i be in
the same class as j , and consider the renewalreward process on {Nj j (t); t ≥ 0} for which
R(t) = 1 whenever the process is in state i (i.e., if Xn = i, then R(t) = 1 for n ≤ t < n + 1).
The reward is 0 whenever the process is in some state other than i. Let E [Rn ] be the
expected reward in an interrenewal interval; this must be positive since i is accessible from
j . From the strong law for renewalreward processes, Theorem 3.6,
Zt
1
E [Rn ]
lim
R(τ )dτ =
with probability 1.
t→1 t
T jj
0
The term on the left is the timeaverage number of transitions into state i, given X0 = j ,
and this is 1/T ii from (5.11). Since E [Rn ] > 0 and T j j < 1, we have 1/T ii > 0, so i is
positiverecurrent. Thus if one state is positiverecurrent, the entire class is, completing the
proof.
If all of the states in a Markov chain are in a nullrecurrent class, then 1/T j j = 0 for each
state, and one might think of 1/T j j = 0 as a “steadystate” probability for j in the sense
that 0 is both the timeaverage rate of occurrence of j and the limiting probability of j .
However, these “probabilities” do not add up to 1, so a steadystate probability distribution
does not exist. This appears rather paradoxical at ﬁrst, but the example of Figure 5.1, with
p = 1/2 will help to clarify the situation. As time n increases (starting in state i, say), the
random variable Xn spreads out over more and more states around i,P thus is less likely
and
n
to be in each individual state. For each j , limn→1 Pij (n) = 0. Thus, j {limn→1 Pij } = 0.
Pn
On the other hand, for every n, j Pij = 1. This is one of those unusual examples where
a limit and a sum cannot be interchanged.
In Chapter 4, we deﬁned the steadystate distribution of a ﬁnitestate Markov chain as a
probability vector π that satisﬁes π = π [P ]. Here we deﬁne {πi ; i ≥ 0} in the same way, as
a set of numbers that satisfy
X
X
πj =
πi Pij for all j ;
πj ≥ 0 for all j ;
πj = 1.
(5.14)
i j Suppose that a set of numbers {πi ; i ≥ 0} satisfying (5.14) is chosen as the initial probability
distribution for a Markov chain, i.e., if Pr {X0 = i} = πi for all i. Then Pr {X1 = j } =
P
i πi Pij = πj for all j , and, by induction, Pr {Xn = j } = πj for all j and all n ≥ 0. The
fact that Pr {Xn = j } = πj for all j motivates the deﬁnition of steadystate distribution 5.1. INTRODUCTION AND CLASSIFICATION OF STATES 205 above. Theorem 5.2 showed that 1/T j j is a ‘steadystate’ probability for state j , both
in a timeaverage and a limiting ensembleaverage sense. The following theorem brings
these ideas together. An irreducible Markov chain is a Markov chain in which all pairs of
states communicate. For ﬁnitestate chains, irreducibility implied a single class of recurrent
states, whereas for countably inﬁnite chains, an irreducible chain is a single class that can
be transient, nullrecurrent, or positiverecurrent.
Theorem 5.4. Assume an irreducible Markov chain with transition probabilities {Pij }. If
(5.14) has a solution, then the solution is unique, πi = 1/T ii > 0 for al l i ≥ 0, and the states
are positiverecurrent. Also, if the states are positiverecurrent then (5.14) has a solution.
Proof*: Let {πj ; j ≥ 0} satisfy (5.14) and be the initial distribution of the Markov chain,
i.e., Pr {X0 =j } = πj , j ≥ 0. Then, as shown above, Pr {Xn =j } = πj for all n ≥ 0, j ≥ 0.
˜
Let Nj (t) be the number of occurrences of any given state j from time 1 to t. Equating
Pr {Xn =j } to the expectation of an occurrence of j at time n, we have,
h
i
X
˜
(1/t)E Nj (t) = (1/t)
Pr {Xn =j } = πj
for all integers t ≥ 1.
1≤n≤t Conditioning this on the possible starting states i, and using the counting processes {Nij (t); t ≥
0} deﬁned earlier,
h
iX
e
πj = (1/t)E Nj (t) =
πi E [Nij (t)/t] for all integer t ≥ 1.
(5.15)
i For any given state i, let Tij be the time of the ﬁrst occurrence of state j given X0 = i.
Then if Tij < 1, we have Nij (t) ≤ Nij (Tij + t). Thus, for all t ≥ 1,
E [Nij (t)] ≤ E [Nij (Tij + t)] = 1 + E [Nj j (t)] . (5.16) The last step follows since the process is in state j at time Tij , and the expected number of
occurrences of state j in the next t steps is E [Nj j (t)].
Substituting (5.16) in (5.15) for each i, πj ≤ 1/t + E [Nj j (t)/t)]. Taking the limit as
P
t → 1 and using (5.12), πj ≤ limt→1 E [Nj j (t)/t]. Since i πi = 1, there is at least one
value of j for which πj > 0, and for this j , limt→1 E [Nj j (t)/t] > 0, and consequently
limt→1 E [Nj j (t)] = 1. Thus, from Lemma 5.1, state j is recurrent, and from Theorem 5.2,
j is positiverecurrent. From Theorem 5.3, all states are then positiverecurrent. For any j
and any integer M , (5.15) implies that
X
πj ≥
πi E [Nij (t)/t] for all t.
(5.17)
i≤M From Theorem 5.2, limt→1 E [Nij (t)/t] = 1/T j j for all i. Subs...
View Full
Document
 Spring '09
 R.Srikant

Click to edit the document details