This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: 10.7 Discrete-Time Markov Chains 363 Figure 10.11 State-Transition Diagram for Example 10.10 10.7.4 Classification of States A state j is said to be accessible (or can be reached ) from state i if, starting from state i , it is possible that the process will ever enter state j . This implies that p ij ( n ) > 0 for some n > 0. Thus, the n-step probability enables us to obtain reach- ability information between any two states of the process. Two states that are accessible from each other are said to communicate with each other. The concept of communication divides the state space into different classes. Two states that communicate are said to be in the same class . All members of one class communicate with one another. If a class is not accessible from any state outside the class, we define the class to be a closed communicating class . A Markov chain in which all states communicate, which means that there is only one class, is called an irreducible Markov chain. For example, the Markov chains shown in Figures 10.10 and 10.11 are irreducible chains. The states of a Markov chain can be classified into two broad groups: those that the process enters infinitely often and those that it enters finitely often. In the long run, the process will be found to be in only those states that it enters infinitely often. Let f ij ( n ) denote the conditional probability that given that the process is presently in state i , the first time it will enter state j occurs in exactly n transitions (or steps). We call f ij ( n ) the probability of first passage from state i to state j in n transitions. The parameter f ij , which is defined as follows: f ij = ∞ summationdisplay n = 1 f ij ( n ) is the probability of first passage from state i to state j . It is the conditional prob- ability that the process will ever enter state j , given that it was initially in state i . 364 Chapter 10 Some Models of Random Processes Obviously f ij ( 1 ) = p ij and a recursive method of computing f ij ( n ) is f ij ( n ) = summationdisplay l negationslash= j p il f lj ( n − 1 ) The quantity f ii denotes the probability that a process that starts at state i will ever return to state i . Any state i for which f ii = 1 is called a recurrent state , and any state i for which f ii < 1 is called a transient state . More formally, we define these states as follows: a. A state j is called a transient (or nonrecurrent ) state if there is a positive prob- ability that the process will never return to j again after it leaves j . b. A state j is called a recurrent (or persistent ) state if, with probability 1, the process will eventually return to j after it leaves j . A set of recurrent states forms a single chain if every member of the set communicates with all other members of the set....
View Full Document
This note was uploaded on 01/05/2010 for the course STAT 350 taught by Professor Carlton during the Fall '07 term at Cal Poly.
- Fall '07
- Markov Chains