Chapt 10d - 10.7 Discrete-Time Markov Chains 363 Figure...

Info icon This preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
10.7 Discrete-Time Markov Chains 363 Figure 10.11 State-Transition Diagram for Example 10.10 10.7.4 Classification of States A state j is said to be accessible (or can be reached ) from state i if, starting from state i , it is possible that the process will ever enter state j . This implies that p ij ( n ) > 0 for some n > 0. Thus, the n -step probability enables us to obtain reach- ability information between any two states of the process. Two states that are accessible from each other are said to communicate with each other. The concept of communication divides the state space into different classes. Two states that communicate are said to be in the same class . All members of one class communicate with one another. If a class is not accessible from any state outside the class, we define the class to be a closed communicating class . A Markov chain in which all states communicate, which means that there is only one class, is called an irreducible Markov chain. For example, the Markov chains shown in Figures 10.10 and 10.11 are irreducible chains. The states of a Markov chain can be classified into two broad groups: those that the process enters infinitely often and those that it enters finitely often. In the long run, the process will be found to be in only those states that it enters infinitely often. Let f ij ( n ) denote the conditional probability that given that the process is presently in state i , the first time it will enter state j occurs in exactly n transitions (or steps). We call f ij ( n ) the probability of first passage from state i to state j in n transitions. The parameter f ij , which is defined as follows: f ij = summationdisplay n = 1 f ij ( n ) is the probability of first passage from state i to state j . It is the conditional prob- ability that the process will ever enter state j , given that it was initially in state i .
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
364 Chapter 10 Some Models of Random Processes Obviously f ij ( 1 ) = p ij and a recursive method of computing f ij ( n ) is f ij ( n ) = summationdisplay l negationslash= j p il f lj ( n 1 ) The quantity f ii denotes the probability that a process that starts at state i will ever return to state i . Any state i for which f ii = 1 is called a recurrent state , and any state i for which f ii < 1 is called a transient state . More formally, we define these states as follows: a. A state j is called a transient (or nonrecurrent ) state if there is a positive prob- ability that the process will never return to j again after it leaves j . b. A state j is called a recurrent (or persistent ) state if, with probability 1, the process will eventually return to j after it leaves j . A set of recurrent states forms a single chain if every member of the set communicates with all other members of the set. c. A recurrent state j is called a periodic state if there exists an integer d , d > 1, such that p jj ( n ) is zero for all values of n other than d , 2 d , 3 d ,... ; d is called the period. If d = 1, the recurrent state j is said to be aperiodic .
Image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern