{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

Discrete-time stochastic processes - DISCRETE STOCHASTIC...

Info icon This preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
DISCRETE STOCHASTIC PROCESSES Draft of 2nd Edition R. G. Gallager August 30, 2009 i
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Contents 1 INTRODUCTION AND REVIEW OF PROBABILITY 1 1.1 Probability models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 The sample space of a probability model . . . . . . . . . . . . . . . . 3 1.1.2 Assigning probabilities for finite sample spaces . . . . . . . . . . . . 4 1.2 The axioms of probability theory . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.1 Axioms for the class of events for a sample space : . . . . . . . . . 6 1.2.2 Axioms of probability . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Probability review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.1 Conditional probabilities and statistical independence . . . . . . . . 8 1.3.2 Repeated idealized experiments . . . . . . . . . . . . . . . . . . . . . 10 1.3.3 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3.4 Multiple random variables and conditional probabilities . . . . . . . 12 1.3.5 Stochastic processes . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3.6 Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.7 Random variables as functions of other random variables . . . . . . 19 1.3.8 Conditional expectations . . . . . . . . . . . . . . . . . . . . . . . . 22 1.3.9 Indicator random variables . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.10 Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.4 The laws of large numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.1 Basic inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 1.4.2 Weak law of large numbers with a finite variance . . . . . . . . . . . 28 1.4.3 Relative frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.4.4 The central limit theorem . . . . . . . . . . . . . . . . . . . . . . . . 31 ii
Image of page 2
CONTENTS iii 1.4.5 Weak law with an infinite variance . . . . . . . . . . . . . . . . . . . 33 1.4.6 Strong law of large numbers (SLLN) . . . . . . . . . . . . . . . . . . 34 1.4.7 Convergence of random variables . . . . . . . . . . . . . . . . . . . . 39 1.5 Relation of probability models to the real world . . . . . . . . . . . . . . . . 42 1.5.1 Relative frequencies in a probability model . . . . . . . . . . . . . . 43 1.5.2 Relative frequencies in the real world . . . . . . . . . . . . . . . . . . 43 1.5.3 Statistical independence of real-world experiments . . . . . . . . . 46 1.5.4 Limitations of relative frequencies . . . . . . . . . . . . . . . . . . . 46 1.5.5 Subjective probability . . . . . . . . . . . . . . . . . . . . . . . . . . 47 1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 1.7 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 1.7.1 Table of standard random variables . . . . . . . . . . . . . . . . . . . 49 1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2 POISSON PROCESSES 58 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.1.1 Arrival processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.2 Definition and properties of the Poisson process . . . . . . . . . . . . . . . . 60 2.2.1 Memoryless property . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.2.2 Probability density of S n and S 1 , . . . S n . . . . . . . . . . . . . . . . 64 2.2.3 The PMF for N ( t ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 2.2.4 Alternate definitions of Poisson processes . . . . . . . . . . . . . . . 67 2.2.5 The Poisson process as a limit of shrinking Bernoulli processes . . . 69 2.3 Combining and splitting Poisson processes . . . . . . . . . . . . . . . . . . . 70 2.3.1 Subdividing a Poisson process . . . . . . . . . . . . . . . . . . . . . . 72 2.3.2 Examples using independent Poisson processes . . . . . . . . . . . . 73 2.4 Non-homogeneous Poisson processes . . . . . . . . . . . . . . . . . . . . . . 74 2.5 Conditional arrival densities and order statistics . . . . . . . . . . . . . . . . 77 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 2.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3 RENEWAL PROCESSES 92 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Image of page 3

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
iv CONTENTS 3.2 Strong Law of Large Numbers for renewal processes . . . . . . . . . . . . . 93 3.3 Expected number of renewals . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.3.1 Laplace transform approach . . . . . . . . . . . . . . . . . . . . . . . 98 3.3.2 Random stopping times . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.3.3 Wald’s equality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.4 Renewal-reward processes; time-averages . . . . . . . . . . . . . . . . . . . . 105 3.4.1 General renewal-reward processes . . . . . . . . . . . . . . . . . . . . 108 3.5 Renewal-reward processes; ensemble-averages . . . . . . . . . . . . . . . . . 112 3.6 Applications of renewal-reward theory . . . . . . . . . . . . . . . . . . . . . 117 3.6.1 Little’s theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 3.6.2 Expected queueing time for an M/G/1 queue . . . . . . . . . . . . . 120 3.7 Delayed renewal processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 3.7.1 Delayed renewal-reward processes . . . . . . . . . . . . . . . . . . . . 124 3.7.2 Transient behavior of delayed renewal processes . . . . . . . . . . . . 125 3.7.3 The equilibrium process . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 3.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 4 FINITE-STATE MARKOV CHAINS 139 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4.2 Classification of states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 4.3 The Matrix representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.3.1 The eigenvalues and eigenvectors of P . . . . . . . . . . . . . . . . . 149 4.4 Perron-Frobenius theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.5 Markov chains with rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 4.6 Markov decision theory and dynamic programming . . . . . . . . . . . . . . 165 4.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 4.6.2 Dynamic programming algorithm . . . . . . . . . . . . . . . . . . . . 167 4.6.3
Image of page 4
Image of page 5
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern