Discrete-time stochastic processes

Discrete-time stochastic processes - DISCRETE STOCHASTIC...

Info iconThis preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
DISCRETE STOCHASTIC PROCESSES Draft of 2nd Edition R. G. Gallager August 30, 2009 i
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Contents 1 INTRODUCTION AND REVIEW OF PROBABILITY 1 1.1 Probability models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 The sample space of a probability model . . . . . . . . . . . . . . . . 3 1.1.2 Assigning probabilities for finite sample spaces . . . . . . . . . . . . 4 1.2 The axioms of probability theory . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.1 Axioms for the class of events for a sample space ≠: . . . . . . . . . 6 1.2.2 Axioms of probability . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Probability review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.1 Conditional probabilities and statistical independence . . . . . . . . 8 1.3.2 Repeated idealized experiments . . . . . . . . . . . . . . . . . . . . . 10 1.3.3 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3.4 Multiple random variables and conditional probabilities . . . . . . . 12 1.3.5 Stochastic processes . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3.6 Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.7 Random variables as functions of other random variables . . . . . . 19 1.3.8 Conditional expectations . . . . . . . . . . . . . . . . . . . . . . . . 22 1.3.9 Indicator random variables . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.10 Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.4 The laws of large numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.1 Basic inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 1.4.2 Weak law of large numbers with a finite variance . . . . . . . . . . . 28 1.4.3 Relative frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.4.4 The central limit theorem . . . . . . . . . . . . . . . . . . . . . . . . 31 ii
Background image of page 2
CONTENTS iii 1.4.5 Weak law with an infinite variance . . . . . . . . . . . . . . . . . . . 33 1.4.6 Strong law of large numbers (SLLN) . . . . . . . . . . . . . . . . . . 34 1.4.7 Convergence of random variables . . . . . . . . . . . . . . . . . . . . 39 1.5 Relation of probability models to the real world . . . . . . . . . . . . . . . . 42 1.5.1 Relative frequencies in a probability model . . . . . . . . . . . . . . 43 1.5.2 Relative frequencies in the real world . . . . . . . . . . . . . . . . . . 43 1.5.3 Statistical independence of real-world experiments . . . . . . . . . 46 1.5.4 Limitations of relative frequencies . . . . . . . . . . . . . . . . . . . 46 1.5.5 Subjective probability . . . . . . . . . . . . . . . . . . . . . . . . . . 47 1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 1.7 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 1.7.1 Table of standard random variables . . . . . . . . . . . . . . . . . . . 49 1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2 POISSON PROCESSES 58 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.1.1 Arrival processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.2 Definition and properties of the Poisson process . . . . . . . . . . . . . . . . 60 2.2.1 Memoryless property . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.2.2 Probability density of S n and S 1 , . . . S n . . . . . . . . . . . . . . . . 64 2.2.3 The PMF for N ( t ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 2.2.4 Alternate definitions of Poisson processes . . . . . . . . . . . . . . . 67 2.2.5 The Poisson process as a limit of shrinking Bernoulli processes . . . 69 2.3 Combining and splitting Poisson processes . . . . . . . . . . . . . . . . . . . 70 2.3.1 Subdividing a Poisson process . . . . . . . . . . . . . . . . . . . . . . 72 2.3.2 Examples using independent Poisson processes . . . . . . . . . . . . 73 2.4 Non-homogeneous Poisson processes . . . . . . . . . . . . . . . . . . . . . . 74 2.5 Conditional arrival densities and order statistics . . . . . . . . . . . . . . . . 77 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 2.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3 RENEWAL PROCESSES 92 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
iv CONTENTS 3.2 Strong Law of Large Numbers for renewal processes . . . . . . . . . . . . . 93 3.3 Expected number of renewals . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.3.1 Laplace transform approach . . . . . . . . . . . . . . . . . . . . . . . 98 3.3.2 Random stopping times . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.3.3 Wald’s equality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.4 Renewal-reward processes; time-averages . . . . . . . . . . . . . . . . . . . . 105 3.4.1 General renewal-reward processes . . . . . . . . . . . . . . . . . . . . 108 3.5 Renewal-reward processes; ensemble-averages . . . . . . . . . . . . . . . . . 112 3.6 Applications of renewal-reward theory . . . . . . . . . . . . . . . . . . . . . 117 3.6.1 Little’s theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 3.6.2 Expected queueing time for an M/G/1 queue . . . . . . . . . . . . . 120 3.7 Delayed renewal processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 3.7.1 Delayed renewal-reward processes . . . . . . . . . . . . . . . . . . . . 124 3.7.2 Transient behavior of delayed renewal processes . . . . . . . . . . . . 125 3.7.3 The equilibrium process . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 3.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 4 FINITE-STATE MARKOV CHAINS 139 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4.2 Classification of states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 4.3 The Matrix representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.3.1 The eigenvalues and eigenvectors of P . . . . . . . . . . . . . . . . . 149 4.4 Perron-Frobenius theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.5 Markov chains with rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 4.6 Markov decision theory and dynamic programming . . . . . . . . . . . . . . 165 4.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 4.6.2 Dynamic programming algorithm . . . . . . . . . . . . . . . . . . . . 167 4.6.3
Background image of page 4
Image of page 5
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

Page1 / 334

Discrete-time stochastic processes - DISCRETE STOCHASTIC...

This preview shows document pages 1 - 5. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online