{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

Discrete-time stochastic processes

# Discrete-time stochastic processes - DISCRETE STOCHASTIC...

This preview shows pages 1–5. Sign up to view the full content.

DISCRETE STOCHASTIC PROCESSES Draft of 2nd Edition R. G. Gallager August 30, 2009 i

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Contents 1 INTRODUCTION AND REVIEW OF PROBABILITY 1 1.1 Probability models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1.1 The sample space of a probability model . . . . . . . . . . . . . . . . 3 1.1.2 Assigning probabilities for finite sample spaces . . . . . . . . . . . . 4 1.2 The axioms of probability theory . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2.1 Axioms for the class of events for a sample space : . . . . . . . . . 6 1.2.2 Axioms of probability . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3 Probability review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3.1 Conditional probabilities and statistical independence . . . . . . . . 8 1.3.2 Repeated idealized experiments . . . . . . . . . . . . . . . . . . . . . 10 1.3.3 Random variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.3.4 Multiple random variables and conditional probabilities . . . . . . . 12 1.3.5 Stochastic processes . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3.6 Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.7 Random variables as functions of other random variables . . . . . . 19 1.3.8 Conditional expectations . . . . . . . . . . . . . . . . . . . . . . . . 22 1.3.9 Indicator random variables . . . . . . . . . . . . . . . . . . . . . . . 24 1.3.10 Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.4 The laws of large numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.4.1 Basic inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 1.4.2 Weak law of large numbers with a finite variance . . . . . . . . . . . 28 1.4.3 Relative frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.4.4 The central limit theorem . . . . . . . . . . . . . . . . . . . . . . . . 31 ii
CONTENTS iii 1.4.5 Weak law with an infinite variance . . . . . . . . . . . . . . . . . . . 33 1.4.6 Strong law of large numbers (SLLN) . . . . . . . . . . . . . . . . . . 34 1.4.7 Convergence of random variables . . . . . . . . . . . . . . . . . . . . 39 1.5 Relation of probability models to the real world . . . . . . . . . . . . . . . . 42 1.5.1 Relative frequencies in a probability model . . . . . . . . . . . . . . 43 1.5.2 Relative frequencies in the real world . . . . . . . . . . . . . . . . . . 43 1.5.3 Statistical independence of real-world experiments . . . . . . . . . 46 1.5.4 Limitations of relative frequencies . . . . . . . . . . . . . . . . . . . 46 1.5.5 Subjective probability . . . . . . . . . . . . . . . . . . . . . . . . . . 47 1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 1.7 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 1.7.1 Table of standard random variables . . . . . . . . . . . . . . . . . . . 49 1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 2 POISSON PROCESSES 58 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.1.1 Arrival processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.2 Definition and properties of the Poisson process . . . . . . . . . . . . . . . . 60 2.2.1 Memoryless property . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 2.2.2 Probability density of S n and S 1 , . . . S n . . . . . . . . . . . . . . . . 64 2.2.3 The PMF for N ( t ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 2.2.4 Alternate definitions of Poisson processes . . . . . . . . . . . . . . . 67 2.2.5 The Poisson process as a limit of shrinking Bernoulli processes . . . 69 2.3 Combining and splitting Poisson processes . . . . . . . . . . . . . . . . . . . 70 2.3.1 Subdividing a Poisson process . . . . . . . . . . . . . . . . . . . . . . 72 2.3.2 Examples using independent Poisson processes . . . . . . . . . . . . 73 2.4 Non-homogeneous Poisson processes . . . . . . . . . . . . . . . . . . . . . . 74 2.5 Conditional arrival densities and order statistics . . . . . . . . . . . . . . . . 77 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 2.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3 RENEWAL PROCESSES 92 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
iv CONTENTS 3.2 Strong Law of Large Numbers for renewal processes . . . . . . . . . . . . . 93 3.3 Expected number of renewals . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.3.1 Laplace transform approach . . . . . . . . . . . . . . . . . . . . . . . 98 3.3.2 Random stopping times . . . . . . . . . . . . . . . . . . . . . . . . . 100 3.3.3 Wald’s equality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 3.4 Renewal-reward processes; time-averages . . . . . . . . . . . . . . . . . . . . 105 3.4.1 General renewal-reward processes . . . . . . . . . . . . . . . . . . . . 108 3.5 Renewal-reward processes; ensemble-averages . . . . . . . . . . . . . . . . . 112 3.6 Applications of renewal-reward theory . . . . . . . . . . . . . . . . . . . . . 117 3.6.1 Little’s theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 3.6.2 Expected queueing time for an M/G/1 queue . . . . . . . . . . . . . 120 3.7 Delayed renewal processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 3.7.1 Delayed renewal-reward processes . . . . . . . . . . . . . . . . . . . . 124 3.7.2 Transient behavior of delayed renewal processes . . . . . . . . . . . . 125 3.7.3 The equilibrium process . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 3.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 4 FINITE-STATE MARKOV CHAINS 139 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4.2 Classification of states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 4.3 The Matrix representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.3.1 The eigenvalues and eigenvectors of P . . . . . . . . . . . . . . . . . 149 4.4 Perron-Frobenius theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.5 Markov chains with rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 4.6 Markov decision theory and dynamic programming . . . . . . . . . . . . . . 165 4.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 4.6.2 Dynamic programming algorithm . . . . . . . . . . . . . . . . . . . . 167 4.6.3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 334

Discrete-time stochastic processes - DISCRETE STOCHASTIC...

This preview shows document pages 1 - 5. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online