Copyright c 2012 by Karl Sigman
1
Notes on the Poisson Process
We present here the essentials of the Poisson point process with its many interesting
properties. As preliminaries, we rst dene what a po
Copyright c 2012 by Karl Sigman
1
Limiting distribution for a Markov chain
In these Lecture Notes, we shall study the limiting behavior of Markov chains as time n .
In particular, under suitable easy-
IEOR 4106, HMWK 2, Professor Sigman
1. Consider modeling the weather where we now assume that the weather today depends (at most) on the previous three days weather (recall the Text, Page 193
Example
IEOR 4106, HMWK 3, Professor Sigman
1. Each of the following transition matrices is for a Markov chain. For each, nd the communication classes for breaking down the state space, S = C1 C2 and for each
IEOR 4106, HMWK 4, Professor Sigman
1. End of Text Chapter 4, Page 281: Exercises 41 and 42.
2. Consider a Markov chain with nite state space S = cfw_1, 2, . . . , b, and transition matrix
P . Suppose
IEOR 4106, HMWK 5, Professor Sigman
1. End of Text Chapter 5, Page 354: Exercises 4,5, 18
2. You arrive at the West 96th Street Subway station to go Downtown. Suppose that the
time until the next Expr
IEOR 4106, HMWK 6, Professor Sigman
1. Suppose that the rat in the closed maze visits Room 4 at times cfw_tn : n 1 that form a
Poisson process at rate . You are not allowed to watch the rat, but each
IEOR 4106, HMWK 7, Professor Sigman
1. Printer with jams: Jobs arrive to a computer printer according to a Poisson process at
rate . Jobs are printed one at a time requiring iid printing times that ar
IEOR 4106, HMWK 8, Professor Sigman
1. Recall Problem 2 from HMWK 7:
Consider 4 iPhones, each independently having a battery lifetime that is exponentially
distributed with mean 2 years (hence rate =
Copyright c 2012 by Karl Sigman
1
Gamblers Ruin Problem
Let N 2 be an integer and let 1 i N 1. Consider a gambler who starts with an
initial fortune of $i and then on each successive gamble either win
Copyright c 2012 by Karl Sigman
1
Review of the exponential distribution
The exponential distribution has many nice properties; we review them here.
A r.v. X has an exponential distribution at rate ,
Copyright c 2012 by Karl Sigman
1
Continuous-Time Markov Chains
A Markov chain in discrete time, cfw_Xn : n 0, remains in any state for exactly one
unit of time before making a transition (change of s
Copyright c 2012 by Karl Sigman
1
Introduction to Renewal Theory
Here, we will present some basic results in renewal theory such as the elementary renewal
theorem and the inspection paradox (Section 1
Copyright c 2012 by Karl Sigman
1
Stopping Times
1.1
Stopping Times: Denition
Given a stochastic process X = cfw_Xn : n 0, a random time is a discrete random variable
on the same probability space as
Copyright c 2012 by Karl Sigman
1
Expected number of visits of a nite state Markov chain to
a transient state
When a Markov chain is not positive recurrent, hence does not have a limiting stationary
d
IEOR 4106, Practice Midterm Exam. Professor Sigman.
1. Let B be a rv with probability mass function: P (B = 2) = 0.2, P (B = 5) = 0.8. Consider
a reservoir lled with water that changes its water level
Copyright c 2012 by Karl Sigman
1
Notes on Brownian Motion
We present an introduction to Brownian motion, an important continuous-time stochastic process that serves as a continuous-time analog to the
IEOR 4106, HMWK 9, Professor Sigman
1. For a renewal process with iid interarrival times cfw_Xn with E(X) = 1/, give an
expression for
1 t 2
lim
A (s)ds,
t t 0
that involves only the moments, E(X n )