Copyright c 2012 by Karl Sigman
1
Notes on the Poisson Process
We present here the essentials of the Poisson point process with its many interesting
properties. As preliminaries, we rst dene what a point process is, dene the renewal
point process and stat
Copyright c 2012 by Karl Sigman
1
Limiting distribution for a Markov chain
In these Lecture Notes, we shall study the limiting behavior of Markov chains as time n .
In particular, under suitable easy-to-check conditions, we will see that a Markov chain po
IEOR 4106, HMWK 2, Professor Sigman
1. Consider modeling the weather where we now assume that the weather today depends (at most) on the previous three days weather (recall the Text, Page 193
Example 4.4, where the weather was assumed to depend upon the p
IEOR 4106, HMWK 3, Professor Sigman
1. Each of the following transition matrices is for a Markov chain. For each, nd the communication classes for breaking down the state space, S = C1 C2 and for each class
Ci tell if it is recurrent or transient.
(a)
1/6
IEOR 4106, HMWK 4, Professor Sigman
1. End of Text Chapter 4, Page 281: Exercises 41 and 42.
2. Consider a Markov chain with nite state space S = cfw_1, 2, . . . , b, and transition matrix
P . Suppose that it is irreducible, and that each column of P sums
IEOR 4106, HMWK 5, Professor Sigman
1. End of Text Chapter 5, Page 354: Exercises 4,5, 18
2. You arrive at the West 96th Street Subway station to go Downtown. Suppose that the
time until the next Express train is exponentially distributed with rate 3 (per
IEOR 4106, HMWK 6, Professor Sigman
1. Suppose that the rat in the closed maze visits Room 4 at times cfw_tn : n 1 that form a
Poisson process at rate . You are not allowed to watch the rat, but each time it enters
Room 4 you are immediately told so. You
IEOR 4106, HMWK 7, Professor Sigman
1. Printer with jams: Jobs arrive to a computer printer according to a Poisson process at
rate . Jobs are printed one at a time requiring iid printing times that are exponentially
distributed with rate . Jobs wait in a
IEOR 4106, HMWK 8, Professor Sigman
1. Recall Problem 2 from HMWK 7:
Consider 4 iPhones, each independently having a battery lifetime that is exponentially
distributed with mean 2 years (hence rate = 0.5). Once a battery breaks down, the
iPhone immediatel
Copyright c 2012 by Karl Sigman
1
Gamblers Ruin Problem
Let N 2 be an integer and let 1 i N 1. Consider a gambler who starts with an
initial fortune of $i and then on each successive gamble either wins $1 or loses $1 independent
of the past with probabili
Copyright c 2012 by Karl Sigman
1
Review of the exponential distribution
The exponential distribution has many nice properties; we review them here.
A r.v. X has an exponential distribution at rate , denoted by X exp(), if X is nonnegative with c.d.f. F (
Copyright c 2012 by Karl Sigman
1
Continuous-Time Markov Chains
A Markov chain in discrete time, cfw_Xn : n 0, remains in any state for exactly one
unit of time before making a transition (change of state). We proceed now to relax this
restriction by allo
Copyright c 2012 by Karl Sigman
1
Introduction to Renewal Theory
Here, we will present some basic results in renewal theory such as the elementary renewal
theorem and the inspection paradox (Section 1), and the renewal reward theorem (Section 2).
Our emph
Copyright c 2012 by Karl Sigman
1
Stopping Times
1.1
Stopping Times: Denition
Given a stochastic process X = cfw_Xn : n 0, a random time is a discrete random variable
on the same probability space as X, taking values in the time set N = cfw_0, 1, 2, . . .
Copyright c 2012 by Karl Sigman
1
Expected number of visits of a nite state Markov chain to
a transient state
When a Markov chain is not positive recurrent, hence does not have a limiting stationary
distribution , there are still other very important and
IEOR 4106, Practice Midterm Exam. Professor Sigman.
1. Let B be a rv with probability mass function: P (B = 2) = 0.2, P (B = 5) = 0.8. Consider
a reservoir lled with water that changes its water level as follows: Let Xn denote the
water level (in units) o
Copyright c 2012 by Karl Sigman
1
Notes on Brownian Motion
We present an introduction to Brownian motion, an important continuous-time stochastic process that serves as a continuous-time analog to the simple symmetric random walk on the one
hand, and shar
IEOR 4106, HMWK 9, Professor Sigman
1. For a renewal process with iid interarrival times cfw_Xn with E(X) = 1/, give an
expression for
1 t 2
lim
A (s)ds,
t t 0
that involves only the moments, E(X n ), n 1.
To do so: (1) Graph cfw_A2 (t) : t 0, and look a