Markov Chains
Stationary Distribution
Consider a (homogeneous) DTMC whose state is cfw_1, 2, . . . , N . If the
state space is countably infinite, then we take N to be .
Stationary Distribution
Recall: For an ergodic irreducible DTMC the DTMC has a unique

Markov Chains
Classification of States and Potential Theory
Classification Example 1: Consider the (homogeneous) DTMC with
the following transition diagram.
3
0.3
0.05
1
5
2
0.99
0.6 0.7
0.03
0.01
0.92
1
4
0.4
Since there is a path (with non-zero probabil

The Art of Counting
Tree Diagram: If the experiment can be described as a sequence of k steps,
then the sample space can be illustrated with a tree diagram. Any path through
the tree signifies a single outcome in the sample space.
Example 1:
Roll a fair s

Introduction to Probability
Definition : A random experiment is an experiment or a process for
which the outcome cannot be predicted with certainty.
Definition : The sample space (denoted ) of a random experiment is
the set of all possible outcomes.
Examp

The Art of Counting
Tree Diagram: If the experiment can be described as a sequence of k steps,
then the sample space can be illustrated with a tree diagram. Any path through
the tree signifies a single outcome in the sample space.
Example 1:
Roll a fair s

f (x)
f (x)
= 0.
x!0 x
o(x),
lim
x
p
x
o(x)
a 2 R f (x)
f (x)
f (x)
o(x)
g(x)
x2
o(x)
a f (x)
o(x)
f (x) + g(x)
f (x) ! 0
o(1)
o(x)
o(x)
x!0
a0 , a1 , a2 , . . .
f (x) =
1
X
ai xi = a0 + a1 x + a2 x2 + . . . .
i=0
1
R
|x| < R
x = R
R > 0 f (x)
R<x<R
ai
xi

A
-<1|:l
(JEI
r%
E
000
|D>
en
U>0
Induction Hypothesis: Assume that
for?" :16.
Induction Step: We need to show that
771 2'
13(5) >15) =ZBMOJ) for r: k+ 1.
i=0
We have
00 Ak'H miceA1
P(Sk+1 > t) = / de.
t .
Use an integration by parts with 'U. : AkH

1. [10 points] Consider a small store that receives 4 copies of a local
newspaper daily. Suppose that the customers that would like to purchase
the newspaper can be modeled as a Poisson process with a rate of 3
customers per day. We assume that copies of

-<1|:l
A
a:
P%
E
I>
en:
so
000
Suggested Exercises
Continuous Models
. The time waiting in the emergency room of a hospital before one is
called to see a doctor often seems to be exponentially distributed with
mean equal to 60 minutes. Let X be the rand

2. [10 points] Let X have a geometric distribution with parameter 39. Its
probability mass function is
px(r)=(1p)t1p, i=1,2,3,.
(a) Using the above probability mass function, Show that X has the
following z-transform.
GX(z)=1_(f:W IZI <1/(1p).
(b) Use the

3. [10 points] When the production of the transistors is under statistical
quality control about 2% of the transistors are going to be defective.
Suppose that a transistor will be defective independently of the defec-
tive status of the other transistors.

1. [10 points]
(a) We want P(N(1) 2 4) = 1 23:0 cfw_es/51).
b Let X be the number of news a ers sold on a articular da . Its
P l3 p 3
probability mass function is
P(N(1) = 0) = e_3(3/01) J: 0
P(N(1) = 2) = e_3(31/11), :r 2
mm) : Pom) : 2) : 5382/21), x e

Dominated Convergence
These notes concern one important result from mathematical analysis. It
is a special case of Lebesgues Dominated Convergence Theorem from Measure Theory, which is one of the most important results in measure theory.
We will restrict

Mathematical Preliminaries
To build our probability models, we will need some mathematical tools.
Here are a few.
little o notation :
f (x) is o(x),
Remark :
if
f (x)
= 0.
x0 x
lim
It means that the functions converges to zero faster than x.
Example :
x i

Introduction to Markov Chains
Consider the stochastic process:
cfw_X(t) : t T ,
that is a set of random variables.
Terminology:
1. X(t) is a random variable;
2. t is a parameter that often represents time;
3. T is the parameter space;
(a) if T is discrete

Random Variables
Denition :
Let be a sample space. A function X : R, that
associates a real number X(s) to each outcome s is called a random variable.
Notation : The range of the random variable is denoted RX .
Note :
We use upper-case letter (often at th

Poisson Process Part II
Event Epochs and Inter-event times
Now that we have the machinery to work with continuous models, we are
ready to think of the Poisson process in terms of its inter-event times.
Definition: Consider the sequence random variables S1

Poisson Process Part I
Simple Point Process
Definition: Consider the sequence random variables S1 , S2 , S3 , . . . such
that
S1 < S2 < S3 < . . .
We say that the sequence is a simple point process. We say that Si is an
event epoch, that is we interpret i

Introduction to R (statistical computing)
R is a free software environment for statistical computing and graphics.
It compiles and runs on a wide variety of Linux platforms, Windows and
MacOS. Visit the following webpage: http:/www.r-project.org/
These no

Discrete Probability Models
One mathematical tool that will allow us to manipulate models is the
probability generating function (sometime called the z-transform). Let X
taken values on cfw_0, 1, 2, 3, . . . with the corresponding probabilities p0 , p1 ,

Continuous Random Variables
Definition: A random variable X is said to be continuous if its cumulative distribution function FX is an absolutely continuous function. This
means that there exits a function f (x) such that
Z x
f (u) du.
FX (x) =
Remark: Abs

Discrete Probability Models
One mathematical tool that will allow us to manipulate models is the
probability generating function (sometime called the z-transform). Let X
taken values on cfw_0, 1, 2, 3, . . . with the corresponding probabilities p0 , p1 ,

Phase type distributions
Motivation:
We have the exponential distribution that can be used as a model for
time to an event. However, the exponential satisfies the memoryless
property. This means that the exponential is often not a reasonable
model.
In t

Markov Chains
Classification of States and Potential Theory
Classification Example 1: Consider the (homogeneous) DTMC with
the following transition diagram.
3
0.3
0.05
1
5
2
0.99
0.6 0.7
0.03
0.01
0.92
1
4
0.4
Since there is a path (with non-zero probabil

Renewal Theory
Goal for these notes:
We will state the strong law of large numbers.
Define the renewal counting process. It is a generalization of the Poisson counting process, where we will permit the inter-event times to be
non-exponential.
Define a

A
-<1|:l
(JEI
P%
E
000
|D>
E
on
Up
Exercise 5.3.4: Consider a discrete random variable X whose probabil-
ity generating function is given by
GX(2)=ezie+272.
Observe that when 2 = 1, Gx(z) = 1. Find the probabilities px(0), px(1), px(2), px(3),
and px(4)

A
-<1|:l
(JEI
P%
E
|D>
en
Suggested Exercises - Chapter 5
Exercise 5.3.3: Let X be a discrete random variable with probability
mass function given by
1/10, x = 1
2/10, 59 = 2
px(a:) = 3/10, 55' = 3
4/10, 55 = 4
0, otherwise.
Find the probability generat