Math 416 Lecture 4
Due Feb. 14: a project proposal which lists the chosen
topic and references to be used. The place to look for project
topics is in the library and on the internet (Google) for topics
like genetic algorithms, genetic programming,neural
Math 416 Lecture 15
p1 p p p 2 is maximal
iff its derivative 1 2p 0
iff p 1/2.
When p 1/2, p1 p 1/21/2 1/4 its max.
When p 1/2, p1 p its max.
t p1 p 1/4,
t4p1 p 1.
STIRLINGS APPROXIMATION: n!
22n 2n 2n /e 2n
2n n n /e n 2
2n n n /e n . T
Math 416 Lecture 19
Recall from last time. Poisson Arrival Queues
Suppose the queue has type Poisson/G(x) /5.
Thus the arrivals are Poisson with some rate .
There are infinitely many lines a customer can choose from
(practically speaking, this just means
Math 416 Lecture 24
Brownian motion continued
From last time:
A random walk X t , t 0, is a Brownian motion with
standard deviation s if
v X 0 0 and f t X t is continuous with probability 1.
v X ts X t , X t s X t , X s all have the same distribution (the
Math 416 Lecture 17
Project Hw due Mar. 29. Exam 3 Mar. 31, Lectures 13-18.
Math 416 Lecture 25
Markov decision processes
Decision processes involve real-time decisions which
affect the outcomes (e.g. instead of just predicting monthly sales, we
Math 416 Lecture 14
Poisson / G(x) / 1 Queues
Suppose the arrivals are a Poisson process with arrival rate
l and expected time 1/l between arrivals and with N t = the
number of arrivals at time t.
Suppose the service times are mutually independent and
Math 416 Lecture 18
Finite Horizon problem -continued
RECURSION EQUATION (DYNAMIC PROGRAMMING EQUATION) FOR
V m1 i. Given optimal V m j with optimal policies v*:
Vm1 i maxaAri, a j Ta i, jVm j
If a gives the max value, let u* be the policy such that
Math 416 Lecture 24
Two-person zero-sum games
A game is played for money between two players: player I
and player II. It is a zero-sum game if I s winnings + IIs
winnings = 0, i.e., what I wins, II loses.
Each player has a finite number of ways or strateg
Math 416 Lecture 22
No, Homework 22. Initial project presentation this week.
A Markov chain of the following form is called a random
walk with reflecting barriers.
This is a rough model of a volatile stock which
Math 416 Lecture 28
Infinite Horizon discounted reward problem
From the last lecture:
The value function of policy u for the infinite horizon
problem with discount factor a and initial state i is
Wi, u E n0 n rX n , uX n |X 0 i.
The optimal value function
Math 416 Lecture 9
Consider T (i, j).
Case j is transient. For any state i, T (i, j) = 0 (see Lecture 10).
Case j is recurrent.
Subcase i and j are in the same recurrence class and the
class matrix is regular. By the previous lecture
T (i, j) = T (j, j) =
Math 416 Lecture 11
Math 416 Lecture 16
Exam 2 next time
Birth and death processes, queueing theory
In arrival processes, the state only jumps up.
In a birth-death process, it can either jump up or down by
A birth-death process counts the number
Math 416 Lecture 7
`A professors computer is replaced when it dies or in its
fifth year, which ever comes first. Let the state X of a
computer be the number of years until it is replaced. The
state space is cfw_1, 2, 3, 4. Here are the replacement times a
Math 416 Lecture 2
DEFINITION. Here are the multivariate versions:
PMF case: p(x, y, z) is the joint Probability Mass Function of
X, Y, Z iff
P(X = x, Y = y, Z =z) = p(x , y, z)
PDF case: f(x, y, z) is the joint Probability Density Function
of X, Y, Z iff
Math 416 Lecture 1
20% of you grade is determined by a class project. Pick an
Operations Research problem and a method of solution. You
get one or two class lectures to present the problem and
solution. You get to pick the presentation day on a fi
Math 416 Lecture 3
The average or mean or expected value of x 1 , x 2 , x 3 , ., x n
x 1 x 2 .x n
x1 1 x2 1 . xn 1 1 x i px i
where px i 1 is the probability of x i assuming the n
numbers are equally likely. If the x i s a
Math 416 Lecture 6
Exam 1 Thursday Feb. 3, covers Lectures 1-6, not Lecture 7.
For a transition matrix T, the n-step transition probabilities
are given by T n. The limit the system approaches as time
goes to 5 is T = lim nd T n.
Diagonalization makes this
Math 416 Lecture 8 Project proposal due Wednesday.
THEOREM. If T is a regular matrix for an irreducible space or
a submatrix for a recurrence class, then:
v The rows of T are identical.
Let p be the common row. Its j th entry = the proportion j
of time in
Math 416 Lecture 7
For a random Markov chain, in the limit as time goes to
infinity, a state is either visited infinitely often, or is visited
only finitely often, in which case, after some final visit, it is
not visited again.
Suppose state x is visited
Math 416 Lecture 10
Project proposal due Today.
THEOREM. If N t is the number of events up to time t and if
the nth event arrives at time T n :
e $t ($t) n
For a given t,
P[N t = n] = n! ,
E[N t ] = $t.
For a given n, P[N t < n] = P[T n > t] = k=0 e $t
Math 416 Lecture 12
Exam 2, March 1, covers Lectures 6-12
Under rather general regularity conditions, in the limit, as
t d , the population probability distribution approaches a
stable distribution p j . In a stable distribution the population
Math 416 Lecture 31
STATE INDEPENDENT STOPPING PROBLEMS.
In each round of a game, a die is rolled.
If the roll is a 1, the game stops and you get nothing.
Otherwise you may choose to
continue (and collect nothing at this time) or
stop and collect $1 for e