Math 416 Lecture 4
Due Feb. 14: a project proposal which lists the chosen
topic and references to be used. The place to look for project
topics is in the library and on the internet (Google) for topic
Math 416 Lecture 15
p1 p p p 2 is maximal
iff its derivative 1 2p 0
iff p 1/2.
When p 1/2, p1 p 1/21/2 1/4 its max.
When p 1/2, p1 p its max.
t p1 p 1/4,
t4p1 p 1.
STIRLINGS APPROXIMATION: n!
2n
n
Math 416 Lecture 19
Recall from last time. Poisson Arrival Queues
Suppose the queue has type Poisson/G(x) /5.
Thus the arrivals are Poisson with some rate .
There are infinitely many lines a customer
Math 416 Lecture 24
Brownian motion continued
From last time:
A random walk X t , t 0, is a Brownian motion with
standard deviation s if
v X 0 0 and f t X t is continuous with probability 1.
v X ts X
Math 416 Lecture 17
Project Hw due Mar. 29. Exam 3 Mar. 31, Lectures 13-18.
Math 416 Lecture 25
Markov decision processes
Decision processes involve real-time decisions which
affect the outcomes (e.g.
Math 416 Lecture 14
Poisson / G(x) / 1 Queues
Suppose the arrivals are a Poisson process with arrival rate
l and expected time 1/l between arrivals and with N t = the
number of arrivals at time t.
Sup
Math 416 Lecture 18
Finite Horizon problem -continued
RECURSION EQUATION (DYNAMIC PROGRAMMING EQUATION) FOR
V m1 i. Given optimal V m j with optimal policies v*:
Vm1 i maxaAri, a j Ta i, jVm j
If a gi
Math 416 Lecture 24
Two-person zero-sum games
A game is played for money between two players: player I
and player II. It is a zero-sum game if I s winnings + IIs
winnings = 0, i.e., what I wins, II lo
Math 416 Lecture 22
No, Homework 22. Initial project presentation this week.
Portfolios
A Markov chain of the following form is called a random
walk with reflecting barriers.
1-p
0
1-p
1
1-p
2
1
.
N
1
Math 416 Lecture 28
Infinite Horizon discounted reward problem
From the last lecture:
The value function of policy u for the infinite horizon
problem with discount factor a and initial state i is
Wi,
Math 416 Lecture 9
Consider T (i, j).
Case j is transient. For any state i, T (i, j) = 0 (see Lecture 10).
Case j is recurrent.
Subcase i and j are in the same recurrence class and the
class matrix is
Math 416 Lecture 11
Math 416 Lecture 16
Exam 2 next time
Birth and death processes, queueing theory
In arrival processes, the state only jumps up.
In a birth-death process, it can either jump up or do
Math 416 Lecture 7
`A professors computer is replaced when it dies or in its
fifth year, which ever comes first. Let the state X of a
computer be the number of years until it is replaced. The
state sp
Math 416 Lecture 2
DEFINITION. Here are the multivariate versions:
PMF case: p(x, y, z) is the joint Probability Mass Function of
X, Y, Z iff
P(X = x, Y = y, Z =z) = p(x , y, z)
PDF case: f(x, y, z) i
Math 416 Lecture 1
Project
20% of you grade is determined by a class project. Pick an
Operations Research problem and a method of solution. You
get one or two class lectures to present the problem and
Math 416 Lecture 3
Expected values
The average or mean or expected value of x 1 , x 2 , x 3 , ., x n
is
x 1 x 2 .x n
n
x1 1 x2 1 . xn 1 1 x i px i
n
n
n
n
where px i 1 is the probability of x i assu
Math 416 Lecture 6
Exam 1 Thursday Feb. 3, covers Lectures 1-6, not Lecture 7.
For a transition matrix T, the n-step transition probabilities
are given by T n. The limit the system approaches as time
Math 416 Lecture 8 Project proposal due Wednesday.
THEOREM. If T is a regular matrix for an irreducible space or
a submatrix for a recurrence class, then:
v The rows of T are identical.
Let p be the c
Math 416 Lecture 7
For a random Markov chain, in the limit as time goes to
infinity, a state is either visited infinitely often, or is visited
only finitely often, in which case, after some final visi
Math 416 Lecture 10
Project proposal due Today.
THEOREM. If N t is the number of events up to time t and if
the nth event arrives at time T n :
e $t ($t) n
For a given t,
P[N t = n] = n! ,
E[N t ] = $
Math 416 Lecture 12
Exam 2, March 1, covers Lectures 6-12
Under rather general regularity conditions, in the limit, as
t d , the population probability distribution approaches a
stable distribution p
Math 416 Lecture 31
STATE INDEPENDENT STOPPING PROBLEMS.
In each round of a game, a die is rolled.
If the roll is a 1, the game stops and you get nothing.
Otherwise you may choose to
continue (and col