THEOREM. Let i, j, k be states of a Markov process. Then (a) ij and jk implies ik. For a random Markov chain, in the limit as time goes to (b) If j is recurrent and jk, then kj and k is recurrent. inf
Math 416 Lecture 18
Finite Horizon problem - continued
v RECURSION EQUATION (DYNAMIC PROGRAMMING EQUATION) FOR
Vm1 i. Given optimal Vm j with optimal policies v*:
V m1 i max aA ri, a j T a i, jV m j a
Exam 3 Review
Lectures 13-18, open notes, open text, laptops allowed.
Project homework due Tuesday March 29
Poisson Arrival Queues
Suppose the queue has type Poisson/Gx/5. That is,
the arrivals are Po
`A Markov decision problem has two states i=1, 2 and
two actions a = c, d. The discount is a = .9. The rewards
Infinite Horizon discounted reward problem
From the last lecture: The value function of p
Math 416 Lecture 16
Brownian motion continued
Recall: DEFINITION. A continuous random walk X t , t 0, is called a Brownian motion if, for some standard deviation s: v X 0 0 and ft X t is continuous wi
Math 416 Lecture 15
p1 p p p is maximal iff its derivative 1 2p 0
iff p 1/2. At p 1/2, p1 p 1/21/2 1/4. When
p 1/2, p1 p 1/4 and so 4p1 p 1.
STIRLINGS APPROXIMATION: n! 2n n n /e n . This gives
2
n
2
Math 416 Exam 2, Tuesday Review Covers Lectures 7-12
Let F k (i, j) = P[P T(j) = k|X 0 = i] be the probability that PT(j) = k steps from the initial state i. THEOREM. F 0 (i, j) = 1 iff i = j, F 1 (i,
Math 416 Lecture 12
Exam 2, March 1, covers Lectures 7-12 Kolmogorov Equations Recall: The long-term the probabilities p 0 , p 1 , p 2 , . of the population having sizes j= 0, 1, 2, . satisfy the foll
Math 416 Lecture 14
Poisson / G(x) / 1 Queues Suppose the arrivals are a Poisson process with arrival rate l and expected time 1/l between arrivals and with N t = the number of arrivals at time t. Sup
Math 416 Lecture 21
STATE INDEPENDENT STOPPING PROBLEMS.
In each round of a game, a die is rolled. If the roll is a
1, the game stops and you get nothing. Otherwise you
may choose to continue (and col
Math 416 Lecture 17 Project Hw due Mar. 29.
Exam 3 Mar. 31, Lectures 13-18. Markov decision processes Decision processes involve real-time decisions which affect the outcomes (e.g. instead of just pre
Nt the state space of possible counts = cfw_0, 1, 2, 3, . or
cfw_0, 1, 2, 3, ., M if there is an upper bound M. Unless
Birth and death processes, queueing theory
specified otherwise, assume N0 = 0.
I
Math 416 Lecture 10
Project proposal due Today. When flipping a coin, the previous history of outcomes has no effect on the next outcome - the coin has no memory. Likewise, Poisson and exponential dis
Math 416 Exam 1 Review Lectures 1-6.
BINOMIAL DISTRIBUTION. Let X = the number of successes. n! p(k) = P[X = k] = k!nk! p k 1 p nk . E(X) = np. GEOMETRIC DISTRIBUTION. X = the number of trials up to a
Math 416 Lecture 8
Project proposal due Tuesday, Feb. 15. THEOREM. If T is a regular matrix for an irreducible space or a submatrix for a recurrence class, then: v The rows of T are identical. Let p b
Math 416 Lecture 9
Consider T (i, j). Case j is transient. Then for any state i, T (i, j) = 0 (see Lecture 10). Case j is recurrent. Subcase i and j are in the same recurrence class and the class matr
Math 416 Lecture 6
Friday's exam covers Lectures 1-6, not Lecture 7. For a transition matrix T, the n-step transition probabilities are given by T n. The limit the system approaches as time goes to 5
Math 416 Lecture 5 `A professor's computer is replaced when it dies or in its fifth year, which ever comes first. Let the state X of a computer be the number of years until it is replaced. The state s
Math 416 Lecture 1
Homework assignments are due at the beginning of the next lecture.
Class website: www.math.hawaii.edu/416. Read "Syllabus".
Operations Research studies optimization and search algor
Math 416 Lecture 2
DEFINITION. Here are the multivariate versions: PMF case: p(x, y, z) is the joint Probability Mass Function of X, Y, Z iff P(X = x, Y = y, Z =z) = p(x , y, z). PDF case: f(x, y, z)
Math 416 Lecture 3
Expected values The average or mean or expected value of x 1 , x 2 , x 3 , ., x n is
x 1 x 2 .x n n x 1 1 x 2 1 . x n 1 1 x i px i n n n n
VARIANCE, STANDARD DEVIATION. For a random
Math 416 Lecture 4
Due Feb. 14: a project proposal which lists the chosen topic and
references to be used. The place to look for project topics is in the
library and on the internet (Google) for topic
Math 416 Lecture 13 not on Exam 2
RECALL. Suppose there are infinitely many lines, the
arrival times are Poisson with rate and the service times
have cumulative probability G(x). Suppose X t is the
nu