THEOREM. Let i, j, k be states of a Markov process. Then (a) ij and jk implies ik. For a random Markov chain, in the limit as time goes to (b) If j is recurrent and jk, then kj and k is recurrent. infinity, a state is either visited infinitely often, or i
Math 416 Lecture 18
Finite Horizon problem - continued
v RECURSION EQUATION (DYNAMIC PROGRAMMING EQUATION) FOR
Vm1 i. Given optimal Vm j with optimal policies v*:
V m1 i max aA ri, a j T a i, jV m j and if a gives
the max value, let u* be the policy such
Exam 3 Review
Lectures 13-18, open notes, open text, laptops allowed.
Project homework due Tuesday March 29
Poisson Arrival Queues
Suppose the queue has type Poisson/Gx/5. That is,
the arrivals are Poisson with some rate l. There are
infinitely many lines
`A Markov decision problem has two states i=1, 2 and
two actions a = c, d. The discount is a = .9. The rewards
Infinite Horizon discounted reward problem
From the last lecture: The value function of policy u for ri, a are r1, c 5, r2, c 2, r1, d 4, r2, d
Math 416 Lecture 16
Brownian motion continued
Recall: DEFINITION. A continuous random walk X t , t 0, is called a Brownian motion if, for some standard deviation s: v X 0 0 and ft X t is continuous with probability 1. v X ts X t , X ts X t , X s all have
Math 416 Lecture 15
p1 p p p is maximal iff its derivative 1 2p 0
iff p 1/2. At p 1/2, p1 p 1/21/2 1/4. When
p 1/2, p1 p 1/4 and so 4p1 p 1.
STIRLINGS APPROXIMATION: n! 2n n n /e n . This gives
2
n
2n
2n!
n! 2
22n 2n 2n /e 2n
2n
n n /e n 2
2 n 2 2n n 2
Math 416 Exam 2, Tuesday Review Covers Lectures 7-12
Let F k (i, j) = P[P T(j) = k|X 0 = i] be the probability that PT(j) = k steps from the initial state i. THEOREM. F 0 (i, j) = 1 iff i = j, F 1 (i, j) = T(i, j), F k (i, j) = xcS-cfw_j T(i, x)F k-1 (x,
Math 416 Lecture 12
Exam 2, March 1, covers Lectures 7-12 Kolmogorov Equations Recall: The long-term the probabilities p 0 , p 1 , p 2 , . of the population having sizes j= 0, 1, 2, . satisfy the following Kolmogorov steady state equations: 0 = -p 0 $ 0 +
Math 416 Lecture 14
Poisson / G(x) / 1 Queues Suppose the arrivals are a Poisson process with arrival rate l and expected time 1/l between arrivals and with N t = the number of arrivals at time t. Suppose the service times are mutually independent and hav
Math 416 Lecture 21
STATE INDEPENDENT STOPPING PROBLEMS.
In each round of a game, a die is rolled. If the roll is a
1, the game stops and you get nothing. Otherwise you
may choose to continue (and collect nothing at this time) or
stop and collect $1 for e
Math 416 Lecture 17 Project Hw due Mar. 29.
Exam 3 Mar. 31, Lectures 13-18. Markov decision processes Decision processes involve real-time decisions which affect the outcomes (e.g. instead of just predicting monthly sales, we must decide how much to spend
Nt the state space of possible counts = cfw_0, 1, 2, 3, . or
cfw_0, 1, 2, 3, ., M if there is an upper bound M. Unless
Birth and death processes, queueing theory
specified otherwise, assume N0 = 0.
In arrival processes, the state only jumps up, in a
birt
Math 416 Lecture 10
Project proposal due Today. When flipping a coin, the previous history of outcomes has no effect on the next outcome - the coin has no memory. Likewise, Poisson and exponential distributions. THEOREM. For a Poisson distribution N t , t
Math 416 Exam 1 Review Lectures 1-6.
BINOMIAL DISTRIBUTION. Let X = the number of successes. n! p(k) = P[X = k] = k!nk! p k 1 p nk . E(X) = np. GEOMETRIC DISTRIBUTION. X = the number of trials up to and including the first success. P[X = k] = p1 p k1 for
Math 416 Lecture 8
Project proposal due Tuesday, Feb. 15. THEOREM. If T is a regular matrix for an irreducible space or a submatrix for a recurrence class, then: v The rows of T are identical. Let p be the common row. Its j th entry = the proportion j of
Math 416 Lecture 9
Consider T (i, j). Case j is transient. Then for any state i, T (i, j) = 0 (see Lecture 10). Case j is recurrent. Subcase i and j are in the same recurrence class and the class matrix is regular. By the previous lecture T (i, j) = T (j,
Math 416 Lecture 6
Friday's exam covers Lectures 1-6, not Lecture 7. For a transition matrix T, the n-step transition probabilities are given by T n. The limit the system approaches as time goes to 5 is T limn T n . Diagonalization makes this limit easy t
Math 416 Lecture 5 `A professor's computer is replaced when it dies or in its fifth year, which ever comes first. Let the state X of a computer be the number of years until it is replaced. The state space is cfw_1, 2, 3, 4. Here are the replacement times
Math 416 Lecture 1
Homework assignments are due at the beginning of the next lecture.
Class website: www.math.hawaii.edu/416. Read "Syllabus".
Operations Research studies optimization and search algorithms
often used in business and engineering. In Math 4
Math 416 Lecture 2
DEFINITION. Here are the multivariate versions: PMF case: p(x, y, z) is the joint Probability Mass Function of X, Y, Z iff P(X = x, Y = y, Z =z) = p(x , y, z). PDF case: f(x, y, z) is the joint Probability Density Function of X, Y, Z if
Math 416 Lecture 3
Expected values The average or mean or expected value of x 1 , x 2 , x 3 , ., x n is
x 1 x 2 .x n n x 1 1 x 2 1 . x n 1 1 x i px i n n n n
VARIANCE, STANDARD DEVIATION. For a random variable X with mean m = EX, the expected value of x 2
Math 416 Lecture 4
Due Feb. 14: a project proposal which lists the chosen topic and
references to be used. The place to look for project topics is in the
library and on the internet (Google) for topics like genetic
algorithms, genetic programming,neural n
Math 416 Lecture 13 not on Exam 2
RECALL. Suppose there are infinitely many lines, the
arrival times are Poisson with rate and the service times
have cumulative probability G(x). Suppose X t is the
number of customers in the system at time t. Then the
exp