Coursehero >> Wisconsin >> Wisconsin Milwaukee >> CS 616

Course Hero has millions of student submitted documents similar to the one below including study guides, homework solutions, papers, exam answer keys and textbook solutions.

Stochastic CS616 Models in Computer Science Instructor: Peter Kemper R 006, phone 221-3462, email:kemper@cs.wm.edu Office hours: Mon,Wed 3-5 pm Grader: Yunlian Jiang, R 101B, email: jiang@cs.wm.edu Office hours: Tue,Thu 2-3 pm Today: Markov Chains Quick Reference: Ross, Ch. 4.1,4.2 D.L. Minh, Applied Probability Models, Duxbury, Thomson Learning, 2001. 1 Overview this is the plan Probability Theory Random Variables Conditional Probability Conditional Expectation Markov Chains Exponential Distribution & Poisson Process Continuous Time Markov Chain Tools: - Mobius 2 Overview Today Stochastic Processes of various kinds Markov chains of order i Chapman-Kolmogorov Equations Later Classification of States Limiting Probabilities Some Applications Mean Time Spent in Transient States Branching Processes Time Reversible Markov Chains Markov Chain Monte Carlo Methods Markov Decision Processes Hidden Markov Chains 3 Stochastic process (Chapter 2.8 revisited) Definition A stochastic process {X(t), t T} is a collection of random variables with some index set T. The process is Discrete time if T is countable, Continuous time if T is an interval of the real line The state space of the process is the set of all possible values of X(t) over all t T. Common interpretation T as time X(t) as the state of the process at time t. Useful to describe the evolution of some (physical) process over time. 4 Reminder: Example from Chapter 2.8 Particle moves along m+1 nodes with labels 0,1, ,m-1. Nodes are located on a circle. Particle can move one node forward or backward (with same probability). Let Xn be the position after n steps with the usual modulo arrangement to move from m-1 to 0. Let the particle start at 0, move until all nodes visited. What is the probability that node i is the last node? For 0 it is 0, for 1, ,m it is 1/m Obviously a special case that does not generalize 5 Some terminology (Sample) path Outcome of stochastic process, i.e., sequence of outcomes obtained from sequence of experiments Note: new term useful to distinguish from outcome of experiment Sample space: Collection of all possible paths Set of all possible distinct outcomes of experiments Individual element is called state State space S: Example: Consider a game like Lotto, e.g. Mega Millions, played over some time. What is its state space, what is a state, what is a path? 6 Four Types of Stochastic Processes Criteria: discrete - continuous state space S discrete - continuous parameter space T 1. Discrete time chain 2. Continuous time chain 3. Discrete time process {Xn, n=0,1, }, S discrete (finite or infinite), T discrete {X(t)}, S discrete (finite or infinite), T continuous, e.g., t in [0, ) {Xn, n=0,1, }, S continuous, T discrete {X(t)}, S continuous, T continuous, e.g., t in [0, ) 4. Continuous time process 7 Special case: i.i.d. RVs Consider a discrete-time chain {Xn, n=0,1, } For any n X0, X1, X2, Xn-1 is the past Xn is the present state Xn+1, Xn+2, is the future If all variables are independent, then This is for sure a nice property, but considering the process instead of individual experiments does not give us anything. It would be more interesting (and useful) if we were able to consider some dependencies 8 General case Consider a discrete-time chain {Xn, n=0,1, } where Xn+1 is dependent on all previous outcomes. Formally and no simplifications. This gets intractable very quickly. This is an overkill for many practical applications. Observation: influence on the chain s earlier outcomes on its future tends to diminish rapidly as time passes. Let s look for a compromise: Rich enough to be useful to model real world phenomena Simple enough to be analyzed (with or without computations) 9 Markov Chains of Order i Consider a discrete-time chain {Xn, n=0,1, } where Xn+1 is dependent on the last i outcomes only. Formally So: Special case of i.i.d rvs is an MC of order 0 Special case of MC of order 1 is the one we will focus on Note: A MC of order i can be transformed into a MC of order 1. 10 Examples given in D.L. Minh Weather (Chin 1977) Chin models daily precipitation as an MC based on data from 1948 to 1973 for more than 100 locations in the US. Order of MC depends primarily on season of the year, on the geographical location only to a lesser degree. Winter months (Jan/Feb): second or higher order conditional dependence for majority of locations Summer months (Jul/Aug): first order conditional dependence for majority of locations. Nine four-man groups observed when engaged in a discussion Order in which persons spoke modeled as MC Parker identified second order as best. Groups dynamics (Parker 1988) 11 Following S. Ross: Consider a stochastic process {Xn,n=0,1,2, } with finite or countable state space S This set is denoted by nonnegative integers For a fixed probability Pij: for all states i0,i1, ,in-1,i,j and all n 0 This is a first order Markov chain The conditional distribution of any future state Xn+1 is dependent only on the present state, Xn and not any previous states The formal notation (or mapping S to {0,1, }) allows us to use a matrix notation where row, column indices match with states. 12 Discrete time MC Pij is the probability that the process, currently in state i, will go to state j One-step transition probabilities Pij 13 Example 1: Forecasting the Weather Suppose that the forecast only depends on the previous day s forecast If it rains today, will it rain tomorrow with probability If it does not rain today, it will rain tomorrow with probability Process is in state 0 while raining, and state 1 while not raining What is the Markov chain? 14 Example 2: Transforming a Process into a Markov Chain If it has rained the past two days, it will rain tomorrow with probability 0.7 If it rained today, but not yesterday, it will rain tomorrow with probability 0.5 If it rained yesterday, but not today, it will rain tomorrow with probability 0.4 If it didn t rain yesterday or today, it will rain tomorrow with probability 0.2 The forecast for tomorrow is determined by both today s weather and yesterday s 15 Example 2 Continued We need to determine states and state transitions: States State State State State 0: 1: 2: 3: it it it it rained both today and yesterday rained today, but not yesterday rained yesterday, but not today didn t rain yesterday or today Transition probabilities 16 Example 3: A Random Walk Model Markov chain state space given by integers i=0, 1, 2, is a random walk if 0<p<1, At each step, the person walking may take a step to the right with probability p, or to the left with probability 1-p at each point in time 17 Example 4: A Gambling Model A gambler wins $1 with probability p or loses $1 with probability 1-p at each point in time The gambler quits when broke or when he has attained $N States 0 and N are absorbing states, once entered they are never left This example is a finite state walk with absorbing barriers 18 Example 5 Annual automobile insurance, a Bonus Malus system Each policyholder gets a positive integer value (the state) The annual premium is a function of this state (along with the type of car, level of insurance, etc) The annual premium changes from year to year, the more claims filed by the policyholder, the higher the less claims filed, the lower the premium Let si(k) denote the next state of a policyholder currently in state i who made k number of claims that year Suppose the number of claims made is a Poisson random variable with parameter The resulting Markov chain is 19 Example 5 Continued Table that gives Next State, row: state, col: k claims s2(0)=1; s2(1)=3; s2(k)=4, k 2 0 1 2 3 4 1 1 2 3 1 2 3 4 4 2 3 4 4 4 k 2 4 4 4 4 ak : probability of k claims in a year Transition probability matrix for a policyholder: 20 Chapman-Kolmogorov Equations n-step transition probabilities Chapman-Kolmogorov equations method for computing n-step transition probabilities In words: probability to go through k after n steps Formally, 21 Chapman-Kolmogorov Equations Let P(n) denote the matrix of n-step transition probabilities In particular 22 Example: Two state weather example revisited Let =0.7 and =0.4, so we get What is the probability that it will rain four days from today given that it is currently raining ? Hence 23 Role of initial distribution So far, all probabilities have been conditional has condition that we are in state i at time 0. If the unconditional distribution of the state at time n is wanted, the initial state probability distribution is needed 24 Weather example continued If 0=0.4, 1=0.6, the (unconditional) probability that it will rain four days after is 25 Absorbing states How can we determine the probability of a Markov chain entering any of the specified set of states A by time n ? Reset the transition probabilities out of states in A Transform all states in A into absorbing states Note: The original and transformed Markov chain follow identical probabilities until a state in A is entered So the probability that the original Markov chain enters a state in A by time n is the same as the probability that the transformed Markov chain does so 26 Example A pensioner Income: 2 (= $2000) at day 1 of each month Spending: independent of the amount he has Amount is i, with probability Pi, i=1,2,3,4 Between 0 and 3 At the end of the month, if the pensioner has more than 3, he gives the rest to his son Savings: If, the pensioner has 5 after receiving his payment, what is the probability that his capital is ever 1 or less at any time within the following four months? 27 Solution Markov chain where state = amount at end of month We only consider states 1,2,3 1 means amount is less or equal 1 at the end of the month OK, since we are only interested in probability of state 1 being reached. Probability matrix Q=[Qi,j] Consider Q2,1, the probability that the pensioner ends a month with 2 and then 1 or less the next month Pensioner begins new month with 2+2=4 His ending capital will be 1 or less if his expenses are either 3 or 4, thus Q2,1=P3+P4 28 Solution Continued Suppose Pi=1/4, i=1,2,3,4 Q4=(Q2)2, so squaring Q and the resulting matrix yields So 29 Conditional probabilities and absorbing states Let {Xn,n 0} be a MC with transition probabilities Pi,j Let Qi,j denote the transition probabilities that transform all states in A into absorbing states The conditional probability of Xn given that the chain starts in state i and has not entered any state in A by time n is for any i,j A 30 Summary Today: Stochastic Processes of various kinds Markov chains of order i Chapman-Kolmogorov Equations Absorbing states Next time we continue with: Classification of States Limiting Probabilities Some Applications Mean Time Spent in Transient States Branching Processes Time Reversible Markov Chains Markov Chain Monte Carlo Methods Markov Decision Processes Hidden Markov Chains 31

Find millions of documents here - Study Guides, Homework Solutions, Papers, Exam Answer Keys and more. Course Hero has millions of course related materials that will enable you to learn better, faster and get an A in all your courses.
Below is a small sample set of documents: