lect0203 - IEOR 4106: Professor Whitt Lecture Notes,...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
IEOR 4106: Professor Whitt Lecture Notes, Tuesday, February 3 Introduction to Markov Chains 1. Markov Mouse: The Closed Maze We start by considering how to model a mouse moving around in a maze. The maze is a closed space containing nine rooms. The space is arranged in a three-by-three array of rooms, with doorways connecting the rooms, as shown in the figure below The Maze 1 2 3 4 5 6 7 8 9 There are doors leading to adjacent rooms, vertically and horizontally. In particular, there are doors from 1 to 2 , 4 from 2 to 1 , 3 , 5 from 3 to 2 , 6 from 4 to 1 , 5 , 7 from 5 to 2 , 4 , 6 , 8 from 6 to 3 , 5 , 9 from 7 to 4 , 8 from 8 to 5 , 7 , 9
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
from 9 to 6 , 8 We assume that the mouse is a Markov mouse; i.e., the mouse moves randomly from room to room, with the probability distribution of the next room depending only on the current room, not on the history of how it got to the current room. (This is the Markov property .) Moreover, we assume that the mouse is equally likely to choose each of the available doors in the room it occupies. (That is a special property beyond the Markov property.) We now model the movement of the mouse as a Markov chain. The state of the Markov chain is the room occupied by the mouse. We let the time index n refer to the n th room visited by the mouse. So we make a discrete-time Markov chain. Specifically, we let X n be the state (room) occupied by the mouse on step (or time or transition) n . The initial room is X 0 . The room after the first transition is X 1 , and so forth. Then { X n : n 0 } is the discrete-time Markov chain (DTMC) ; it is a discrete-time discrete-state stochastic process. The mouse is in room X n ( a random variable) after making n moves, after having started in room X 0 , which could also be a random variable. We specify the evolution of the Markov chain by specifying the one-step transition prob- abilities. We specify these transition probabilities by specifying the transition matrix . For our example, making a discrete-time Markov chain model means that we define a 9 × 9 Markov transition matrix consistent with the specification above. For the most part, specifying the transition matrix is specifying the model. (We also must say how we start. The starting point could be random, in which case the initial position would be specified by a probability vector.) Notation. It is common to denote the transition matrix by P and its elements by P i,j ; i.e., P i,j denotes the probability of going to state j next when currently in state i . (When the state space has m states (is finite), P is a square m × m matrix.) For example, for our Markov mouse model, we have P 1 , 2 = 1 / 2 and P 1 , 4 = 1 / 2, with P 1 ,j = 0 for all other j , 1 j 9. And we have P 2 , 1 = P 2 , 3 = P 2 , 5 = 1 / 3 with P 2 ,j = 0 for all other j . And so forth. Here is the total transition matrix:
Background image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 01/26/2011 for the course IEOR 4106 taught by Professor Whitt during the Spring '08 term at Columbia.

Page1 / 11

lect0203 - IEOR 4106: Professor Whitt Lecture Notes,...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online