notes216

# notes216 - An idiosyncratic introduction to stochastic...

This preview shows pages 1–3. Sign up to view the full content.

An idiosyncratic introduction to stochastic processes Class notes for Math 216 Notes - Fall 2010 Jonathan C. Mattingly September 15, 2010 1 Finite State Markov Chains A discrete time stochastic process ( X n ) n 0 is a collection of random variables indexed by the non-negative integers Z + = { n Z : n 0 } . The set in which the X n take values is called the state space of the stochastic process. Definition. A stochastic process ( X n ) n 0 is a Markov chain if P ( X n +1 = j | X n = i n , · · · , X 0 = i 0 ) = P ( X n +1 = j | X n = i n ) for all j, i n , · · · , i 0 I . Definition. A Markov chain is time homogeneous if for all k Z + and i, j I P ( X k +1 = i | X k = j ) = P ( X 1 = i | X = j ) Unless we say otherwise we will always assume that all Markov chains are time homogeneous. In such cases we will write p n ( i, j ) = P ( X n = j | X 0 = i ) By the Markov property one has P ( X n = x n , X n - 1 - x n - 1 , · · · X 1 = x 1 | X 0 = x 0 ) = p 1 ( x n , x n - 1 ) p 1 ( x n - 1 , x n - 2 ) · · · p 1 ( x 1 , x 0 ) We will begin by concentrating on stochastic processes on a finite state space I . With out loss of generality, we can take the state space to be I = { 0 , 1 . . . . , N } . 1.1 Markov chains and matrices There is a very fruitful correspondence between finite state Markov chains and Matrices. We begin by considering random variables on a state space I = { 0 , . . . , N - 1 } . Such a random variable X can be specified completely by N non-negative numbers { λ i : i I } such that P ( X = i ) = λ i . Clearly we have that i I λ i = 1. It is convenient to organize the λ i in a row-vector λ = ( λ 0 , . . . , λ N - 1 ) R N . The vector λ is called the distribution of the random variable X . With this in mind we make the following definition. Definition. A row vector λ = ( λ 0 , . . . , λ N - 1 ) R N called a distribution if λ i 0. If in addition N - 1 i =0 λ i = 1, it is called a probability distribution . 1

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
Let P R N,N be a matrix with non-negative entries. We will write P i,j for the i - jth entry of P , that is to say P = p 0 , 0 · · · p 0 ,N - 1 . . . . . . . . . p N - 1 , 0 · · · p N - 1 ,N - 1 Definition. A square matrix P with non-negative entries is called a stochastic matrix if all rows sum to one. That is to say, for all j , i P ji = 1. Stochastic matrices are in one-to-one correspondence with time homogeneous Markov processes on a finite state space. The correspondence is given by P ij = P ( X 1 = j | X 0 = i ) It then follows that P ( X n = j | X 0 = i ) = ( P n ) ij Or in other words, the distribution of the random variable X n when conditioned to have X 0 = i is given by the row vector ( P n ) i, * by which we mean the i th row of the matrix P n . In other words, ( ( P n ) i, 0 , . . . , ( P n ) i,N - 1 ) (1) If we denote by e ( i ) the row vector with 1 in the i th slot and 0 in the remaining slots, then (1) can be written compactly as e ( i ) P n . If instead of starting from a deterministic initial condition, we let X 0 be random with distribution given by the probability distribution λ = ( λ 1 , · · · , λ N - 1 ). This means that P ( X 0 = i ) = λ i .
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern