Markov Tutorial: Chains
Steve Gu Feb 28, 2008
Outline
Markov chain Applications
Weather forecasting Enrollment assessment Sequence generation Rank the web page Life cycle analysis
Summary
History
The origin of Markov chains is due to Markov, a Russian mathematician who first published in the Imperial Academy of Sciences in St. Petersburg in 1907, a paper studying the statistical behavior of the letters in Onegin, a well known poem of Pushkin.
A Markov Chain
P01 P00 "0" "1"
P 11
P10
Transition Probability Table
P11 P P 21 P31 P12 P22 P32 P13 P23 P33
Pij 0, i = 1,..., n; j = 1,..., n and
Pij
j =1
n
1
P11 = 0.7 P12 = 0.2 P13 = 0.1 P21 = 0. P22 = 0.6 P23 = 0.4 P31 = 0.3 P32 = 0.5 P33 = 0.2
Example 1: Weather Forecasting[1]
Weather Forecasting
Weather forecasting example:
Suppose tomorrow's weather depends on today's weather only. We call it an Order-1 Markov Chain, as the transition function depends on the current state only. Given today is sunny, what is the probability that the coming days are sunny, rainy, cloudy, cloudy, sunny ? Obviously, the answer is : (0.5)(0.4)(0.3)(0.5) (0.2) = 0.0054 0.1
0.5 0.4 sunny 0.3 rainy 0.3 0.2 0.4 0.3
cloudy 0.5
Weather Forecasting
Weather forecasting example:
Given today is sunny, what is the probability that it will be rainy 4 days later? We only knows the start state, the final state and the input length = 4 There are a number of possible combinations of states in between.
0.1
0.5 0.4 sunny 0.3 rainy 0.3 0.2 0.4 0.3
cloudy 0.5
Weather Forecasting
Weather forecasting example:
Chapman-Kolmogorov Equation: Transition Matrix: s s 0.5 r 0.3
P
r
( nm) ij
( ( Pikn ) Pkj m ) k 0
c
0.4 0.1 0.4 0.3 c 0.2 0.3 0.5
0.1
0.5 0.4 sunny 0.3 rainy 0.3 0.2 0.4 0.3
cloudy 0.5
Weather Forecasting
Weather forecasting example:
Two days:
(00 x 01) + (01 x 11) + (02 x 21) 01
0.5 0.4 0.1 0.5 0.4 0.1 0.39 0.39 0.22 0.3 0.4 0.3 0.3 0.4 0.3 0.33 0.37 0.30 0.2 0.3 0.5 0.2 0.3 0.5 0.29 0.35 0.36
Four days:
0.5 0.4 0.1 0.5 0.4 0.1 0.3446 0.3734 0.2820 0.3 0.4 0.3 0.3 0.4 0.3 0.3378 0.3706 0.2916 0.2 0.3 0.5 0.2 0.3 0.5 0.3330 0.3686 0.2984
2
2
0.1
0.5 0.4 sunny 0.3 rainy 0.3 0.2 0.4 0.3
cloudy 0.5
Weather Forecasting
Weather forecasting example:
What is the probability that today is cloudy? There are infinite number of days before today. It is equivalent to ask the probability after infinite number of days. We do not care the "start state" as it brings little effect when there are infinite number of states. We call it the "Limiting probability" when the machine becomes steady. 0.1
0.5 0.4 sunny 0.3 rainy 0.3 0.2 0.4 0.3
cloudy 0.5
Weather Forecasting
Weather forecasting example:
Since the start state is "don't care", instead of forming a 2-D matrix, the limiting probability is express a a single row matrix :
0 , 1 , 2
Since the machine is steady, the limiting probability does not change even it goes one more step. 0.1
0.5 0.4 sunny 0.3 rainy 0.3 0.2 0.4 0.3
cloudy 0.5
Weather Forecasting
Weather forecasting example:
So the limiting probability can be computed by:
0 , 1 , 2
0.5 0.4 0.1 0.3 0.4 0.3 0.2 0.3 0.5
0 , 1 , 2
18
We have 0 , 1 , 2 ( 21 , 23 , 18 ) probability that today is cloudy = 62 62 62 62 0.1
0.5 0.4 sunny 0.3 rainy 0.3 0.2 0.4 0.3
cloudy 0.5
Example 2: Enrollment Assessment [1]
Undergraduate Enrollment Model
Stop Out
Freshmen
Sophomore
Junior
Senior
Graduate
State Transition Probabilities
Fr
Fr So .2 0 0
So
.65 .25 0
Jr
0 .6 .3
Sr
0 0
S/O
.14 .02 .13
Gr
.01 .03
TP
= Jr
.55 .12
Sr
0
0
0.1
0
0.4
.4
.05
.55
0
S/O 0.1
0.1 0.3
Gr
0
0
0
0
0
1
Enrollment Assessment
Stop Out
Freshmen
Sophomore
Junior
Senior
Graduate
TP
=
Fr So Jr
Fr .2 0 0 0 0.1 0
So .65 .25 0 0 0.1 0
Jr 0 .6 .3 0 0.4 0
Sr 0 0 .55 .4 0.1 0
S/O .14 .13 .12 .05 0.3 0
Gr .01 .02 .03 .55 0 1
Given: Transition probability table & Initial enrollment estimation, we can estimate the number of students at each time point
Sr S/O Gr
Example 3: Sequence Generation[3]
Sequence Generation
Markov Chains as Models of Sequence Generation s s1s2 s3s4 ttacggt
0th-order
N N
P0 s pt 1 pt pa s3pc ps psi s ps2 p g p i
1st-order 1th-order
i 1 i 1 N N
P s pt 1 pts2 | spaps3 |ps2 | ps1 s p|si |1 si 1 s p | t 1 | t c a p 1 psi 1
2 2nd-order
i 2 i 2
N N
P22s ptt1s2pa s3 |s1sc|tas4p|s2 s3 p1s2s2 psis|i 2i s21i1 P s ps p | tt p2 p g | ac p s 1 p si | s i s
i 3 3 i
A Fifth Order Markov Chain
Example 4: Rank the web page
How to rank the importance of web pages?
PageRank
PageRank
http://en.wikipedia.org/wiki/Image:PageRanks-Example.svg
PageRank: Markov Chain
For N pages, say p1,...,pN
Write the Equation to compute PageRank as:
where l(i,j) is define to be:
PageRank: Markov Chain
Written in Matrix Form:
PR(p1,n + 1) PR(p1,n) l(1,N) l(1,1) l(1,2) PR(p 2 ,n + 1) PR(p 2 ,n) l(2,1) l(2,2) l(2,N) PR(pN-1,n) PR(pN-1,n + 1) l(N,1) l(N,N - 1) l(N,N) PR(p ,n) PR(p ,n + 1) N N
Example 5: Life Cycle Analysis[4]
How to model life cycles of Whales?
http://www.specieslist.com/images/external/Humpback_Whale_underwater.jpg
Life cycle analysis
In real application, we need to specify or learn the transition probability table
calf
immature
mature
mom
Post-mom
dead
Application: The North Atlantic right whale (Eubalaena glacialis)
June 2006
Hal Caswell -- Markov Anniversary Meeting
30
Endangered, by any standard N < 300 individuals Minimal recovery since 1935 feeding Ship strikes Entanglement with fishing gear
calving
June 2006
Hal Caswell -- Markov Anniversary Meeting
31
Mortality and serious injury due ...