hw4sol - HOMEWORK 4 SOLUTIONS 1 A Markov chain with state...

Info iconThis preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: HOMEWORK 4: SOLUTIONS 1. A Markov chain with state space { 1 , 2 , 3 } has transition probability matrix P =   1 / 3 1 / 3 1 / 3 1 / 2 1 / 2 1   Show that state 3 is absorbing and, starting from state 1, find the expected time until absorption occurs. Solution. Let ψ ( i ) be the expected time to reach state 3 starting from state i , where i ∈ { 1 , 2 , 3 } . We have ψ (3) = 0 ψ (2) = 1 + 1 2 ψ (2) + 1 2 ψ (3) ψ (1) = 1 + 1 3 ψ (1) + 1 3 ψ (2) + 1 3 ψ (3) . We solve and find ψ (3) = 0 , ψ (2) = 2 , ψ (1) = 5 / 2 . 2. Smith is in jail and has 3 dollars; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability 0.4 and loses A dollars with probability 0.6. Find the probability that he wins 8 dollars before losing all of his money if (a) he bets 1 dollar each time (timid strategy). (b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy). (c) Which strategy gives Smith the better chance of getting out of jail? Solution. (a) The Markov chain ( X n , n = 0 , 1 , . . . ) representing the evolution of Smith’s money has diagram 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.6 0.6 0.6 0.6 0.6 0.6 0.6 1 2 3 4 5 6 7 8 1 1 Let ϕ ( i ) be the probability that the chain reaches state 8 before reaching state 0, starting from state i . In other words, if S j is the first n ≥ 0 such that X n = j , ϕ ( i ) = P i ( S 8 < S ) = P ( S 8 < S | X = i ) . Using first-step analysis (viz. the Markov property at time n = 1), we have ϕ ( i ) = 0 . 4 ϕ ( i + 1) + 0 . 6 ϕ ( i- 1) , i = 1 , 2 , 3 , 4 , 5 , 6 , 7 ϕ (0) = 0 ϕ (8) = 1 . 1 We solve this system of linear equations and find ϕ = ( ϕ (1) , ϕ (2) , ϕ (3) , ϕ (4) , ϕ (5) , ϕ (6) , ϕ (7)) = (0 . 0203 , . 0508 , . 0964 , . 1649 , . 2677 , . 4219 , . 6531 , 1) . E.g., the probability that the chain reaches state 8 before reaching state 0, starting from state 3 is the third component of this vector and is equal to 0.0964. Note that ϕ ( i ) is increasing in i , which was expected. (b) Now the chain is 1 2 3 4 5 6 7 8 1 1 0.4 0.4 0.4 0.6 0.6 0.6 and the equations are: ϕ (3) = 0 . 4 ϕ (6) ϕ (6) = 0 . 4 ϕ (8) + 0 . 6 ϕ (4) ϕ (4) = 0 . 4 ϕ (8) ϕ (0) = 0 ϕ (8) = 1 ....
View Full Document

This note was uploaded on 01/05/2010 for the course ECE 01 taught by Professor All during the Spring '09 term at Aarhus Universitet.

Page1 / 6

hw4sol - HOMEWORK 4 SOLUTIONS 1 A Markov chain with state...

This preview shows document pages 1 - 3. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online