This preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: HOMEWORK 4: SOLUTIONS 1. A Markov chain with state space { 1 , 2 , 3 } has transition probability matrix P = ï£« ï£ 1 / 3 1 / 3 1 / 3 1 / 2 1 / 2 1 ï£¶ ï£¸ Show that state 3 is absorbing and, starting from state 1, find the expected time until absorption occurs. Solution. Let Ïˆ ( i ) be the expected time to reach state 3 starting from state i , where i âˆˆ { 1 , 2 , 3 } . We have Ïˆ (3) = 0 Ïˆ (2) = 1 + 1 2 Ïˆ (2) + 1 2 Ïˆ (3) Ïˆ (1) = 1 + 1 3 Ïˆ (1) + 1 3 Ïˆ (2) + 1 3 Ïˆ (3) . We solve and find Ïˆ (3) = 0 , Ïˆ (2) = 2 , Ïˆ (1) = 5 / 2 . 2. Smith is in jail and has 3 dollars; he can get out on bail if he has 8 dollars. A guard agrees to make a series of bets with him. If Smith bets A dollars, he wins A dollars with probability 0.4 and loses A dollars with probability 0.6. Find the probability that he wins 8 dollars before losing all of his money if (a) he bets 1 dollar each time (timid strategy). (b) he bets, each time, as much as possible but not more than necessary to bring his fortune up to 8 dollars (bold strategy). (c) Which strategy gives Smith the better chance of getting out of jail? Solution. (a) The Markov chain ( X n , n = 0 , 1 , . . . ) representing the evolution of Smithâ€™s money has diagram 0.4 0.4 0.4 0.4 0.4 0.4 0.4 0.6 0.6 0.6 0.6 0.6 0.6 0.6 1 2 3 4 5 6 7 8 1 1 Let Ï• ( i ) be the probability that the chain reaches state 8 before reaching state 0, starting from state i . In other words, if S j is the first n â‰¥ 0 such that X n = j , Ï• ( i ) = P i ( S 8 < S ) = P ( S 8 < S  X = i ) . Using firststep analysis (viz. the Markov property at time n = 1), we have Ï• ( i ) = 0 . 4 Ï• ( i + 1) + 0 . 6 Ï• ( i 1) , i = 1 , 2 , 3 , 4 , 5 , 6 , 7 Ï• (0) = 0 Ï• (8) = 1 . 1 We solve this system of linear equations and find Ï• = ( Ï• (1) , Ï• (2) , Ï• (3) , Ï• (4) , Ï• (5) , Ï• (6) , Ï• (7)) = (0 . 0203 , . 0508 , . 0964 , . 1649 , . 2677 , . 4219 , . 6531 , 1) . E.g., the probability that the chain reaches state 8 before reaching state 0, starting from state 3 is the third component of this vector and is equal to 0.0964. Note that Ï• ( i ) is increasing in i , which was expected. (b) Now the chain is 1 2 3 4 5 6 7 8 1 1 0.4 0.4 0.4 0.6 0.6 0.6 and the equations are: Ï• (3) = 0 . 4 Ï• (6) Ï• (6) = 0 . 4 Ï• (8) + 0 . 6 Ï• (4) Ï• (4) = 0 . 4 Ï• (8) Ï• (0) = 0 Ï• (8) = 1 ....
View
Full
Document
This note was uploaded on 01/05/2010 for the course ECE 01 taught by Professor All during the Spring '09 term at Aarhus Universitet.
 Spring '09
 all

Click to edit the document details