132B_1_Sec10_DTMC_090710

132B_1_Sec10_DTMC_090710 - Discrete-Time Markov Chains...

Info iconThis preview shows pages 1–7. Sign up to view the full content.

View Full Document Right Arrow Icon
Discrete-Time Markov Chains Professor Izhak Rubin Electrical Engineering Department UCLA © 2010-2011 by Izhak Rubin
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
© Prof. Izhak Rubin 2 Discrete-Time Markov Chain: Definition z X = {X k , k = 0,1,2,…} is a discrete time stochastic process; states assume values in a countable state space S, such as S = {0,1,2….}, or S = {0,1,2…, N), or S = {a 1 , a 2 , a 3 ,….}. It is said to be a Markov Chain if it satisfies the Markov Property: k+1 0 0 k-1 k-1 k k+1 k k Markov Property (Given present and past states, distribution of future states is independent of the pa : P(X = j| X = i ,..., X = i , X = i) = P(X i ) = P (i,j), for e st) ach t 01 1 ime 1,and states ( , , ,. .. ) . Assume a time homogeneous process: its statistical behavior is charaterized by the (stationary) transition probability function (TPF) (, ) ( | ) k kk k ki j i i S Pij PX jX i + ≥∈ == = = 10 ( | ), i,j S,k 1. Transition Probability Matrix: { ( , ), , }. T j X i PP i j i j S = =∈ X k k 0 1 2 3 4 5 X 2 X 3 X 4
Background image of page 2
© Prof. Izhak Rubin 3 Transition Probability Function (TPF): Properties z Properties of P T z Initial Distribution: z Calculation of joint state distribution: 1. ( , ) 0,each , ; 2. ( , ) 1,each . jS Pi j i j S i S ≥∈ =∈ S i i X P i P = = ), ( ) ( 0 0 00 11 0 0 1 0 1 ( , ,..., ) ( ) ( , ), for 1,( ,. .., ) k kk jj k j PX i X i X i Pi Pi i k i i S = == = =
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
© Prof. Izhak Rubin 4 Example 1: – Two State Markov Chain z X= DTMC with binary RVs on state space S={0,1} z Transition probability function is given by: 1 01 ; 1 P λλ λμ μμ ⎡⎤ =≤ ⎢⎥ ⎣⎦
Background image of page 4
© Prof. Izhak Rubin 5 Example 2– Binomial Counting Process; Geometric Point Process th 1 0 No. of arrivals during the k slot 0 discrete-time counting process where No arrivals in first k slots, 1,2,3. .. 0 Assume:{M , 1} - i.i.d. RVs, with 1, 0 () ,1 Then N is a k k, k ki i k k M N{ N k ) . NM k N: k pj PM j = = =≥ == = = −= = DT Markov Chain with 1-p if j=i P(i,j) = pi f j = i + 1
Background image of page 5

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
© Prof. Izhak Rubin 6 Example 2 (Cont.) – Binomial Counting Process and Geometric Point Process {} 11 1 2 2 1 112 2 2 1 Markov Property holds: |, , . . . , , . . . , 1, (, ) is a DT MC , 1 Associated discrete time point process { , kk ii k n PN jN nN n N i PM j M n M M n M
Background image of page 6
Image of page 7
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 17

132B_1_Sec10_DTMC_090710 - Discrete-Time Markov Chains...

This preview shows document pages 1 - 7. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online