501: TCOM Networking Theory & Fundamentals
Lecture 6 February 19, 2003 Prof. Yannis A. Korilis
1
6-2
Topics
Time-Reversal of Markov Chains Reversibility Truncating a Reversible Markov Chain Burkes Theorem Queues in Tandem
6-3
Time-Reversed Markov Chains
{Xn: n=0,1,} irreducible aperiodic Markov chain with transition probabilities Pij
j =0
Pij = 1, i = 0,1,...
Unique stationary distribution (j > 0) if and only if:
j = i =0 i Pij , j = 0,1,...
Process in steady state:
Pr{ X n = j} = j = lim Pr{ X n = j | X 0 = i}
n
Starts at n=-, that is {Xn: n = ,-1,0,1,} Choose initial state according to the stationary distribution
How does {Xn} look reversed in time?
6-4
Time-Reversed Markov Chains
Define Yn=X-n, for arbitrary >0 {Yn} is the reversed process.
Proposition 1: {Y } is a Markov chain with transition probabilities: n
Pij* =
j Pji i
,
i, j = 0,1,...
{Yn} has the same stationary distribution j with the forward chain {Xn}
6-5
Time-Reversed Markov Chains
Proof of Proposition 1:
Pij* = P{Ym = j | Ym 1 = i, Ym 2 = i2 , K , Ym k = ik } = P{ X m = j | X m +1 = i, X m + 2 = i2 , K , X m + k = ik } = P{ X n = j | X n +1 = i, X n + 2 = i2 , K , X n + k = ik } = = = =
P{ X n = j , X n +1 = i, X n + 2 = i2 , K , X n + k = ik } P{ X n +1 = i, X n + 2 = i2 , K , X n + k = ik } P{ X n + 2 = i2 , K , X n + k = ik | X n = j, X n +1 = i}P{ X n = j , X n +1 = i} P{ X n + 2 = i2 , K , X n + k = ik | X n +1 = i}P{ X n +1 = i} P{ X n = j , X n +1 = i} = P{ X n = j | X n +1 = i} = P{Ym = j | Ym 1 = i} P{ X n +1 = i} P{ X n +1 = i | X n = j}P{ X n = j} Pji j = P{ X n +1 = i} i
* i ij
i =0 P = i =0 i
j Pji i
= j i =0 Pji = j
6-6
Reversibility
Stochastic process {X(t)} is called reversible if (X(t ), X(t ),, X(t )) and (X(-t ), X(-t ),, X(-t )) 1 2 n 1 2 n
have the same probability distribution, for all , t1,, tn Markov chain {Xn} is reversible if and only if the transition * P = Pij probabilities of forward and reversed chains are equal ij or equivalently, if and only if
i Pij = j Pji , i, j = 0,1,...
Detailed Balance Equations Reversibility
6-7
Reversibility Discrete-Time Chains
Theorem 1: If there exists a set of positive numbers {j}, that sum up to 1 and satisfy: Then: 1. {j} is the unique stationary distribution
2.
i Pij = j Pji ,
i, j = 0,1,...
The Markov chain is reversible
Example: Discrete-time birth-death processes are reversible, since they satisfy the DBE
6-8
Example: Birth-Death Process
P01
Pn 1,n
S
c
0
P00
1
P 10
2
Pn ,n 1
n
Pn ,n
Pn ,n +1 S
n+1
Pn +1,n
One-dimensional Markov chain with transitions only between neighboring states: Pij=0, if |i-j|>1 Detailed Balance Equations (DBE) n Pn ,n +1 = n +1Pn +1,n n = 0,1,... Proof: GBE nwith S ={0,1,,n} give: n
P =P
j =0 i = n +1 j ji j =0 i = n +1
i ij
n Pn ,n +1 = n +1Pn +1,n
6-9
Time-Reversed Markov Chains (Revisited)
Theorem 2: Irreducible Markov chain with transition probabilities Pij. If there exist: A set of transition probabilities Q , with Q =1, i 0, and ij j ij
A set of positive numbers {j}, that sum up to 1, such that i Pij = j Q ji , i, j = 0,1,... (1)
Then: Q are the transition probabilities of the reversed chain, and ij
{j} is the stationary distribution of the forward and the reversed chains
Remark: Use to find the stationary distribution, by guessing the transition probabilities of the reversed chain even if the process is not reversible
6-10
Continuous-Time Markov Chains
{X(t): -< t <} irreducible aperiodic Markov chain with transition rates qij, ij Unique stationary distribution (pi > 0) if and only if:
p j i j q ji = i j pi qij , j = 0,1,...
Process in steady state e.g., started at t =-:
t
Pr{ X (t ) = j} = p j = lim Pr{ X (t ) = j | X (0) = i}
If {j}, is the stationary distribution of the embedded discrete-time chain: /
pj =
i i / i
j
j
, j i j q ji ,
j = 0,1,...
6-11
Reversed Continuous-Time Markov Chains
1.
Reversed chain {Y(t)}, with Y(t)=X(-t), for arbitrary >0 Proposition 2: {Y(t)} is a continuous-time Markov chain with transition rates:
* qij =
p j q ji pi
,
i, j = 0,1,..., i j
1.
{Y(t)} has the same stationary distribution {pj} with the forward chain Remark: The transition rate out of state i in the reversed chain is equal to the transition rate out of state i in the forward chain
j i
q
* ij
=
j i
p j q ji
pi
=
pi j i qij pi
= j i qij = i ,
i = 0,1,...
6-12
Reversibility Continuous-Time Chains
Markov chain {X(t)} is reversible if and only if the transition rates of * forward and reversed chains are equal qij = qij , or equivalently pi qij = p j q ji , i , j = 0,1,..., i j Detailed Balance Equations Reversibility
Theorem 3: If there exists a set of positive numbers {pj}, that sum up to 1 and satisfy: pi qij = p j q ji , i , j = 0,1,..., i j Then: 1. {pj} is the unique stationary distribution
2.
The Markov chain is reversible
6-13
Example: Birth-Death Process
0
1
n 1
S
c
0
1
1
2
2
n
n
n S
n+1
n +1
Transitions only between neighboring states qi ,i +1 = i , qi ,i 1 = i , qij = 0, | i j |> 1 Detailed Balance Equations n pn = n +1 pn +1 , n = 0,1,... Proof: GBE with S ={0,1,,n} give:
n
j =0 i = n +1
p j q ji =
n
j = 0 i = n +1
pi qij n pn = n +1 pn +1
M/M/1, M/M/c, M/M/
6-14
Reversed Continuous-Time Markov Chains (Revisited)
Theorem 4: Irreducible continuous-time Markov chain with transition rates qij. If there exist:
A set of transition rates ij, with ji ij=ji qij, i 0, and A set of positive numbers {pj}, that sum up to 1, such that piij = p j q ji , i, j = 0,1,..., i j
Then: are the transition rates of the reversed chain, and ij
{pj} is the stationary distribution of the forward and the reversed chains
Remark: Use to find the stationary distribution, by guessing the transition probabilities of the reversed chain even if the process is not reversible
6-15
Reversibility: Trees
q23 q12 q01
4 3
2
q21 q61
q32 q67
0
q10
1
q16
5 7
6
q76
Theorem 5: For a Markov chain form a graph, where states are the nodes, and for each qij>0, there is a directed arc ij
Irreducible Markov chain, with transition rates that satisfy qij>0 qji>0 If graph is a tree contains no loops then Markov chain is reversible
Remarks: Sufficient condition for reversibility Generalization of one-dimensional birth-death process
6-16
Kolmogorovs Criterion (Discrete Chain)
Detailed balance equations determine whether a Markov chain is reversible or not, based on stationary distribution and transition probabilities Should be able to derive a reversibility criterion based only on the transition probabilities! Theorem 6: A discrete-time Markov chain is reversible if and only if:
Pi1i2 Pi2i3 L Pin1in Pini1 = Pi1in Pinin1 L Pi3i2 Pi2i1
for any finite sequence of states: i1, i2,, in, and any n
Intuition: Probability of traversing any loop i1i2ini1 is equal to the probability of traversing the same loop in the reverse direction i1ini2i1
6-17
Kolmogorovs Criterion (Continuous Chain)
Detailed balance equations determine whether a Markov chain is reversible or not, based on stationary distribution and transition rates Should be able to derive a reversibility criterion based only on the transition rates! Theorem 7: A continuous-time Markov chain is reversible if and only if:
qi1i2 qi2i3 L qin1in qini1 = qi1in qinin1 L qi3i2 qi2i1
for any finite sequence of states: i1, i2,, in, and any n
Intuition: Product of transition rates along any loop i1i2ini1 is equal to the product of transition rates along the same loop traversed in the reverse direction i1ini2i1
6-18
Kolmogorovs Criterion (proof)
Proof of Theorem 6: Necessary: If the chain is reversible the DBE hold 1 Pi1i2 = 2 Pi2i1 2 Pi2i3 = 3 Pi3i2 M Pi1i2 Pi2i3 L Pin1in Pini1 = Pi1in Pinin1 L Pi3i2 Pi2i1 n 1 Pin1in = n Pinin1 n Pini1 = 1 Pi1in
Sufficient: Fixing two states i1=i, and in=j and summing over all states i2,, in-1 we have n Pi ,i2 Pi2i3 L Pin1 , j Pji = Pij Pj ,in1 L Pi3i2 Pi2 ,i 1 Pijn Pji = Pij Pji 1 Taking the limit n lim Pijn 1 Pji = Pij lim ji 1 Pji = Pij Pn j
n n i
6-19
Example: M/M/2 Queue with Heterogeneous Servers
1A
A
0
(1 )
B A
B
2
3
A + B A + B
1B
M/M/2 queue. Servers A and B with service rates A and B respectively. When the system empty, arrivals go to A with probability and to B with probability 1-. Otherwise, the head of the queue takes the first free server. Need to keep track of which server is busy when there is 1 customer in the system. Denote the two possible states by: 1A and 1B. Reversibility: we only need to check the loop 01A21B0:
q0,1 A q1 A,2 q2,1B q1B ,0 = A B q0,1B q1B ,2 q2,1 A q1 A,0 = (1 ) B A
Reversible if and only if =1/2. What happens when A=B, and 1/2?
6-20
Example: M/M/2 Queue with Heterogeneous Servers
S3
1A
A
0
(1 )
S1
B A
B
2
S2
3
A + B A + B
1B
pn = p2 A + B
n2
, n = 2,3,...
p0 = A p1 A + B p1B ( A + B ) p2 = ( p1 A + p1B ) ( A + ) p1 A = p0 + B p2
+ ( A + B ) A 2 + A + B + (1 )( A + B ) p1B = p0 B 2 + A + B
p1 A = p0
2 + (1 ) A + B p2 = p0 AB 2 + A + B
1
2 + (1 ) A + B p0 + p1 A + p1B + n =2 pn = 1 p0 = 1 + 2 + A + B A + B A B
6-21
Multidimensional Markov Chains
Theorem 8: {X (t)}, {X (t)}: independent Markov chains 1 2
{Xi(t)}: reversible {X(t)}, with X(t)=(X1(t), X2(t)): vector-valued stochastic process {X(t)} is a Markov chain {X(t)} is reversible
Multidimensional Chains: Queueing system with two classes of customers, each having its own stochastic properties track the number of customers from each class Study the joint evolution of two queueing systems track the number of customers in each system
6-22
Example: Two Independent M/M/1 Queues
Two independent M/M/1 queues. The arrival and service rates at queue i are i and i respectively. Assume i= i/i<1. {(N1(t), N2(t))} is a Markov chain. Probability of n1 customers at queue 1, and n2 at queue 2, at steady-state
n p( n1 , n2 ) = (1 1 ) 1n1 (1 2 ) 2 2 = p1 ( n1 ) p2 ( n2 )
Product-form distribution Generalizes for any number K of independent queues, M/M/1, M/M/c, or M/M/. If pi(ni) is the stationary distribution of queue i:
p( n1 , n2 ,K , nK ) = p1 ( n1 ) p2 ( n2 )K pK ( nK )
6-23
Example: Two Independent M/M/1 Queues
Stationary distribution:
p( n1 , n2 ) = 1 1 1 1 2 2 1 1 2 2
n1 n2
1
1
1
03
2
2
13
1
1
2
2
23
1
33
2
Detailed Balance Equations:
1 p( n1 + 1, n2 ) = 1 p( n1 , n2 ) 2 p( n1 , n2 + 1) = 2 p( n1 , n2 )
2
1
1
2
1
1
2
2
02
2
12
1
1
2
2
22
1
1 2
2
32
1
1
2
2
01
1
11
2
21
1
2
2 2
31
1
1
Verify that the Markov chain is reversible Kolmogorov criterion
2
1
1
2
1
1
2
2
00
10
20
30
6-24
Truncation of a Reversible Markov Chain
Theorem 9: {X(t)} reversible Markov process with state space S, and stationary distribution {pj: jS}. Truncated to a set ES, such that the resulting chain {Y(t)} is irreducible. Then, {Y(t)} is reversible and has stationary distribution:
% pj =
pj
k E
pk
,
jE
Remark: This is the conditional probability that, in steady-state, the original process is at state j, given that it is somewhere in E Proof: Verify that:
% % p j q ji = pi qij pj q ji = =1 pi qij p j q ji = pi qij , i , j S ; i j
kE pk
pj
kE pk
% jE p j = jE
kE pk
6-25
Example: Two Queues with Joint Buffer
The two independent M/M/1 queues of the previous example share a common buffer of size B arrival that finds B customers waiting is blocked State space restricted to
E = {( n1 , n2 ) : ( n1 1)+ + ( n2 1)+ B}
1
03
2
2
1
13
1
1
2
2
1
02
2
2
12
1
1
2
22
2
1
Distribution of truncated chain:
n p( n1 , n2 ) = p(0,0) 1n1 2 2 , ( n1 , n2 ) E
1
1
2
2
1
01
2
2
11
1
1
21
1
31
2
Normalizing:
n p(0,0) = 1n1 2 2 ( n1 ,n2 )E
1
2
2
1
1
2
1
1
2
2
00
10
20
30
1
Theorem specifies joint distribution up to the normalization constant Calculation of normalization constant is often tedious
State diagram for B =2
6-26
Burkes Theorem
{X(t)} birth-death process with stationary distribution {pj} Arrival epochs: points of increase for {X(t)} Departure epoch: points of increase for {X(t)} {X(t)} completely determines the corresponding arrival and departure processes
Arrivals Departures
6-27
Burkes Theorem
Poisson arrival process: j=, for all j
Birth-death process called a (, j)-process Examples: M/M/1, M/M/c, M/M/ queues
Poisson arrivals LAA: For any time t, future arrivals are independent of {X(s): st} (, j)-process at steady state is reversible: forward and reversed chains are stochastically identical Arrival processes of the forward and reversed chains are stochastically identical Arrival process of the reversed chain is Poisson with rate The arrival epochs of the reversed chain are the d...