MS&E 221
CA: Erick Delage
Problem Set 1 Solutions
January 24, 2007
Problem 1 This is a variant of the famous Monty Hall problem. Let EA , EB and EC denote the
events A is to be executed, B is to be executed and C is to be executed respectively. Let SB
den

MS&E 221
Ramesh Johari
Problem Set 1
Due: January 21, 2016, 5:00 PM, in the basement of Huang Eng. Ctr.
Problem 1 (Probability review). (From Ross, Chapter 3) An individual traveling on the real
line is trying to reach the origin. However, the larger the

MS&E 221 Ramesh Johari
Problem Set 1 Due: January 19, 2011, 5:00 PM, in the basement of Huang Eng. Ctr.
Reading. Read Sections 4.1, 4.2, 4.3, and 4.5.1 in Ross. Problem 1 (Probability review). Ross, Chapter 3, problem 45:
Problem 2 (Probability review). I

MS&E 221
CA: Erick Delage
Problem Set 4 Solutions
February 28, 2007
Problem 1
1. Let Sn be the set of all permutations of cfw_1, . . . , n. Note that |Sn | = n!. If Sn we denote
by (j ) the j -th element of . For example, if n = 6 and = (2, 3, 1, 5, 4, 6)

MS&E 221
Ramesh Johari
Problem Set 1
Due: January 21, 2016, 5:00 PM, in the basement of Huang Eng. Ctr.
Reading. Read Sections 4.1, 4.2, 4.3, and 4.5.1 in Ross.
Problem 1 (Probability review). Ross, Chapter 3, problem 45: An individual traveling on
the re

MS&E 221 Midterm Examination
Ramesh Johari February 14, 2007
Instructions
1. Take alternate seating.
2. Answer all questions in the spaces provided on these sheets. If needed, additional paper will
be available at the front of the room. Answers give

MS&E 221
Ramesh Johari
Problem Set 2
Due: February 7, 2007, 5:00 PM outside Terman 319
Reading. Same as last week: read Section 4.4 in Ross.
Problem 1. Bertsekas and Tsitsiklis, Chapter 6, Problem 11 (note that steady state means
the current distribution

MS&E 221
CA: Erick Delage
Problem Set 2 Solutions
January 31, 2007
Problem 1 First, well identify the communicating classes then for each class we want to check
if it is closed or not. Since the classes will be nite for all the examples below, the class w

MS&E 221 Ramesh Johari
Problem Set 2 Due: Weds., February 2, 2011, 5:00 PM in the basement of HEC
Reading. 4.1-4.4, 4.5.1, 4.7 in Ross. Problem 1. Ross, Chapter 4, problem 14:
Problem 2. (A queueing model) Consider a queue (or a waiting room) that can hol

MS&E 221
Ramesh Johari
Problem Set 4
Due: February 28, 2007, 5:00 PM outside Terman 319
Reading. Read Sections 4.8-4.9 and 5.1-5.3 in Ross.
Problem 1. Ross, Chapter 4, problem 74:
1
Problem 2. DELETED to be assigned on Problem Set 5
Problem 3. (Gibbs samp

MS&E 221
CA: Erick Delage
Problem Set 5 Solutions
March 7, 2007
Problem 1
1.
P cfw_L1 = 0 = em
2. For 0 < x < m,
P cfw_L1 < x = e(mx)
3.
P cfw_R1 = 1 = e(1m)
4. For m < x < 1,
P cfw_R1 > x = e(xm)
5.
1
E [R] =
1
P (R > x)dx = m +
0
P (R > x)dx
m
1
= m+
en

MS&E 221
Ramesh Johari
Problem Set 1
Due: January 24, 2007, 5:00 PM outside Terman 319
Reading. Read Sections 4.1, 4.2, 4.3, and 4.5.1 in Ross.
Problem 1 (OPTIONAL; probability review). Ross, Chapter 1, problem 46:
Problem 2 (OPTIONAL; probability review)

MS&E 221
Ramesh Johari
Problem Set 2
Due: January 31, 2007, 5:00 PM outside Terman 319
Reading. Read Section 4.4 in Ross.
Problem 1. Ross, Chapter 4, problem 14:
Problem 2. Ross, Chapter 4, problem 57:
Problem 3. A town is planning to protect itself from

Problem 1 (Further Practice on Modeling) Ross, Chapter 4, Problem 22:
Let Yn be the sum of n independent rolls of a fair die. Find
lim Pcfw_Yn is a multiple of 13.
n
Solution: Let Xn = Yn %13 be the remainder when Yn is divided by 13. Then cfw_Xn : n 0
is

MS&E 221: Stochastic Modeling
Lecture 8: A law of large numbers for Markov chains
Ramesh Johari
[email protected]
Winter 2016
1 / 37
Standard LLN
Recall the standard law of large numbers:
If X1 , X2 , . . . are i.i.d. with mean , then:
Pn
i=1 Xi

MS&E 221: Stochastic Modeling
Lecture 9: Class decomposition of Markov chains
Ramesh Johari
[email protected]
Winter 2016
1 / 40
Summarizing what weve learned
In this set of lecture notes we combine and synthesize what weve
learned so far:
I
Clas

Problem 3 (22 points).
In this problem we explore PageRank. (Note: The problem is entirely self-contained, and does
not require anything you learned in the lecture notes on PageRank.)
PageRank is an algorithm for ranking web pages. It works as follows: Le

MS&E 221: Stochastic Modeling
Lecture 9: Class decomposition of Markov chains
Ramesh Johari
[email protected]
1 / 40
Summarizing what weve learned
In this set of lecture notes we combine and synthesize what weve
learned so far:
I
Class decomposit

Problem 3. Suppose M customers of a cable company are served by N on-demand streaming
video servers, where M > N > 0; for simplicity assume that each server can serve exactly one
customer at a time. If a customer attempts to watch a video when all servers

MS&E 221: Stochastic Modeling
Q
Lecture 11: Markov chain Monte Carlo
Ramesh Johari
[email protected]
Winter 2016
1 / 36
Monte Carlo
2 / 36
Monte Carlo estimation
Suppose we have a method to generate i.i.d. copies of a random
variable X.
If we gen

MS&E 221: Stochastic Modeling
Lecture 11: Simulation
Ramesh Johari
[email protected]
1/7
What is simulation?
Simulation refers to computational procedures that generate
samples of random variables according to their distribution.
In this lecture

MS&E 221: Stochastic Modeling
Lecture 7: Limiting behavior of finite state Markov chains
Ramesh Johari
[email protected]
1 / 20
Finite state space
Throughout this lecture we assume a finite state space:
X = cfw_1, 2, . . . , N .
2 / 20
Perron-Fro

MS&E 221: Stochastic Modeling
Lecture 8: A law of large numbers for Markov chains
Ramesh Johari
[email protected]
1 / 37
Standard LLN
Recall the standard law of large numbers:
If X1 , X2 , . . . are i.i.d. with mean , then:
Pn
i=1 Xi
n
as n .1
Ca

MS&E 221: Stochastic Modeling
Lecture 2: Basic calculations
Ramesh Johari
[email protected]
1 / 20
Marginal distributions
Suppose that cfw_Xn is a Markov chain with initial distribution
and transition matrix P . What is the distribution of X1 ?

MS&E 221: Stochastic Modeling
Lecture 5: Recurrence, transience, and the strong Markov
property
Ramesh Johari
[email protected]
1 / 22
The strong Markov property (SMP)
Suppose X0 , X1 , . . . is a Markov chain.
Let Ti be the first time the chain

MS&E 221: Stochastic Modeling
Lecture 4: Birth-and-death chains
Ramesh Johari
[email protected]
1 / 14
The birth-and-death chain
Consider a chain with state space cfw_0, . . . , N where:
P (i, i + 1) = p, i = 1, . . . , N 1;
P (i, i 1) = q, i =