15-359: Probability and Computing
Fall 2009
Lecture 25: Elements of Information Theory
Information Theory is a major branch of applied mathematics, studied by electrical engineers,
computer scientists, and mathematicians among others. Almost everyone agre
1. Model of information communication and noisy channel
To quote Shannon from his paper A Mathematical theory of communication: The fundamental
problem of communication is that of reproducing at one point either exactly or approximately a
message selected
Probability and Computing
CMU 15-359, Fall 2010
Homework 11
Due: Tuesday, November 30, beginning of class
1. Too much coee. One shot of espresso keeps you caeinated for an amount of time which has mean
30 minutes and standard deviation 6 minutes. As soon
Probability and Computing
CMU 15-359, Fall 2010
Homework 9
Due: Thursday, November 11, beginning of class
There are six problems, including a bonus problem.
1. Independence.
(i) Show that continuous random variables X and Y are independent if and only if
Probability and Computing
CMU 15-359, Fall 2010
Homework 10
Due: Thursday, November 18, beginning of class
There are ve problems.
1. CTMC basics.
Consider a CTMC on 3 states with Q-matrix
4 2
2
0
1 1
1
1
2
where the rows 1, 2, 3 correspond to states 1,
CMU 15-359: Probability and Computing, Fall 2010
Practice Final Exam
125 total points
Part 0: Please write your name. [1 point]
Part I: Short answer. [3 points each, 24 total]
Problems a) d) below refer to the following experiment:
X Exponential(1)
Y Expo
Probability and Computing
CMU 15-359, Fall 2010
H OMEWORK 5
Due: Friday, October 8, by 5pm
1. Randomized algorithms. A randomized algorithm is designed to solve a certain hard problem, for
which the answer is either yes or no. We know that if we run our a
Probability and Computing
CMU 15-359, Fall 2010
Homework 4
Due: Thursday, September 30, beginning of class
There are seven problems, including a bonus problem.
1. Calculate. An OCR program scans a 356-page book. Assume that on page i it makes Xi mistakes,
Probability and Computing
CMU 15-359, Fall 2010
Homework 8
Due: Thursday, November 4, beginning of class
1. Use the CDF!
a) Let X and Y be continuous random variables. Suppose we now dene a random variable Z as follows:
with probability p we set Z = X; ot
Probability and Computing
CMU 15-359, Fall 2010
Homework 7
Due: Thursday, October 28, beginning of class
1. Practice, practice, practice. In this problem we consider 5 dierent Markov Chains. Their transition
matrices are as follows:
2
1 2
1
0 0
0 0 3
0 2
Probability and Computing
CMU 15-359, Fall 2010
Homework 6
Due: Thursday, October 21, beginning of class
1. Triangles in a random graph, part 1. A triangle in a graph is a clique of size 3. Consider a graph
G drawn from the ErdsRnyi Gn,p model. In this pr
Probability and Computing
CMU 15-359, Fall 2010
Homework 3
Due: Thursday, September 23, beginning of class
1. Calculation.
Consider the following experiment:
X (RandInt(8) mod 3)
1
Y Geometric( X+2 )
(b) Give the joint PMF of X and Y .
Also, write down th
Probability and Computing
CMU 15-359, Fall 2010
Homework 2
Due: Tuesday, September 14, beginning of class
1. Bayes Rule. You train a learning algorithm to detect spam email messages and decide to test it on
your own andrew account. The algorithms false ne
Probability and Computing
CMU 15-359, Fall 2010
Homework 1
Due: Thursday, September 2, beginning of class
1. Practice, practice, practice. I ip a fair coin (Bernoulli(1/2). If it comes up heads, I roll a
3-sided die and a 4-sided die. If it comes up tails