15-359: Probability and Computing
Fall 2009
Lecture 25: Elements of Information Theory
Information Theory is a major branch of applied mathematics, studied by electrical engineers,
computer scientists
1. Model of information communication and noisy channel
To quote Shannon from his paper A Mathematical theory of communication: The fundamental
problem of communication is that of reproducing at one p
Probability and Computing
CMU 15-359, Fall 2010
Homework 11
Due: Tuesday, November 30, beginning of class
1. Too much coee. One shot of espresso keeps you caeinated for an amount of time which has mea
Probability and Computing
CMU 15-359, Fall 2010
Homework 9
Due: Thursday, November 11, beginning of class
There are six problems, including a bonus problem.
1. Independence.
(i) Show that continuous r
Probability and Computing
CMU 15-359, Fall 2010
Homework 10
Due: Thursday, November 18, beginning of class
There are ve problems.
1. CTMC basics.
Consider a CTMC on 3 states with Q-matrix
4 2
2
0
1 1
CMU 15-359: Probability and Computing, Fall 2010
Practice Final Exam
125 total points
Part 0: Please write your name. [1 point]
Part I: Short answer. [3 points each, 24 total]
Problems a) d) below ref
Probability and Computing
CMU 15-359, Fall 2010
H OMEWORK 5
Due: Friday, October 8, by 5pm
1. Randomized algorithms. A randomized algorithm is designed to solve a certain hard problem, for
which the a
Probability and Computing
CMU 15-359, Fall 2010
Homework 4
Due: Thursday, September 30, beginning of class
There are seven problems, including a bonus problem.
1. Calculate. An OCR program scans a 356
Probability and Computing
CMU 15-359, Fall 2010
Homework 8
Due: Thursday, November 4, beginning of class
1. Use the CDF!
a) Let X and Y be continuous random variables. Suppose we now dene a random var
Probability and Computing
CMU 15-359, Fall 2010
Homework 7
Due: Thursday, October 28, beginning of class
1. Practice, practice, practice. In this problem we consider 5 dierent Markov Chains. Their tra
Probability and Computing
CMU 15-359, Fall 2010
Homework 6
Due: Thursday, October 21, beginning of class
1. Triangles in a random graph, part 1. A triangle in a graph is a clique of size 3. Consider a
Probability and Computing
CMU 15-359, Fall 2010
Homework 3
Due: Thursday, September 23, beginning of class
1. Calculation.
Consider the following experiment:
X (RandInt(8) mod 3)
1
Y Geometric( X+2 )
Probability and Computing
CMU 15-359, Fall 2010
Homework 2
Due: Tuesday, September 14, beginning of class
1. Bayes Rule. You train a learning algorithm to detect spam email messages and decide to test
Probability and Computing
CMU 15-359, Fall 2010
Homework 1
Due: Thursday, September 2, beginning of class
1. Practice, practice, practice. I ip a fair coin (Bernoulli(1/2). If it comes up heads, I rol