10-704: Information Processing and Learning
Spring 2012
Homework 3: Solution
Lecturer: Aarti Singh
Note: The TA graciously thank Rafael Stern for providing all of these solutions
3.1
3.1.1
Problem 1
(a)
If p 0.5, X = Y and, if p > 0.5, X = (1 Y ). The pro

Homework 1
10-704 Information Processing and Learning
Instructor: Aarti Singh
The HW is worth 40 pts and is due on Feb 3 at noon. Hand in to: Michelle Martin
GHC 8001. If she is not around, note down the time on your HW sheet and slide it under
her door.

Homework 2
10-704 Information Processing and Learning
Instructor: Aarti Singh
The HW is worth 50 pts and is due on Feb 20 at noon. Hand in to: Michelle Martin
GHC 8001. If she is not around, note down the time on your HW sheet and slide it under
her door.

Homework 3
10-704 Information Processing and Learning
Instructor: Aarti Singh
The HW is worth 50 pts and is due on Mar 9 at noon. Hand in to: Michelle Martin
GHC 8001. If she is not around, note down the time on your HW sheet and slide it under
her door.

10-704: Information Processing and Learning
Spring 2012
Lecture 3: Fanos, Dierential Entropy, Maximum Entropy Distributions
Lecturer: Aarti Singh
Scribes: Min Xu
Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal pub

10-704: Information Processing and Learning
Spring 2012
Lecture 2: Gibbs, Data Processing and Fanos Inequalities
Lecturer: Aarti Singh
Scribes: Min Xu
Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.

10-704: Information Processing and Learning
Spring 2012
Lecture 6: Source coding, AEP and Typicality
Lecturer: Aarti Singh
Scribes: Mark McCartney
Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
They

10-704: Information Processing and Learning
Spring 2012
Lecture 5: Burgs Maximum Entropy Theorem
Lecturer: Aarti Singh
Scribe: Ina Fiterau
Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
They may be

10-704: Information Processing and Learning
Spring 2012
Lecture 4: Maximum Entropy Distributions and Exponential Family
Lecturer: Aarti Singh
Scribes: Min Xu
Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publica

10-704: Information Processing and Learning
Spring 2012
Homework 1: Solution
Lecturer: Aarti Singh
Acknowledgement: The TA graciously thanks Rafael Stern for providing most of the solutions.
1.1
1.1.1
Problem 1
1.1
Weighing 6 balls vs 6 balls yields only

10-704: Information Processing and Learning
Spring 2012
Homework 4: Solution
Lecturer: Aarti Singh
Note: The TA graciously thank Rafael Stern for providing most of these solutions
4.1
4.1.1
Problem 1
(a)
Recall that a tree is uniquely identied by the set

10-704: Information Processing and Learning
Spring 2012
Homework 2: Solution
Lecturer: Aarti Singh
Acknowledgement: The TA graciously thanks Rafael Stern for providing most of these solutions.
2.1
Problem 1
D(q|p) =
q
p
q log
dx
Hence,
q
p
D(q|p) = 1 + lo

10-704: Information Processing and Learning
Spring 2012
Lecture 1: Introduction, Entropy and ML estimation
Lecturer: Aarti Singh
Scribes: Min Xu
Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
They m

Homework 5
10-704 Information Processing and Learning
Instructor: Aarti Singh
The HW is worth 40 (+5 bonus) pts and is due on May 9 at noon. Hand in to:
Michelle Martin GHC 8001. If she is not around, note down the time on your HW sheet and
slide it under

Homework 4
10-704 Information Processing and Learning
Instructor: Aarti Singh
The HW is worth 40 pts and is due on April 20 at noon. Hand in to: Michelle Martin
GHC 8001. If she is not around, note down the time on your HW sheet and slide it under
her doo