ECE 534: Elements of Information Theory, Fall 2010
Homework 1
Solutions
Ex. 2.1 (Davide Basilio Bartolini)
Text
Coin Flips. A fair coin is ipped until the rst head occurs. Let X denote the number of ips
required.
(a) Find the Entropy H(X) in bits
(b) A ra
ECE 534: Elements of Information Theory, Fall 2010
Homework: 8
Solutions
Exercise 8.9 (Johnson Jonaris GadElkarim)
Gaussian mutual information. Suppose that (X, Y, Z ) are jointly Gaussian and that X Y Z
forms a Markov chain. Let X and Y have correlation
12
Entropy, Relative Entropy and Mutual Information
since t log t 0 for 0 t 1 , and is strictly p ositive for t not equal to 0 or 1.
Therefore the conditional entropy H (Y |X ) is 0 if and only if Y is a function of X .
6. Conditional mutual information v
ECE 534: Elements of Information Theory, Fall 2011
Homework 10
Solutions - all by Xiaokai Wei
Problem 15.1
(a) When senders have access to both indices, the channel can be considered as a channel with
input alphabet X1 X2 and message set W1 W2 . The capac
ECE 534: Elements of Information Theory, Fall 2011
Homework 8
Solutions - ALL by Yi Xie
Problem 1 (10.2)
The rate distortion function is given:
R(D ) =
p(|x):
x
min
P
(x,x)
p(x)p(|x)d(x,x)D
x
I (X ; X )
Also, we have d(0, 1) = . So, p(0, 1) must be 0 for
ECE 534: Elements of Information Theory, Fall 2011
Homework 7
Solutions
Problem 8.1 (Xiaokai Wei)
(a)
h(X ) =
f ln f dx
(1)
ex (ln x) dx
=
(2)
0
= ln + ln e
e
= ln (nat)
e
= log (bit)
(3)
f ln f dx
(6)
(4)
(5)
(b)
h(X ) =
1 x
1
e
(ln + ln x) dx
2
2
1
e
ECE 534: Elements of Information Theory, Fall 2011
Homework 6
Solutions
Problem 7.13 (Xiaokai Wei)
(a) By symmetry (despite the channel not being weakly symmetric, it should be clear, or can
alternatively be optimized for), the p(x) that maximizes I (X ;
ECE 534: Elements of Information Theory, Fall 2011
Homework 5
Solutions
1. Problem 5.42 (Hamid Dadkhahi)
(a). No. Consider the tree representation of the code with the given lengths, as given below.
Obviously, the second branch can be shortened by one. Th
ITMD 534: Human Computer
Interaction
Subhashish Ghosh
Lecture 1
Spring, 2016
1
Illinois Institute of Technology
Course:
ITMD 534: Human Computer Interaction
Mondays, 6:25pm 9:05pm
All lectures videotaped
All the slides used in the lectures will be
shared
ECE 534: Elements of Information Theory, Fall 2011
Homework 3
Solutions
Problem 3.7 (Xiaokai Wei)
(a) The number of 100-bit sequence containing less than or equal to 3 ones is
3
i=0
100
i
100
100
100
100
+
+
+
0
1
2
3
=
= 166751
So, the necessary code len
ECE 534: Elements of Information Theory, Fall 2011
Homework 2
Solutions
1. 2.23 (Nan Xu)
There is a sequence of n binary random variables X1 , X2 , ., Xn . Every sequence of length n with
an even number of 1s is equally likely and has probability 2(n1) .
ECE 534: Elements of Information Theory, Fall 2011
Homework 1
Solutions
Problem 2.2 (Hamid Dadkhahi) Let Y = g (X ). Suppose PX (x) and PY (y ) denote the probability mass functions of random variables X and Y , respectively. If g is an injective mapping,
1
ECE 534: Elements of Information Theory
Midterm Exam I (Spring 2007)
Problem 1 [20 pts]
What are the relations (, =, ) between the following pairs of expressions? Explain why.
1) H (5X ) and H (X );
2) H (X0 |X1 ) and H (X0 |X1 , X1 );
3) H (X1 , X2 , .
1
ECE 534: Elements of Information Theory
Solutions to Midterm Exam (Spring 2006)
Problem 1 [20 pts.]
A discrete memoryless source has an alphabet of three letters, xi , i = 1, 2, 3, with probabilities 0.4, 0.4,
and 0.2, respectively.
(a) Find the binary
ECE 534 Information Theory - Midterm 2
Nov.4, 2009. 3:30-4:45 in LH103.
You will be given the full class time: 75 minutes. Use it wisely! Many of the problems have short
answers; try to nd shortcuts.
You may bring and use two 8.5x11 double-sided crib sh
University of Illinois at Chicago
Department of Electrical and Computer Engineering
ECE 534: Information Theory
Fall 2009
Midterm 1 - Solutions
NAME:
This exam has 4 questions, each of which is worth 15 points.
You will be given the full class time: 75
ECE 534 Information Theory
Daniela Tuninetti
UIC Fall Semester 2015
August 24 2015
HW2: Chapter 2
Each problem will be given a grade on a scale from 0 to 2. If your solution is
entirely correct, you get 2 point. If your solution is more than 50% correct o
Each problem will be given a grade on a scale from 0 to 2. If your solution is
entirely correct, you get 2 point. If your solution is more than 50% correct on a
single-part problem, or if you solve at least half the parts entirely correctly for
a multi-pa
ECE 534 Information Theory
Daniela Tuninetti
UIC Fall Semester 2014
August 24 2014
HW3: Chapter 3
Each problem will be given a grade on a scale from 0 to 2. If your solution is
entirely correct, you get 2 point. If your solution is more than 50% correct o
ECE 534: Elements of Information Theory
Daniela Tuninetti,
Fall 2015
danielat@uic.edu
Chapter 8 outline
Motivation
Denitions
Relation to discrete entropy
Joint and conditional dierential entropy
Relative entropy and mutual information
Properties
AE
ECE 534: Elements of Information Theory
Daniela Tuninetti,
Fall 2015
danielat@uic.edu
Roadmap
So far we studied chapters 1 (Introduction), 2 (denitions and properties), 3
(typicality and lossless source coding for iid sources) and 4 (extensions to
source
ECE 534: Elements of Information Theory
Daniela Tuninetti,
Fall 2014
danielat@uic.edu
Chapter 3
Strong vs. Weak Typicality
Convergence and Law of Large Numbers (LLN)
Asymptotic Equipartition Property (AEP) Theorem
High-probability set vs. typical set
ECE 534: Elements of Information Theory
Daniela Tuninetti,
Fall 2015
danielat@uic.edu
Instructor
PhD in 2002 in IT, and most part of my work ever since
Published 130+ papers in IT conferences and journals
2006-2009 Editor-in-chief of the IEEE Informati
3.6
Example: exercise 3.13:
% m file
clear all, close all, clc
p = 0.6,
n = 25,
epsilonvar = 0.1,
deltavar
= epsilonvar,
for k = 0:n
numk (k+1) = k;
binnk(k+1) = nchoosek(n,k);
probk(k+1) = p^(k)*(1-p)^(n-k);
end
entk = -1/n*log2(probk);
totprobk = binnk.
ECE 534: Elements of Information Theory
Daniela Tuninetti,
Fall 2014
danielat@uic.edu
Addition to chapter 3
Strong Typicality
Example
Joint strong typicality
Conditional strong typicality
Weak typicality: recap
Let cfw_Xi , i 2 N iid pX (). Then, by
ECE 534 Information Theory
Daniela Tuninetti
UIC Fall Semester 2014
December ?, 2014
Final
You have 90 minutes to complete this exam.
There are 90 points for this exam. Points for the individual problems are marked in the problem
statement.
If somethin
ECE 534 Information Theory
Daniela Tuninetti
UIC Fall Semester 2014
October 15, 2014
Midterm
You have 75 minutes to complete this exam.
There are 60 points for this exam. Points for the individual problems are marked in the problem
statement.
If someth
ECE 534 Information Theory
Daniela Tuninetti
UIC Fall Semester 2015
September 30th, 2015
Midterm
This exam is composed of 8 short-answer problems for a total of 80 points.
You have 105 minutes to complete this exam. Use your time wisely.
Good luck!
Name a