# hw3 - X n i.e ˆ X n = n-1 X i =1 b i X n-i Let F be the...

This preview shows pages 1–2. Sign up to view the full content.

EE 376B/Stat 376B Handout #8 Information Theory Thursday, April 20, 2006 Prof. T. Cover Due Thursday, April 27, 2006 Homework Set #3 1. Maximum entropy discrete processes. (a) Find the maximum entropy rate binary stochastic process { X i } i = -∞ , X i ∈ { 0 , 1 } , satisfying Pr { X i = X i +1 } = 1 3 , for all i . (b) What is the resulting entropy rate? 2. Maximum entropy of sums. Let Y = X 1 + X 2 . Find the maximum entropy (over all distributions on X 1 and X 2 ) of Y under the constraint EX 2 1 = P 1 , EX 2 2 = P 2 , (a) if X 1 and X 2 are independent. (b) if X 1 and X 2 are allowed to be dependent. 3. Estimation. Here is the estimation counterpart to Fano’s inequality. Let X be a random variable with diﬀerential entropy h ( X ). Let ˆ X be an estimate of X , and let E ( X - ˆ X ) 2 be the expected prediction error. (a) Show E ( X - ˆ X ) 2 1 2 πe e 2 h ( X ) . (b) Given side information Y and estimator ˆ X ( Y ), show E ( X - ˆ X ( Y )) 2 1 2 πe e 2 h ( X | Y ) . 1

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
4. Mean squared error. Let { X i } n i =1 satisfy EX i X i + k = R k , k = 0 , 1 ,...,p . Consider linear estimators for
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: X n , i.e., ˆ X n = n-1 X i =1 b i X n-i . Let F be the set all densities f ( x n ) satisfying R ,R 1 ,...,R p . Assume n > p . Find max f ( x n ) min b E ( X n-ˆ X n ) 2 . This identiﬁes the process which is hardest to estimate from the past. 5. Hadamard. Let K be a 2 n × 2 n nonnegative deﬁnite symmetric matrix. Show det( K ) ≤ n Y i =1 det( K (2 i-1 , 2 i )) , where K ( i,j ) denotes the 2 × 2 submatrix ± K ii K ij K ji K jj ¶ . 6. Maximum entropy. (a) What is the parametric form maximum entropy density f ( x ) satisfying the two conditions EX 8 = a EX 16 = b. Don’t solve for the λ ’s. (b) What is the maximum entropy density satisfying the condition E ( X 8 + X 16 ) = a + b ? 2...
View Full Document

{[ snackBarMessage ]}

### Page1 / 2

hw3 - X n i.e ˆ X n = n-1 X i =1 b i X n-i Let F be the...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online