Note: The solution to this problem is from Section 9.7 of Cover & Thomas (First Edition).As in the hint, we haveY=X+U, and therefore the distribution ofYhas the shape ofa histogram (that is,fY(y) =pifori≤y < i+ 1). It is clear thatH(X) =H(X), since3

discrete entropy depends only on the probabilities and not on the values of the outcomes.NowH(X)=-∞i=ipilog2pi=-∞i=ii+1ifY(y)dylog2i+1ifY(y)dy=-∞i=ii+1ifY(y) log2fY(y)dy=-∞1fY(y) log2fY(y)dy=h(Y),sincefY(y) =pifori≤y < i+ 1.Hence we have the following chain of inequalities:H(X)=H(X)=h(Y)≤12log2(2πe)(Var(Y))=12log2(2πe)(Var(X ) + Var(U))=12log2(2πe)∞i=1pii2-∞i=1pii2+112.Since entropy is invariant with respect to permutation ofp1, p2, . . ., we can also obtain a boundby a permutation of thepi’s. We conjecture that a good bound on the variance will be achievedwhen the high probabilities are close together, i.e., by the assignment. . . , p5, p3, p1, p2, p4, . . .forp1≥p2≥. . ..8.8Channels with uniformly distributed noise.C=maxp(x)I(X;Y)=maxp(x)h(Y)-h(Y|X)=maxp(x)h(Y)-h(X+Z|X)=maxp(x)h(Y)-h(Z)=maxp(x)h(Y)-log22,where in the last line we have used the fact that the differential entropy of a random variablethat is distributed uniformly betweenαandα+ais log2abits.4

Furthermore, we see that the output of the channel,Y, is limited to values in the range [-3,3].From a result on distributions with maximum entropy (specifically, see Chapter 12, Example12.2.4), we see thath(Y) will be maximized if we selectp(x) such that the distribution ofYis uniform in the range [-3,3].Now, forp(x= 0) =p(x= 2) =p(x=-2) =13,it is easy to see that the distribution ofYis uniform in the range [-3,3], and from ourprevious discussion,h(Y) = log26.Therefore, we haveC= log26-log22 = log23.Note that one could have arrived at this result using calculus (without explicit knowledgethat the uniform distribution maximizes entropy for a variable with bounded range), but itwould have involved lengthy (and tedious) calculations.

#### You've reached the end of your free preview.

Want to read all 6 pages?

- Spring '10
- sd
- Normal Distribution, Probability theory, Yi, max H