HW11s[1] - ECE 534 Elements of Information Theory Fall 2010...

Info icon This preview shows pages 1–3. Sign up to view the full content.

View Full Document Right Arrow Icon
ECE 534: Elements of Information Theory, Fall 2010 Homework 11 Solutions – all by Kenneth S. Palacio Baus November 17, 2010 1. Problem 10.14. Rate distortion for two independent sources . Can one compress two inde- pendent sources simultaneously better than by compressing the sources individually? The following problem addresses this question. Let { X i } be i.i.d. p ( x ) with distortion d ( x, ˆ x ) and rate distortion function R X ( D ). Similarly, let { Y i } be i.i.d. p ( y ) with distortion d ( y, ˆ y ) and rate distortion function R Y ( D ). Suppose we now wish to describe the process { ( X i , Y i ) } subject to distortions Ed ( X, ˆ X ) D 1 and Ed ( Y, ˆ Y ) D 2. Thus, a rate R X,Y ( D 1 , D 2) is sufficient, where R X,Y ( D 1 , D 2 ) = min p x, ˆ y | x,y ): Ed ( X, ˆ X ) D 1 ,Ed ( Y, ˆ Y ) D 2 I ( X, Y ; ˆ X, ˆ Y ) Now suppose that the { X i } process and the { Y i } process are independent of each other. (a) Show that: R X,Y ( D 1 , D 2 ) R X ( D 1 ) + R Y ( D 2 ) Solution: Given: R X,Y ( D 1 , D 2 ) = min p x, ˆ y | x,y ): Ed ( X, ˆ X ) D 1 ,Ed ( Y, ˆ Y ) D 2 I ( X, Y ; ˆ X, ˆ Y ) (1) Considering that { X i } and the { Y i } are independent of each other we have: I ( X, Y ; ˆ X, ˆ Y ) = H ( X, Y ) - H ( X, Y | ˆ X, ˆ Y ) (2) = H ( X ) + H ( Y ) - H ( X | ˆ X, ˆ Y ) - H ( Y | X, ˆ X, ˆ Y ) (3) = H ( X ) + H ( Y ) - H ( X | ˆ X ) - H ( Y | ˆ Y ) (4) H ( X ) - H ( X | ˆ X ) + H ( Y ) - H ( Y | ˆ Y ) (5) I ( X ; ˆ X ) + I ( Y ; ˆ Y ) (6) Now we obtain: R X,Y ( D 1 , D 2 ) = min p x, ˆ y | x,y ): Ed ( X, ˆ X ) D 1 ,Ed ( Y, ˆ Y ) D 2 I ( X, Y ; ˆ X, ˆ Y ) (7) min p x, ˆ y | x,y ): Ed ( X, ˆ X ) D 1 ,Ed ( Y, ˆ Y ) D 2 h I ( X ; ˆ X ) + I ( Y ; ˆ Y ) i (8) min p x | x ): Ed ( X, ˆ X ) D 1 I ( X ; ˆ X ) + min p y | y ): Ed ( Y, ˆ Y ) D 2 I ( Y ; ˆ Y ) (9) R X ( D 1 ) + R Y ( D 2 ) (10) 1
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
(b) Does equality hold? From part (a), we have that: R X,Y ( D 1 , D 2 ) R X ( D 1 ) + R Y ( D 2 ). Because of the independence assumed between processes { X i } and { Y i } we have that: p ( x, y, ˆ x, ˆ y ) = p x, ˆ y | x, y ) p ( x, y ) (11) = p x | x ) p y | y ) p ( x ) p ( y ) (12) = p ( x, ˆ x ) p ( y, ˆ y ) (13) Then, for distributions p ( x, ˆ x ) and p ( y, ˆ y ) achieving rate distortions R X ( D 1 ) and R Y ( D 2 ) we have that the mutual information of the product of the two distributions such that p ( x, y, ˆ x, ˆ y ) = p ( x, ˆ x ) p ( y, ˆ y ) gives: R X,Y ( D 1 , D 2 ) = min p x, ˆ y | x,y ): Ed ( X, ˆ X ) D 1 ,Ed ( Y, ˆ Y ) D 2 I ( X, Y ; ˆ X, ˆ Y ) (14) = R X ( D 1 ) + R Y ( D 2 ) (15) Hence, the equality holds.
Image of page 2
Image of page 3
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern