{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

Module2_2 - Module 2 Lecture 2 Fundamental Concepts Entropy...

Info icon This preview shows pages 1–8. Sign up to view the full content.

View Full Document Right Arrow Icon
Module 2, Lecture 2 Fundamental Concepts: Entropy, Relative Entropy, Mutual Information G.L. Heileman Module 2, Lecture 2
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Entropy Definition (Entropy) The entropy of a RV X p ( x ) is: H ( X ) = x ∈X p ( x ) log 1 p ( x ) = E log 1 p ( X ) Properties of H ( X ): 1 H ( X ) 0 with equality when p ( x ) = 1 for some one x ∈ X . 2 H ( X ) log( |X| ) with equality when p ( x ) = 1 |X| x ∈ X . 3 H b ( X ) = (log b 2) H ( X ). Proof: 1 0 p ( x ) 1 log 1 p ( x ) 0. 2 we’ll prove this momentarily. 3 log b c = (log b a )(log a c ). G.L. Heileman Module 2, Lecture 2
Image of page 2
Binary Entropy Function p 0 1/2 1 0 1 H(p,1-p) G.L. Heileman Module 2, Lecture 2
Image of page 3

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Entropy Theorem H ( X ) log( |X| ) with equality when p ( x ) = 1 |X| x ∈ X . Proof: Consider H ( 1 n , . . . , 1 n ) - H ( p 1 , . . . , p n ) where the p i ’s are arbitrary probabilities satisfying the probability axioms. We will show that the above quantity is 0 for any valid choices of the p i ’s. To do this, rewrite the previous equation as: log b n + n i =1 p i log b p i = log b e · log e n + n i =1 p i (log e p i · log b e ) = log b e · (ln n + n i =1 p i ln p i ) G.L. Heileman Module 2, Lecture 2
Image of page 4
Entropy Proof (con’t): = log b e · (ln n · n i =1 p i + n i =1 p i ln p i ) = log b e · n i =1 p i ln( np i ) . Now, using the fact that ln 1 x 1 - x x > 0, we can write H 1 n , . . . , 1 n - H ( p 1 , . . . , p n ) log b e · n i =1 p i 1 - 1 np i = log b e · n i =1 p i - n i =1 1 n = 0 . Thus, equiprobable events produce maximum entropy. G.L. Heileman Module 2, Lecture 2
Image of page 5

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Joint Entropy Definition (Joint Entropy) The joint entropy of a pair of RVs ( X , Y ) p ( x , y ) is: H ( X , Y ) = x ∈X y ∈Y p ( x , y ) log 1 p ( x , y ) = E log 1 p ( X , Y ) When X and Y are independent, p ( x , y ) = p ( x ) p ( y ), and H ( X , Y ) = x , y p ( x ) p ( y ) log 1 p ( x ) + log 1 p ( y ) = y p ( y ) x p ( x ) log 1 p ( x ) + x p ( x ) y p ( y ) log 1 p ( y ) = H ( X ) + H ( Y ) G.L. Heileman Module 2, Lecture 2
Image of page 6
Conditional Entropy The conditional entropy of X given that Y = y is observed can be written as: H ( X | Y = y ) = x p ( x | Y = y ) log 1 p ( x | Y = y ) I.e., this is the entropy of the probability distribution p ( x | Y = y ).
Image of page 7

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Image of page 8
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern