Lecture 9 - ECE 771 Lecture 9 Dierential entropy Objective...

Info icon This preview shows pages 1–2. Sign up to view the full content.

ECE 771 Lecture 9 – Differential entropy Objective: Differential entropy is entropy defined for distributions with a con- tinuous random variable. We will learn about differential entropy. Reading: 1. Read chapter 9. Differential entropy A continuous random variable (for the purpose of this class) is one in which the distribution function F ( x ) is continuous: there are no jumps (discrete outputs). Definition 1 The differential entropy h ( X ) of a continuous random variable X with density f ( x ) and support S is h ( X ) = - Z S f ( x ) log f ( x ) dx. 2 Example 1 Let X ∼ U (0 , a ). (Uniform). Then h ( X ) = - Z a 0 1 a log 1 a dx = log a. Note that the differential entropy can be negative (if a < 1). This is why we refer to this as differential entropy, since entropy should always be positive. 2 Example 2 Normal: X 1 σ 2 π e - x 2 / 2 σ 2 = φ ( x ) . (We have zero mean, but that will make no difference on the entropy.) We will compute the entropy in nats. h ( X ) = - Z φ ln φdx = - Z φ ( x )[ - x 2 2 - ln 2 πσ 2 ] dx = EX 2 2 σ 2 + 1 2 ln 2 πσ 2 = 1 2 ln πeσ 2 nats 2 Having defined the differential entropy, we can now go through and define all the sorts of things we did before.
Image of page 1

Info icon This preview has intentionally blurred sections. Sign up to view the full version.

Image of page 2
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

What students are saying

  • Left Quote Icon

    As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

    Student Picture

    Kiran Temple University Fox School of Business ‘17, Course Hero Intern

  • Left Quote Icon

    I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

    Student Picture

    Dana University of Pennsylvania ‘17, Course Hero Intern

  • Left Quote Icon

    The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

    Student Picture

    Jill Tulane University ‘16, Course Hero Intern