# lecture-28 - Chapter 28 Shannon Entropy and...

This preview shows pages 1–3. Sign up to view the full content.

Chapter 28 Shannon Entropy and Kullback-Leibler Divergence Section 28.1 introduces Shannon entropy and its most basic prop- erties, including the way it measures how close a random variable is to being uniformly distributed. Section 28.2 describes relative entropy, or Kullback-Leibler di- vergence, which measures the discrepancy between two probability distributions, and from which Shannon entropy can be constructed. Section 28.2.1 describes some statistical aspects of relative entropy, especially its relationship to expected log-likelihood and to Fisher information. Section 28.3 introduces the idea of the mutual information shared by two random variables, and shows how to use it as a measure of serial dependence, like a nonlinear version of autocovariance (Section 28.3.1). Information theory studies stochastic processes as sources of information, or as models of communication channels. It appeared in essentially its modern form with Shannon (1948), and rapidly proved to be an extremely useful mathe- matical tool, not only for the study of “communication and control in the animal and the machine” (Wiener, 1961), but more technically as a vital part of prob- ability theory, with deep connections to statistical inference (Kullback, 1968), to ergodic theory, and to large deviations theory. In an introduction that’s so limited it’s almost a crime, we will do little more than build enough theory to see how it can fit in with the theory of inference, and then get what we need to progress to large deviations. If you want to learn more (and you should!), the deservedly-standard modern textbook is Cover and Thomas (1991), and a good treatment, at something more like our level of mathematical rigor, is Gray 189

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
CHAPTER 28. ENTROPY AND DIVERGENCE 190 (1990). 1 28.1 Shannon Entropy The most basic concept of information theory is that of the entropy of a random variable, or its distribution, often called Shannon entropy to distinguish it from the many other sorts. This is a measure of the uncertainty or variability asso- ciated with the random variable. Let’s start with the discrete case, where the variable takes on only a finite or countable number of values, and everything is easier. Definition 356 (Shannon Entropy (Discrete Case)) The Shannon entropy , or just entropy , of a discrete random variable X is H [ X ] ≡ - x P ( X = x ) log P ( X = x ) = - E [log P ( X )] (28.1) when the sum exists. Entropy has units of bits when the logarithm has base 2, and nats when it has base e . The joint entropy of two random variables, H [ X, Y ] , is the entropy of their joint distribution. The conditional entropy of X given Y , H [ X | Y ] is H [ X | Y ] y P ( Y = y ) x P ( X = x | Y = y ) log P ( X = x | Y = y )(28.2) = - E [log P ( X | Y )] (28.3) = H [ X, Y ] - H [ Y ] (28.4) Here are some important properties of the Shannon entropy, presented with- out proofs (which are not hard). 1. H [ X ] 0 2. H [ X ] = 0 i ff x 0 : X = x 0 a.s.
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### What students are saying

• As a current student on this bumpy collegiate pathway, I stumbled upon Course Hero, where I can find study resources for nearly all my courses, get online help from tutors 24/7, and even share my old projects, papers, and lecture notes with other students.

Kiran Temple University Fox School of Business ‘17, Course Hero Intern

• I cannot even describe how much Course Hero helped me this summer. It’s truly become something I can always rely on and help me. In the end, I was not only able to survive summer classes, but I was able to thrive thanks to Course Hero.

Dana University of Pennsylvania ‘17, Course Hero Intern

• The ability to access any university’s resources through Course Hero proved invaluable in my case. I was behind on Tulane coursework and actually used UCLA’s materials to help me move forward and get everything together on time.

Jill Tulane University ‘16, Course Hero Intern