This preview shows pages 1–3. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.View Full Document
Unformatted text preview: Chapter 28 Shannon Entropy and Kullback-Leibler Divergence Section 28.1 introduces Shannon entropy and its most basic prop- erties, including the way it measures how close a random variable is to being uniformly distributed. Section 28.2 describes relative entropy, or Kullback-Leibler di- vergence, which measures the discrepancy between two probability distributions, and from which Shannon entropy can be constructed. Section 28.2.1 describes some statistical aspects of relative entropy, especially its relationship to expected log-likelihood and to Fisher information. Section 28.3 introduces the idea of the mutual information shared by two random variables, and shows how to use it as a measure of serial dependence, like a nonlinear version of autocovariance (Section 28.3.1). Information theory studies stochastic processes as sources of information, or as models of communication channels. It appeared in essentially its modern form with Shannon (1948), and rapidly proved to be an extremely useful mathe- matical tool, not only for the study of “communication and control in the animal and the machine” (Wiener, 1961), but more technically as a vital part of prob- ability theory, with deep connections to statistical inference (Kullback, 1968), to ergodic theory, and to large deviations theory. In an introduction that’s so limited it’s almost a crime, we will do little more than build enough theory to see how it can fit in with the theory of inference, and then get what we need to progress to large deviations. If you want to learn more (and you should!), the deservedly-standard modern textbook is Cover and Thomas (1991), and a good treatment, at something more like our level of mathematical rigor, is Gray 189 CHAPTER 28. ENTROPY AND DIVERGENCE 190 (1990). 1 28.1 Shannon Entropy The most basic concept of information theory is that of the entropy of a random variable, or its distribution, often called Shannon entropy to distinguish it from the many other sorts. This is a measure of the uncertainty or variability asso- ciated with the random variable. Let’s start with the discrete case, where the variable takes on only a finite or countable number of values, and everything is easier. Definition 356 (Shannon Entropy (Discrete Case)) The Shannon entropy , or just entropy , of a discrete random variable X is H [ X ] ≡ - x P ( X = x ) log P ( X = x ) =- E [log P ( X )] (28.1) when the sum exists. Entropy has units of bits when the logarithm has base 2, and nats when it has base e . The joint entropy of two random variables, H [ X, Y ] , is the entropy of their joint distribution. The conditional entropy of X given Y , H [ X | Y ] is H [ X | Y ] ≡ y P ( Y = y ) x P ( X = x | Y = y ) log P ( X = x | Y = y )(28.2) =- E [log P ( X | Y )] (28.3) = H [ X, Y ]- H [ Y ] (28.4) Here are some important properties of the Shannon entropy, presented with- out proofs (which are not hard)....
View Full Document
This note was uploaded on 12/20/2011 for the course STAT 36-754 taught by Professor Schalizi during the Spring '06 term at University of Michigan.
- Spring '06