Lecture2 - Colorado State University Ft Collins ECE 516...

Info icon This preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 2 August 28, 2008 Recap No class on Sep. 4 and Oct. 16. 0 Non-technical Remarks Key objective: learning about how to formulate a research problem. Follow Shannon’s soul, but not necessarily his method. 0.2 Some properties Highly mathematical. Theoretical understandings may seem to be useless. Theoretical and Engineering researches. 2 Information Measures: Entropy, Relative Entropy, and Mutual Information (Chapter 2) 2.1 Entropy Define information of event x as ( ) ( ) x p x f 1 log = . Definition: The entropy of a discrete r.v. X is ( ) ( ) ( ) ( ) ( ) = = X X x x x p x p x p x p X H 1 log log Lemma: ( ) 0 X H Lemma: ( ) ( ) X aH X H a b b log =
Image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon