1Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 2 August 28, 2008 Recap No class on Sep. 4 and Oct. 16. 0 Non-technical Remarks Key objective: learning about how to formulate a research problem. Follow Shannon’s soul, but not necessarily his method. 0.2Some propertiesHighly mathematical. Theoretical understandings may seem to be useless. Theoretical and Engineering researches. 2 Information Measures: Entropy, Relative Entropy, and Mutual Information (Chapter 2) 2.1 Entropy Define information of event xas ( )( )xpxf1log=. Definition: The entropy of a discrete r.v. Xis ()( )( )( )( )∑∑∈∈=−=XXxxxpxpxpxpXH1loglogLemma:()0≥XHLemma:()()XaHXHabblog=
has intentionally blurred sections.
Sign up to view the full version.