Lecture2 - Colorado State University Ft Collins ECE 516...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 2 August 28, 2008 Recap No class on Sep. 4 and Oct. 16. 0 Non-technical Remarks Key objective: learning about how to formulate a research problem. Follow Shannon’s soul, but not necessarily his method. 0.2 Some properties Highly mathematical. Theoretical understandings may seem to be useless. Theoretical and Engineering researches. 2 Information Measures: Entropy, Relative Entropy, and Mutual Information (Chapter 2) 2.1 Entropy Define information of event x as () x p x f 1 log = . Definition: The entropy of a discrete r.v. X is ( ) () () () = = X X x x x p x p x p x p X H 1 log log Lemma: ( ) 0 X H Lemma: ( ) X aH X H a b b log =
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 Example: = = = = = 8 / 1 3 8 / 1 2 4 / 1 1 2 / 1 0 p p p p X , = = = = = 4 / 1 3 4 / 1 2 4 / 1 1 4 / 1 0 p p p p Y () 75 . 1 8 log 8 1 8 log 8 1 4 log 4 1 2 log 2 1 = + + + = X H bits X is less random than Y . How do I represent X and Y with bits most efficiently? (Compression – Chp. 5) 0 0 = X 00 0 = Y 10 1 = X 01 1 = Y 110 2 = X 10 2 = Y 111 3 = X 11 3 = Y [] bits 75 . 1 8 1 3 8 1 3 4 1 2 2 1 1 = × + × + × + × = X L E ( ) [ ] bits 2 1 2 = × = Y L E Less entropy less randomness, less self-information more compression, less average number of bits to present
Background image of page 2
3 2.2 Joint Entropy and Conditional Entropy Definition: The joint entropy of a pair of r.v. ( ) Y X , with joint PMF () y x p , is ( ) = = = ∑∑ ∈∈ Y X p E y x p y x p y x p y x p Y X H xy , 1 log , 1 log , , log , , XY This definition is nothing new, just view X , Y as a single vector ransom variable.
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 03/17/2010 for the course ECE 516 taught by Professor Rocky during the Spring '08 term at Colorado State.

Page1 / 8

Lecture2 - Colorado State University Ft Collins ECE 516...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online