Lecture5 - Colorado State University, Ft. Collins ECE 516:...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 5 September 11, 2008 Recap: Theorem: () q p D || is convex in the pair ( ) q p , . That is, if ( ) 1 1 , q p and 2 2 , q p are two pairs of PMFs, then ( ) ( ) ( )( ) 2 2 1 1 2 1 2 1 || 1 || 1 || 1 q p D q p D q q p p D λ + + + Note this is an extended definition of convexity. Corollary: q p D || is convex in p for any fixed q . Theorem: (concavity of entropy) ( ) p H is a concave function of p . Theorem: 0 | ; Z Y X I w.e., iff X and Y are conditionally independent given Z . Corollary: ( ) Z Y X H Z X H , | | (further conditioning decreases entropy) Theorem: (chain rule of information) ( ) ( ) ( ) L L L + + + = = = 2 1 3 1 2 1 1 1 1 1 , | ; | ; ; , , | ; ; , , X X Y X I X Y X I Y X I X X Y X I Y X X I n i i i n 2.8 Data Processing Inequality No clever manipulation (or processing) of data can improve the mutual information. X Y any processing Z Z X I Y X I ; ; Y says more about X than Z says about X .
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 Definition: The random variables X , Y , Z form a Markov chain (in this order), denoted as Z Y X if following equivalent conditions are satisfied by their joint and conditional PMFs 1. () ( ) y z p x y z p | , | = 2. ( ) ( ) ( ) y z p x y p x p z y x p | | , , = X Y Z ( ) ( ) ( ) ( )() ( ) x p x y p y z p x p x y p y x z p y x p y x z p z y x p | | | , | , , | , , = = = (the last equality is due to the Markov property) 3. ( ) ( ) y z p y x p y z x p | | | , = X and Z are conditionally independent given Y . ( ) ( ) ( ) ( ) ()() y x p y z p y x p y x z p y p y x p y x z p y p z y x p y z x p | | | , | , , | , , | , = = = = Note if Z Y X , then X Y Z , easy to see from 3 (or 2).
Background image of page 2
3 Theorem: (data processing inequality) If Z Y X , then () Z X I Y X I ; ; Proof: By chain rule, we can expand mutual information in two different ways.
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

Page1 / 9

Lecture5 - Colorado State University, Ft. Collins ECE 516:...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online