Lecture5

# Lecture5 - Colorado State University Ft Collins ECE 516...

This preview shows pages 1–4. Sign up to view the full content.

1 Colorado State University, Ft. Collins Fall 2008 ECE 516: Information Theory Lecture 5 September 11, 2008 Recap: Theorem: () q p D || is convex in the pair ( ) q p , . That is, if ( ) 1 1 , q p and 2 2 , q p are two pairs of PMFs, then ( ) ( ) ( )( ) 2 2 1 1 2 1 2 1 || 1 || 1 || 1 q p D q p D q q p p D λ + + + Note this is an extended definition of convexity. Corollary: q p D || is convex in p for any fixed q . Theorem: (concavity of entropy) ( ) p H is a concave function of p . Theorem: 0 | ; Z Y X I w.e., iff X and Y are conditionally independent given Z . Corollary: ( ) Z Y X H Z X H , | | (further conditioning decreases entropy) Theorem: (chain rule of information) ( ) ( ) ( ) L L L + + + = = = 2 1 3 1 2 1 1 1 1 1 , | ; | ; ; , , | ; ; , , X X Y X I X Y X I Y X I X X Y X I Y X X I n i i i n 2.8 Data Processing Inequality No clever manipulation (or processing) of data can improve the mutual information. X Y any processing Z Z X I Y X I ; ; Y says more about X than Z says about X .

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2 Definition: The random variables X , Y , Z form a Markov chain (in this order), denoted as Z Y X if following equivalent conditions are satisfied by their joint and conditional PMFs 1. () ( ) y z p x y z p | , | = 2. ( ) ( ) ( ) y z p x y p x p z y x p | | , , = X Y Z ( ) ( ) ( ) ( )() ( ) x p x y p y z p x p x y p y x z p y x p y x z p z y x p | | | , | , , | , , = = = (the last equality is due to the Markov property) 3. ( ) ( ) y z p y x p y z x p | | | , = X and Z are conditionally independent given Y . ( ) ( ) ( ) ( ) ()() y x p y z p y x p y x z p y p y x p y x z p y p z y x p y z x p | | | , | , , | , , | , = = = = Note if Z Y X , then X Y Z , easy to see from 3 (or 2).
3 Theorem: (data processing inequality) If Z Y X , then () Z X I Y X I ; ; Proof: By chain rule, we can expand mutual information in two different ways.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 9

Lecture5 - Colorado State University Ft Collins ECE 516...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online