L14-informationtheory-2009

L14-informationtheory-2009 - Information Theory Trac D Tran...

Info iconThis preview shows pages 1–4. Sign up to view the full content.

View Full Document Right Arrow Icon
1 Information Theory Information Theory Trac D. Tran ECE Department The Johns Hopkins University Baltimore, MD 21218 Outline Outline ± Probability ² Definition ² Properties ² Examples ² Random variable ± Information theory ² Self-information ² Entropy ² Entropy and probability estimation ² Examples Dr. Claude Elwood Shannon
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
2 Deterministic versus Random Deterministic versus Random ± Deterministic ² Signals whose values can be specified explicitly ² Example: a sinusoid ± Random ² Digital signals in practice can be treated as a collection of random variables or a random process ² The symbols which occur randomly carry information ± Probability theory ² The study of random outcomes/events ² Use mathematics to capture behavior of random outcomes and events Probability Probability ± Events and outcomes ² Let X be an event with N possible mutually exclusive outcomes ² Example ² A coin toss is an event with 2 outcomes: Head ( H ) or Tail ( T ) ² A dice toss is an event with 6 outcomes: {1,2,3,4,5,6} ± Probability ² The likelihood of observing a particular outcome above ² Standard notation { } N X X X , , , 2 1 K i X [ ] i X X P =
Background image of page 2
3 Important Properties Important Properties ± Probability computation or estimation ± Basic properties ² Every probability measure lies inclusively between 0 and 1 ² Sum of probabilities of all outcomes is unity: ² For N equally likely outcomes ² For two statistically independent event [] outcomes of number total outcomes possible of number i total i i X N N X X P = = = [ ] i X X P i = , 1 0 = = = N i i X X P 1 1 [ ] N X X P X X P X
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Image of page 4
This is the end of the preview. Sign up to access the rest of the document.

This note was uploaded on 02/02/2010 for the course ENGINEERIN 520.101 taught by Professor Tracdtran during the Winter '06 term at Johns Hopkins.

Page1 / 8

L14-informationtheory-2009 - Information Theory Trac D Tran...

This preview shows document pages 1 - 4. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online