info_theory_course_reader_stanford

info_theory_course_reader_stanford - Tsachy Weissman...

Info iconThis preview shows pages 1–6. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: Tsachy Weissman Information Theory – EE376A Course reader Winter 2010 Springer Preface These notes form an outline of the core of the material I plan to cover in the course. Most of the theorems, lemmas, and auxiliary results are stated without their proofs. In the lectures, I will follow these notes, filling in proofs, details, emphasis, and intuition. Time permitting we will cover some additional topics, such as channels with feedback and the colored Gaussian channel, which are not included here. There will be some handouts on these topics if and when relevant. I wish us lots of fun during this quarter, as we explore the fundamentals of this exciting field. Stanford January 2010 Tsachy Weissman Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 What is Information Theory ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2-A Binary Source and Channel . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2-B Lossless Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2-C Lossy Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.2-D AWGN Channel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Expectations from the Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Information Measures and some of their Properties . . . . . . . . 9 2.1 Axiomatic Derivation of an Uncertainty Measure . . . . . . . . . . . . 9 2.2 Additional Information Measures and some Properties . . . . . . . 11 3 Fixed-Length Lossless Source Coding . . . . . . . . . . . . . . . . . . . . . . 13 3.1 Fixed-Length “Near Lossless” Coding . . . . . . . . . . . . . . . . . . . . . . 13 3.2 The Asymptotic Equipartition Property (AEP) . . . . . . . . . . . . . 14 3.3 A Direct and a Converse Theorem for Block Coding . . . . . . . . . 15 4 Variable Length Lossless Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4.1 Dyadic Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.2 General Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.3 Huffman Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4 Other UD Code Constructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.5 Kraft-McMillan Inequality and a Converse Result . . . . . . . . . . . 25 5 Channel Coding and Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 5.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5.1-A Coding with a Cost Constraint . . . . . . . . . . . . . . . . . . . . . . 29 5.2 Memoryless Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 5.3 Main Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315....
View Full Document

This note was uploaded on 10/27/2010 for the course ECE 221 taught by Professor Sd during the Spring '10 term at Huston-Tillotson.

Page1 / 84

info_theory_course_reader_stanford - Tsachy Weissman...

This preview shows document pages 1 - 6. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online