This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: Chapter14 Entropy 141 Introduction Probability of an event A , denoted as ( ) P A , H event A j 3 P uncertainty r measurer If ( ) 1, no unceretainty at all If ( ) 0, P A A P A A = = ( ) 0.5 P A = A b + 3S A P uncertainty r maximum Entropy t 3 experiment S r partition U r uncertainty measure Probability space S P elementary events (outcomes) r Partition { } 1 , , N U A A = L 1 2 N A A A S = L i j A A φ = Entropy of U is defined as 1 1 ( ) log log N N H U p p p p =  L , where ( ) i i p P A = . Information = uncertainty S 3* F F r information b F 3 information Ex. 41 r fairdie r experiment r (a) r { , } U even odd = { } { } 0.5 P even P odd = = Hence 1 1 1 1 ( ) log log log2 2 2 2 2 H U =  = (b) r V P elementary events { } i V f = ( 3 1 { } 6 i P f = ( ) log6 H V = Entropy r log3 r uncertainty about V assuming U (conditional entropy) b F 3 even r oddH uncertainty r log3 Ex. 142 For coin experimentr { } P h p = ( ) log (1 )log(1 ) ( ) H V p p p p h p =  g Fig.142 r ( ) h p verse pP p =0.5 r maximumr p= 0 r 1 r 0r 142 Basic Concepts Let { } { } 1 , , N i U A A A = = L r partition of SP i A events (1) 14 eventsr binary, In this case, we usually denote U by { } , U A A = A is called the complement of A (2) r U P events r elementary events { } i ξ denote it by V and call it element partition (3) A refinement of U is a partition B of S , such that for j B B 2200 j B P i A P subsetr denoted by B U p , r B P U P events 14 events (see Fig.144)r B U p iff j i B A P A common refinement r partitions r refinement (4) The product of U and B is a partitionr elements r i j A B (r )r denoted by U B P U B P P U P B P common refinement Properties V U p for any U U B B U = , ( ) ( ) U B C U B C = If 1 2 3 U U U p p , then 1 3 U U p If B U p , then U B B = Entropy Definition: 1 1 1 ( ) ( log log ) ( ) N N N i i H U p p p p p ϕ = =  + + = L where ( ) i i p P A = ( ) log p p p ϕ =  Q ( ) p ϕ for 1 p P , ∴ ( ) H U P ( ) H U = if r 1 i p = i p = For binary partitionr ( ) p P A = ( ) 1 P A p =  ∴ ( ) log (1 )log(1 ) ( ) H U p p p p h p =  = If 1 1 2 N p p p p = = = = L then 1 1 1 1 ( ) log log log H U N N N N N =  = L In this case, if 2 m N = , then ( ) H U m = . (Here, the base of log operation is 2) Inequalityr ( ) log p p p ϕ =  convex function 1 2 1 2 1 2 ( ) ( ) ( ) ( ) ( ) p p p p p p ϕ ϕ ϕ ϕ ε ϕ ε + < + < + + where 1 1 2 2 p p p p ε ε < + < . t{ ℵ♠ 3 ( ) p ϕ ℵ (see Fig.146) inequality t 3 (1) If { } 1 2 , , , N U A A A = L and { } 2 , , , , a b N B B B A A = L where a B P b B P 1 A split t{ eventsr (see Fig.147) r ( ) ( ) H U H B P , 1 1 2 1 1 2 ( ) ( ) ( ), ( ( )) ( ( )) ( ( )) P A P B P B P A P B P B ϕ ϕ ϕ = + + Q (convex function r ) (2) If B U p , then ( ) ( ) H B H U P . (r generalization of (1)) (3) For U 2200 , ( ) ( ) H U H V P . ( Q V U p ) (4) For U 2200 and B P ( )...
View
Full Document
 Fall '08
 SinHorngChen
 Information Theory

Click to edit the document details