This preview shows pages 1–10. Sign up to view the full content.
This preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full DocumentThis preview has intentionally blurred sections. Sign up to view the full version.
View Full Document
Unformatted text preview: 1 Chapter 14 Entropy Teacher: SinHorng Chen Office: Engineering Bld. #4, Room 805 Tel: ext. 31822 Email: schen@mail.nctu.edu.tw 2 141 Introduction The probability of an event A , denoted as ( ) P A , can be interpreted as an uncertainty measure of the occurrence of the event. If ( ) 1, we are definitely sure that will occur If ( ) 0, we are definitely sure that will not occur no unceretainty at all P A A P A A = = On the other hand, for ( ) 0.5 P A = , the probabilities that A will occur and that A will not occur are equal. We therefore can say that the uncertainty of the event A is maximum. 3 Entropy is defined as an uncertainty measure of a partition U of the probability space of an experiment S . The probability space S is the set of all elementary events (or outcomes). For a partition { } 1 , , N U A A = , 1 2 N A A A S = and i j A A = . Entropy of U is defined as 1 1 ( ) log log N N H U p p p p =  , where ( ) i i p P A = . Information = uncertainty1 It is a viewpoint that the occurrence of an uncertain event brings more information while a certain event (either occur or not occur) carries no information. 4 5 Ex. 41: Experiment of tossing fairdie: (a) Let { , } U even odd = . Obviously, { } { } 0.5 P even P odd = = . Hence 1 1 1 1 ( ) log log log2 2 2 2 2 H U =   = (b) Let V be the set of elementary events { } i V f = . Obviously, 1 { } 6 i P f = . Then, ( ) log6 H V = . The difference of their entropy is log3. It is the uncertainty about V assuming that U is known. This means that if we know that the outcome is even (or odd), then the uncertainty of outcome reduces to log3. It is known as conditional entropy . 6 Ex. 142: For a coin experiment, { } P h p = . Then ( ) log (1 )log(1 ) ( ) H V p p p p h p =     Fig.142 displays ( ) h p verse p . The maximum occurs at p =0.5. It is 0 at p= 0 and p =1. 7 8 142 Basic Concepts Let { } { } 1 , , N i U A A A = = be a partition of S , where , 1, , , i A i N = are events. (1) If there are only two events, we call it a binary case. Usually, U is denoted as { } , U A A = , were A is called the complement of A . (2) If the events of U are all elementary events { } i , we denote it by V and call it element partition. (3) A refinement of U is a partition B of S , such that for j B B 2200 , j B is a subset of some event i A . It is denoted by B U P . In other words, some events i A are divided into subevents contained in B . 9 (4) The product of two partitions U and B is a partition which contains all intersection i j AB of their elements, and is denoted by U B . U B is the largest common refinement of U and B ....
View
Full
Document
This note was uploaded on 07/21/2009 for the course CM EM5102 taught by Professor Sinhorngchen during the Fall '08 term at National Chiao Tung University.
 Fall '08
 SinHorngChen

Click to edit the document details