{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

Chapter 14_English

# Chapter 14_English - Chapter 14 Entropy Te r S ache...

This preview shows pages 1–13. Sign up to view the full content.

1 Chapter 14 Entropy Teacher: Sin-Horng Chen Office: Engineering Bld. #4, Room 805 Tel: ext. 31822 Email: [email protected]

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
2 14-1 Introduction The probability of an event A , denoted as ( ) P A , can be interpreted as an uncertainty measure of the occurrence of the event. If ( ) 1, we are definitely sure that will occur If ( ) 0, we are definitely sure that will not occur no unceretainty at all P A A P A A = = On the other hand, for ( ) 0.5 P A = , the probabilities that A will occur and that A will not occur are equal. We therefore can say that the uncertainty of the event A is maximum.
3 Entropy is defined as an uncertainty measure of a partition U of the probability space of an experiment S . The probability space S is the set of all elementary events (or outcomes). For a partition { } 1 , , N U A A = , 1 2 N A A A S = and i j A A φ = . Entropy of U is defined as 1 1 ( ) log log N N H U p p p p = - - - , where ( ) i i p P A = . Information = uncertainty 1 It is a viewpoint that the occurrence of an uncertain event brings more information while a certain event (either occur or not occur) carries no information.

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
4
5 Ex. 4-1: Experiment of tossing fair-die: (a) Let { , } U even odd = . Obviously, { } { } 0.5 P even P odd = = . Hence 1 1 1 1 ( ) log log log2 2 2 2 2 H U = - - = (b) Let V be the set of elementary events { } i V f = . Obviously, 1 { } 6 i P f = . Then, ( ) log6 H V = . The difference of their entropy is log3. It is the uncertainty about V assuming that U is known. This means that if we know that the outcome is even (or odd), then the uncertainty of outcome reduces to log3. It is known as conditional entropy .

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
6 Ex. 14-2: For a coin experiment, { } P h p = . Then ( ) log (1 )log(1 ) ( ) H V p p p p h p = - - - - Fig.14-2 displays ( ) h p verse p . The maximum occurs at p =0.5. It is 0 at p= 0 and p =1.
7

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
8 14-2 Basic Concepts Let { } { } 1 , , N i U A A A = = be a partition of S , where , 1, , , i A i N = are events. (1) If there are only two events, we call it a binary case. Usually, U is denoted as { } , U A A = , were A is called the complement of A . (2) If the events of U are all elementary events { } i ξ , we denote it by V and call it element partition. (3) A refinement of U is a partition B of S , such that for j B B 2200 , j B is a subset of some event i A . It is denoted by B U P . In other words, some events i A are divided into sub-events contained in B .
9 (4) The product of two partitions U and B is a partition which contains all intersection i j A B of their elements, and is denoted by U B . U B is the largest common refinement of U and B . Properties V U P is true for any U . U B B U = If 1 2 3 U U U P , then 1 3 U U P . If B U P , then U B B = .

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
10
11

This preview has intentionally blurred sections. Sign up to view the full version.

View Full Document
12 Entropy Definition: 1 1 1 ( ) ( log log ) ( ) N N N i i H U p p p p p ϕ = = - + + = where ( ) i i p P A = and ( ) log p p p ϕ = - . h ( ) 0 p ϕ for 0 1 p , ( ) 0 H U ( ) 0 H U = if there exists a probability 1 i p = and 0 for all .
This is the end of the preview. Sign up to access the rest of the document.

{[ snackBarMessage ]}

### Page1 / 56

Chapter 14_English - Chapter 14 Entropy Te r S ache...

This preview shows document pages 1 - 13. Sign up to view the full document.

View Full Document
Ask a homework question - tutors are online