Review Probability CS 221 Section 3
Olga Russakovsky October 12, 2009
1
Random variables
Consider running a probabilistic experiment, e.g., tossing a coin. Let be the set of all possible outcomes of this experiment, called the sample space. In this case, = {Heads, Tails} If we were to toss a coin 3 times, then = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT} A random variable is a function that map outcomes of a probabilistic experiment to a real number. More formally, a random variable X is a function X : R. For example, we can dene a random variable X = the number of heads in two tosses of a coin (1)
If a random variable X takes on only a nite number of values, i.e., X {0, 1, . . . n}, then it is called a discrete random variable. Otherwise, it is called a continuous random variable. The example above in (1) is a discrete random variable. The simplest example of a continuos random variable is a random number generator that can return any real number between 0 and 1. In CS221 we will be mostly dealing just with discrete variables.
0 Many thanks to Quoc Le, Arian Maleki and Tom Do. For more details, please refer to the very thorough probability review handout found at http://cs229.stanford.edu/section/cs229-prob.pdf
1
2
2.1
Probability Distributions
Discrete random variables
For a discrete random variable, a probability distribution is a list of probabilities associated with each of its possible values. For example, in the example above, if we have a fair coin, then P (X = 0) = P (TT outcome) = 1 4 1 2 1 4
P (X = 1) = P (HT or TH outcome) = P (X = 2) = P (HH outcome) =
For discrete variables, this is called the probability mass function, or PMF. We denote P (X = x) = PX (x) = p(x) The requirements imposed on the probability mass function are: p(x) 0 for all x
x
(2)
p(x) = 1
Additionally, dene an event A to be a subset of the sample space, i.e., A . For example, A might be the event that at least one toss turns up heads, or X 1. Then pX (A) =
xA
pX (x)
(3)
So in the case where A is the event of at least one heads, pX (A) = P (X = 1) + P (X = 2) = Two common examples of discrete random variables are X Bernoulli(p) with 0 p 1 corresponds to taking a coin which has probability p of coming up heads, and ipping it once. X is 1 if heads comes up, 0 otherwise. p(x) = p 1p if x = 1 if x = 0 (4) 11 3 += 24 4
X Binomial(n, p) with 0 p 1 corresponds to taking a coin which has probability p of coming up Heads, and ipping it n times. X is the number of heads. p(x) = n k px (1 p)(nx) (5)
1 The example discussed above is Binomial(2, 2 ).
2.2
Continous random variables
In case of a continous random variable, the probability distribution is called the probability density function, or PDF, and is often denoted f (x). Here for example, can be the set of all real numbers between 0 and 1, and an event A can correspond to the set of all real numbers 1 between 5 and 3 . As before, the properties are 5 f (x) 0 for all x
x
f (x) = 1, so the total probability of all outcomes is 1
P (A) = P (x A) = xA f (x), so the probability of an event A is just computed as a part of the area under the curve f (x) We wont be using continuous random variables much in CS221, but two common examples are: X U nif orm(a, b) with a < b corresponds a random number generator where any real number between a and b is equally likely f (x) =
1 b a
0
if a x b otherwise
(6)
X N ormal(, 2 ) is also known as a Gaussian distribution f (x) = 1 2 2 e 22 (x)
1 2
(7)
2.3
Properties of probability distributions
Consider a probability distribution P and two events A and B . Then If A B , then P (A) P (B ) P (A B ) min(P (A), P (B )) Union bound P (A B ) P (A) + P (B ) P ( A) = 1 P (A) If we impose additional restrictions on the events, we can obtain even more properties: If A and B are independent, then P (A B ) = P (A)P (B ) If A and B are disjoint, so A B = , then P (A B ) = P (A) + P (B ) By induction, if A1 , A2 , . . . Ak are disjoint events, so Ai Aj = 0 whenever i = j , then P (i Ai ) = P (Ai )
i
(8)
Law of total probability If A1 , A2 , . . . Ak are disjoint events such that i Ai = , then
i
P (i Ai ) = 1
(9)
3
Joint distributions
Consider the case of two random variables X and Y . The joint probability distribution denes the probability of two events ocurring together.
3.1
Discrete random variables
If X and Y are discrete, the joint probability mass function is denoted pX,Y (x, y ) = P (X = x, Y = y ) As a specic example, let X = outcome of a fair coin toss, 1 if heads, 0 if tails Y = outcome of the roll of a fair 6-sided die Then pX,Y (tails, roll a 3) = P (X = tails and Y = roll a 3) = We can represent this joint distribution as a 2 6 table: X=0 X=1 Y=1 1/12 1/12 Y=2 1/12 1/12 Y=3 1/12 1/12 Y=4 1/12 1/12 Y=5 1/12 1/12 Y=6 1/12 1/12 1 12 (11) (10)
The marginal probability is the overall probability of the event X = x, regardless of what happens with Y , and is obtained by p(x) =
y Domain(Y )
p(x, y )
(12)
where Domain(Y ) is just the set of all possible values of Y . This process is called marginalization. In the table representation, it corresponds to summing out one of the dimensions (e.g., in this case, summing over all the rows or all the columns in the table). Specically, in example (11),
6 6
P (X = tails) =
i=1
P (X = tails, Y = roll i) =
i=1
1 1 = 12 2
The conditional probability mass function denes the probability of an event x ocurring given that weve already observed Y = y as p(X = x|Y = y ) = p(x|y ) = p(x, y ) p(y ) (13)
assuming p(y ) = 0. In the 2-dimensional table representation, this means we focus our attention on just a single row or column in the table and ignore the rest. In this example, we might consider P (X = heads|Y = roll a 4) = P (X = heads and Y = roll a 4) = P (Y = roll a 4)
1 12 1 6
=
1 2
3.2
Bayes rule
In order to derive Bayes rule, recall the denition of conditional distribution in (13) and consider two statements that follow from it: p(x, y ) = p(x|y )p(y ) Applying (13) rst followed by (15), derive: p(x|y ) = p(x, y ) p(y |x)p(x) = p(y ) p(y ) (16) (14) (15)
p(x, y ) = p(y |x)p(x)
Another (equivalent) formulation of Bayes rule, derived by applying (12) followed by (15) is p(x|y ) = = p(y |x)p(x) xDomain(X ) p(x, y ) p(y |x)p(x) xDomain(X ) p(y |x)p(x) (17)
3.2.1
Example
A typical example of an application of Bayes rule is where X is a binary variable corresponding to the presence or absence of a certain disease, and Y is a random variable corresponding to some trait of a patient, such as blood pressure. Suppose we have a patient with trait y , and want to compute the probability that this patient has the disease, i.e., P (X = 1|Y = y ). One way to do it is to collect a group of people all of which share the trait Y = y , and estimate P (X = 1|Y = y ) directly by counting the number of people in this group that have the disease. However, as you might imagine, this would be very dicult in practice, especially if Y is multi-dimensional, and, if Y is continuous (e.g, blood pressure), then even impossible. On the other hand, using Bayes rule, all we need to know is the proportion of people that have the disease, or P (X = 1), which also tells us P (X = 0), and the conditional distributions P (Y |X = 1) and P (Y |X = 0) corresponding to the proles of sick and healthy people respectively relative to this trait The proportion of people that have the same trait as the patient, or P (Y = y ) can then be computed using (13, 12) P (Y = y ) = P (Y = y |X = 0)P (X = 0) + P (Y = y |X = 1)P (X = 1) Note that in practice nding a group of sick patients and a group of healthy patients to estimate P (Y = |X 1) and P (Y |X = 0), along with estimating P (X = 1) for the general population, is easier than attempting to compute P (X = 1|Y = y ) directly.
3.3
Independence
Two variables are considered independent if knowing Y doesnt change your belief about X . More formally, X and Y are independent if for all values x Domain(X ) and y Domain(Y ): p(X = x|Y = y ) = p(X = x) p(X = x, Y = y ) = p(X = x)p(Y = y ) (18)
or, equivalently , using the denition of conditional probability, (19)
1
The coin toss and the die roll random variables from the previous example are independent random variables. However, if we consider two tosses of the same fair coin and dene the random variables as X = the number of heads in the two tosses of this coin Y= 1 if any of the two tosses comes up heads 0 otherwise
then the two variables are not independent, since P (X = 2|Y = 0) = P (HH outcome|neither toss comes up heads) = 0 1 P (X = 2) = P (HH outcome) = 4 Going back to the table representation, if X and Y are independent then we can represent their joint probability distribution by two 1-dimensional tables (one for (P (X ) and one for P (Y )) instead of a full 2-dimensional table: Y=1 1/6 Y=2 1/6 Y=3 1/6 Y=4 1/6 Y=5 1/6 Y=6 1/6 X=0 1/2 X=1 1/2
The advantage is that number of entries in this case would be size(textDomain(X )) + size(Domain(Y )) = 8 instead of size(Domain(X )) size(Domain(Y )) = 12
3.4
Conditional independence
Two variables X and Y are conditionally independent given a third variable Z if p(X = x, Y = y |Z = z ) = p(X = x|Z = z )p(Y = y |Z = z ) for all x Domain(X ), y Domain(Y ), z Domain(Z ), such that P (Z = z ) = 0.
1 Formulation (19) implicitly deals with the case where P (Y = y ) = 0. In formulation (18) to be precise we actually have to consider this as a special case, and change the denition from for all y Domain(Y ) to for all y Domain(Y ) such that P (Y = y ) = 0.
(20)
3.4.1
Conditional independence does not imply independence
Note that it might be the case that X and Y are not independent, but they are conditionally independent given Z . For example, suppose X and Y are as above, so X = the number of heads in the two tosses of this coin Y= 1 if any of the two tosses comes up heads 0 otherwise
Recall that we have already argued that these are not independent. Now we dene Z to be the same as Y , so we obtain p(X = x, Y = y |Z = z ) = 0 if y = z P (X = x|Z = z ) otherwise
This is because the variables Y and Z are identical, so it cant be the case that y = z . If y = z , since P (Y = y |Z = z ) = 0 we have p(X = x, Y = y |Z = z ) = 0 = P (X = x|Z = z )P (Y = y |Z = z ) and conditional independence holds so far. When y = z , the fact that Y = y gives us no more information about X than if we just knew that Z = z . If y = z , then since P (Y = y |Z = z ) = 1, P (X = x, Y = y |Z = z ) = P (X = x|Z = z ) = P (X = x|Z = z )P (Y = y |Z = z ) and we have satised the denition of conditional independence. This example is somewhat degenerate, but there are plenty of more complicated examples where conditional independence does not imply independence. This is an important result to keep in mind. 3.4.2 Independence does not imply conditional independence
Alternatively, note that we can also have two independent variables that become dependent when conditioned on a third one. As a simple example, consider ipping two fair coins again and let X= Y= Z= 1 if coin1 comes up heads 0 otherwise 1 if coin2 comes up heads 0 otherwise 1 if both coins come up heads 0 otherwise (21)
Clearly X and Y are independent since they are dealing with dierent coins. Now consider what happens when we observe that Z = 0 and condition on that. We know that only one of three possible outcomes have occurred: HT, TH or TT. P (X = 0, Y = 0|Z = 0) = P (coin1 = T and coin2 = T| HT, TH or TT) = P (X = 0|Z = 0) = P (coin1 = T| HT, TH or TT) = 2 3 2 P (Y = 0|Z = 0) = P (coin2 = T| HT, TH or TT) = 3 1 3
So P (X = 0, Y = 0|Z = 0) = P (X = 0|Z = 0)P (Y = 0|Z = 0) So two variables can become correlated when conditioned on a third.
3.5
Chain rule
Using the denition of conditional distribution from (13), we can derive the chain rule P (X = x, Y = y, Z = z ) = P (X = x|Y = y, Z = z )P (Y = y |Z = z )P (Z = z ) or, more generally, P (X1 , X2 , . . . Xn ) = P (Xn |X1 . . . Xn1 )P (Xn1 |X1 . . . Xn2 ) . . . P (X2 |X1 )P (X1 ) This holds true for all variables X1 . . . Xn . (23) (22)
4
4.1
Statistics of random variables
Expectation
The expectation, or mean of a discrete random variable is dened as E[X ] =
xDomain(X )
x P (x)
(24)
Given an arbitrary function g : R R, we have E[g (X )] =
xDomain(X )
g (x) P (x)
(25)
4.1.1
Expectations of standard distributions
For the Bernoulli variable dened in (4), we have E[X ] = 1 p + 0 (1 p) = p For the Binomial dened in (5), the derivation is trickier, but E[X ] = np. For the Gaussian N (, 2 ) of (7), its E[X ] = . 4.1.2 Properties (26)
Some properties of expectation are E[c] = c for any constant c E[cg (X )] = cE[g (X )] Linearity of expectation E[f (X ) + g (X )] = E[f (X )] + E[g (X )]. This holds whether or not the variables are independent!
For discrete random variable X , E[1{X = k }] = P (X = k ) The linearity of expectation property is very important in probability theory, and will come up a lot in CS221, so lets emphasize it again: For all random variables X and Y , whether they are independent or not, E[X + Y ] = E[X ] + E[Y ] 4.1.3 Linearity of expectation examples
Since this is such an important concept, lets consider three linearity of expectation examples. Example 1 First, consider two independent variables X and Y , dened as in (21), so corresponding to two fair coins. We compute 11 + =1 22 E(X + Y ) = 2 P (HH) + 1 P (HT orTH) + 0 P (TT) 1 1 =2 +1 =1 4 2 E(X ) + E(Y ) = Example 2 Now verify that this works with dependent variables. Consider tossing just one fair coin, and dening 1 if coin comes up heads X=Y = 0 otherwise This time X + Y can take on only two values: its 2 if the coin comes up heads (since both X and Y will have values of 1), and 0 otherwise. Again we have E(X ) + E(Y ) = 11 + =1 22 1 =1 2
E(X + Y ) = 2 P (H) + 0 P (T) = 2
Example 3 Finally, lets consider an even more interesting example, where we roll a single fair six-sided die and let X = the roll of the die Y= 1 if die roll 3 0 otherwise
These variables consider the same die and are clearly not independent. However, we compute E(X ) = 1 P (roll 1) + 2 P (roll 2) + 3 P (roll 3) + 4 P (roll 4)+ + 5 P (roll 5) + 6 P (roll 6) 1 1 = (1 + 2 + 3 + 4 + 5 + 6) = 3 6 2 1 E(Y ) = 0 P (roll 1, 2, 3) + 1 P (roll 4, 5, 6) = 2 E(X ) + E(Y ) = 4
To compute E(X + Y ), we consider what happens on each roll of the die: Roll Value of X + Y So E(X + Y ) = 7 P (roll 6) + 6 P (roll 5) + 5 P (roll 4)+ + 3 P (roll 3) + 2 P (roll 2) + 1 P (roll 1) 1 = (7 + 6 + 5 + 3 + 2 + 1) = 4 6 Thus in this case still E[X + Y ] = E[X ] + E[Y ] 1 0+1=1 2 0+2=2 3 0+3=3 4 1+4=5 5 1+5 = 6 6 1+ 6 + 7
4.2
Variance
V ar[V ] = E[(X E[X ])2 ]
The variance of a random variable is the measure of deviation from the mean, dened as (27)
Alternatively, V ar[V ] = E[(X E[X ])2 ]
= E[X 2 ] E[X ]2 4.2.1 Variances of standard distributions
= E[X 2 ] 2E[X ]2 + E[X ]2
= E[X 2 ] E[2X E[X ]] + E[E[X ]2 ] (28)
= E[X 2 2X E[X ] + E[X ]2 ]
Again, for the Bernoulli variable dened in (4), we have E[X 2 ] = 12 p + 02 (1 p) = p E[X ]2 = p2 V ar[X ] = p p2 = p(1 p) For the Binomial dened in (5), V ar[X ] = np(1 p). For the Gaussian N (, 2 ) in (7), V ar[X ] = 2 . 4.2.2 Properties (29)
Some properties of variance are V ar[c] = 0 for any constant c V ar[cf (X )] = c2 V ar[f (X )]