2700 ENGRD Basic Engineering Probability and Statistics Lecture 6: Independence; Random Variables and their Distributions
David S. Matteson
School of Operations Research and Information Engineering Rhodes Hall, Cornell University Ithaca NY 14853 USA
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 1 of 37
[e-mail] February 4, 2009
Go Back Full Screen Close Quit
1.
Independence.
Independence. Random Variables. Discrete RV's
What does it mean for 2 events to be independent? Intuition: Knowledge about likelihood of one event occurring, does not affect estimate of likelihood that the other occurs.
BUT:
Intuition can be wrong, immature or totally out to lunch. Some examples would fool anyone's intuition. Need to check the formal definition. Formal Definition. Events A and B are independent in the probability model (S, A, P) if P(AB) = P(A)P(B). If P(A) > 0 and P(B) > 0 then P(A|B) = P (A) and P(B|A) = P(B).
pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 2 of 37 Go Back Full Screen Close Quit
Note 1. This is a technical definition which requires a technical verification. 2. The definition is relative to a given model and dependent on a given P. If you change the P, you may change whether 2 events are independent. It could be that A, B are independent in (S, A, P1 ) BUT A, B are not independent in (S, A, P2 ) 3. Independence is a function of the probability measure and is sometimes confused for bad reasons with disjointness which is a pure set concept.
Page 3 of 37 Go Back Full Screen Close Quit
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Example: Choose a card at random from a deck of 52. S = {1, . . . , 52}. Define events A = [chosen card is spade] B = [chosen card is ace ] Should these be independent? Note 13 1 = ; 52 4 AB = [ace of spades] so P(A) = P(B) = 1 . 52
Page 4 of 37 Go Back Full Screen Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution
4 1 = . 52 13
Expectation Title Page
P(AB) = Check independence condition
P(AB) = P(A)P(B) ?? Yes (intuitive??) since 1 1 1 = . 52 4 13
Close Quit
Definition for more than 2 sets. Although one rarely has to check this because independence can be built into models by the model builder, the general definition is worth knowing. Given events {A1 , . . . , An }. What does independence mean? Let I {1, . . . , n} be a subset of the index set. We need P
iI
Independence. Random Variables. Discrete RV's pmf Independence
Ai ) =
iI
P Ai )
Bernoulli trials Binomial distribution Expectation Title Page
and this must hold for all I {1, . . . , n}. For instance when n = 3, we need to check all pairs of sets are independent as well as P(A1 A2 A3 ) = P(A1 )P(A2 )P(A3 ).
Page 5 of 37 Go Back Full Screen Close Quit
Final Example Experiment: Pick a point at random from the unit square (eg, facility location, pollution source, location of infective, etc). S = {(x, y) : 0 x 1, 0 y 1}. Define the probability measure P for nice regions R by P(R) = area of R = |R|. Define two functions X, Y on S by X : S [0, 1]; and X(x, y) = x, Y (x, y) = y.
Page 6 of 37 Go Back Full Screen Close Quit
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation
Y : S [0, 1];
Title Page
Pick two subintervals of [0, 1], say [a, b], [c, d] and define A = X [a, b] = {(x, y) S : X(x, y) [a, b]} = {(x, y) S : x [a, b]} B = Y [c, d] = {(x, y) S : Y (x, y) [c, d]} = {(x, y) S : y [a, b]}.
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 7 of 37 Go Back Full Screen Close Quit
Independence? P(A) = (b - a), and therefore P(AB) = area of rectangle = (b - a)(d - c) = P(A)P(B). P(B) = (d - c)
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 8 of 37 Go Back Full Screen Close Quit
2.
Random Variables.
Independence. Random Variables. Discrete RV's
Like X, Y in the previous example, random variables help us focus attention on a particular aspect of an experiment. For example: Suppose we have repeated trials S = (j1 , . . . , jn ) : jl {0, 1}; l = 1, . . . , n . Define
n
pmf Independence Bernoulli trials Binomial distribution Expectation
N (j1 , . . . , jn ) =
l=1
jl = # successes ,
Title Page
T (j1 , . . . , jn ) = min{l : jl = 1} = index of the first trial to give success.
Page 9 of 37 Go Back Full Screen Close Quit
Definition: A random variable X is a function on S (the domain of the function) with range some subset of the real numbers. Schematically: X : S R. So X is merely a rule assigning to each element of S a number. It helps us focus on an aspect of the experiment of interest. Notation: For B R [X B] = {s S : X(s) B}. Logic: If (S, A, P) is a probability model, then P only knows how to assign probabilities to events. The red statement guarantees that [X B] is an event. Thus we can assign a probability to [X B] which we write P [X B]. Abbreviation: rv = random variable.
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 10 of 37 Go Back Full Screen Close Quit
Example: For the counting example where S = {{(j1 , . . . , jn ) : jl = 0 or 1; l = 1, . . . , n}, each element of S represents a possible history of successes (1) or failures (0) in n trials. Recall N : S {0, 1, . . . , n} is defined by
n
Independence. Random Variables. Discrete RV's pmf
N (j1 , . . . , jl ) =
l=1
jl ,
Independence Bernoulli trials Binomial distribution Expectation
and can be thought of as the total number of successes in n trials. [N = k] ={(j1 , . . . , jn ) : N (j1 , . . . , jn ) = k}
n
Title Page
={(j1 , . . . , jn ) S :
l=1
jl = k}.
Page 11 of 37 Go Back Full Screen Close Quit
We will consider how to compute P [N = k], in a bit. k = 0, 1, 2, . . . , n,
Other examples. 1. Indicator functions. Suppose (S, A, P) is a probability model and A is an event. Define the random variable 1A (s) = 1A is called the indicator of A. 2. Gender distribution of 3 child families. Experiment: among all 3 child families, note the genders of the children in order of birth. Then S has 8 elements: S = {f f f, f f m, f mf, mf f, f f m, mf m, f mm, mmm}. Define the random variable N = # of girls.
Page 12 of 37 Go Back Full Screen Close Quit
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
1, 0,
if s A, if s A .
As a function of the sample space, the values of N are S N fff 3 ffm 2 fmf 2 mff 2 ffm 1 mfm 1 fmm 1 mmm 0
Independence. Random Variables.
How do we assign probabilities to events such as [N = 2]? Suppose the probability model is S P fff 1/8 ffm 1/8 fmf 1/8 mff 1/8 ffm 1/8 mfm 1/8 fmm 1/8 mmm 1/8
Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation
This leads to the probabilities of events such as [N = k]: k [N = k] P[N = k] 0 {mmm} 1/8 1 {mmf, mf m, f mm} 3/8 2 {f f m, f mf, mf f } 3/8 3 {f f f } 1/8
Title Page
Page 13 of 37 Go Back Full Screen Close Quit
3. Pick a point at random on the unit square. Recall the model where S = [0, 1]2 = {(x, y) : 0 x 1, 0 y 1}. Events are subsets which have area. P assigns "area" to a region. Define three random variables X(x, y) = x, Y (x, y) = y, R(x, y) = x2 + y 2 .
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution
Note: A sample space S can have several random variables defined on it. Then for 0 t 1 P[X t] =P {(x, y) : 0 x t} = t, P[Y t] =P {(x, y) : 0 y t} = t, P[R t] =P {(x, y) : x2 + y 2 t2 } 1 1 = area of disc radius t = t2 . 4 4 0 t 1, 0 t 1,
Expectation Title Page
Page 14 of 37 Go Back Full Screen Close Quit
3.
Discrete RV's
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials
Definition. A random variable X is discrete if its range can be written as a list; that is, if the range is finite or countable. Examples: 1. S = {mmm, mmf, mf m, f mm, f f m, f mf, mf f, f f f } is discrete and N = # of girls, has range {0, 1, 2, 3} and hence is discrete. (Both S and N are discrete.) 2. S = unit square and X, Y, R defined as before. None of these random variables is discrete and S is not discrete. However define the indicator 1RECT (x, y) = where 1, 0, if (x, y) RECT, otherwise, 3 1 , x 4 4 the random 3 }, 4 variable is.
Binomial distribution Expectation Title Page
1 x 4 and in this case, S is not discrete but RECT = {(x, y) : 3. Let S = [0, ) and define
Page 15 of 37 Go Back Full Screen Close Quit
X : S [0, ) by X(s) = s. X is not discrete.
4.
Probability distribution of a discrete random variable
Independence. Random Variables. Discrete RV's pmf Independence
Suppose X is a discrete random variable on (S, A, P). Call x a possible value of X if {s S : X(s) = x} = . Examples: 1. Gender distribution of 3 children families. The possible values of N are 0, 1, 2, 3. 2. Indicator random variable. The possible values of 1A for an event A are 0, 1. The probability mass function (pmf) of the discrete random variable X is the function p(x) =P[X = x], x is a possible value of X
Bernoulli trials Binomial distribution Expectation Title Page
Page 16 of 37 Go Back Full Screen Close Quit
=P {s S : X(s) = x} .
Synonyms probability density function (pdf = portable document format) commonly used; slighly ambiguous. probability distributionambiguous and easily confused with distribution function. Sometimes if there are several rv's floating around in the discussion, we make it clear which pmf goes with which rv by writing pX (x), x is a possible value. Note x is a dummy variable. Example: Gender distribution among 3 child families. The random variable N =# of girls has possible values 0, 1, 2, 3 and pmf k pN (k) = P[N = k] Properties of the pmf of X: p(x) 0 for all x which are possible values. We have p(x) = P(S) = 1.
x{ possible values }
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
0 1/8
1 3/8
2 3/8
3 1/8
Page 17 of 37 Go Back Full Screen Close Quit
Another example: Number of coin flips necessary to get the first head. Experiment: flip a coin until you get a head and note how many flips were necessary. The probability model: For 0 p 1 and q = 1 - p, suppose p is the success probability or the probability of a head on each flip, whereas the complementary probability q is the probability of a tail. Then we assume the following model:
Table 1: The model.
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
S h th tth ttth . . . P p qp q 2 p q 3 p . . .
Note that
Page 18 of 37
p + qp + q 2 p + q 3 p + = p
i=0
qi = p
1 = 1. 1-p
Go Back Full Screen
So the Table 1 presents a valid model.
Close Quit
Define the random variable T : S {1, 2, . . . } where T = # of flips necessary to get the first head. get We the following table for S, T, pT (s):
Table 2: RV T and its pmf.
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
S P T (s) pT (s) = P [T = s]
h th tth ttth . . . p qp q 2 p q 3 p . . . 1 2 3 4 ... 2 p qp q p q 3 p . . .
Page 19 of 37 Go Back Full Screen Close Quit
Note: For this model, p is a parameter that must be estimated. So we really have a family of models. If you, the super-modeller, believe the coin we are flipping is fair, you might want to (provisionally) hypothesize that p = 1/2. Then the model becomes:
Independence. Random Variables. Discrete RV's pmf
S h th tth ttth . . . P 1/2 1/4 1/8 1/16 . . . T (s) 1 2 3 4 ... pT (s) = P [T = s] 1/2 1/4 1/8 1/16 . . .
Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 20 of 37 Go Back Full Screen Close Quit
5.
Building independence into the model.
Independence. Random Variables. Discrete RV's pmf
There is a way to construct the model so independence of certain events is guaranteed. This will lead naturally to repeated Bernoulli trials. We illustrate this in the discrete case. Given two (generalize to n) probability models: (S (1) , A(1) , P1 ) and (S (2) , A(2) , P2 ) where We imagine (S (1) , A(1) , P1 ) describes the outcomes of a first experiment; (S (2) , A(2) , P2 ) describes the outcomes of the second experiment. We wish to give a model for the compound experiment and our goal is S (i) = {s1 , s2 , . . . , s(i) }, ni
(i) (i)
Independence Bernoulli trials
i = 1, 2.
Binomial distribution Expectation Title Page
Page 21 of 37
Build a model for the compound experiment where we first do experiment 1, then do experiment 2 and the two are done in such a way that they do not influence each other.
Go Back Full Screen Close Quit
The compound model. Here are the components of the compound model: Sample space: S=S
(1)
Independence. Random Variables.
S
(2)
={
(1) (2) si , sj
:
(1) si
S
(1)
(2) , sj
S
(2)
}.
Discrete RV's pmf
Class of events: A = A(1) A(2) = subsets of S. Probability measure: P=P defined in the discrete case by P {(si , sj )} = P1 {(si } P2 {(sj } .
(1) (2) (1) (2) (1)
Independence Bernoulli trials Binomial distribution Expectation Title Page
P
(2)
Page 22 of 37 Go Back Full Screen Close Quit
What this accomplishes. Definition. An event in the compound space A(1) A(1) depends only on the first coordinate if it is of the form A
(1)
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
S
(2)
=
(1) (2) {(si , sj )
:
(1) si
A },
(1)
that is, if to check if it happens, we only need to check the first component. Similarly we can define an event which depends only on the 2nd component. Accomplishment: Any event depending only on the first coordinate, is independent of any event depending only on the 2nd coordinate. Reason: Compute P(A
(1)
S
(2)
S
(1)
A ) = P(A
(2)
(1)
A )=
(1) si A(1) (2) sj A(2)
(2)
P
(1) (2) {(si , sj )}
=
si A(1) sj A(2)
(1) (2)
P1 {si } P2 {sj }
(1)
(2)
Page 23 of 37 Go Back Full Screen Close Quit
=P1 (A(1) )P2 (A(2) ) =P(A(1) S (2) )P(S (1) A(2) ). Hence the probability of the intersection is the product of the probabilities in the compound model.
6.
Repeated Bernoulli trials.
Success or failure Heads or tails 1 or 0 etc
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
A Bernoulli trial is an experiment with two outcomes:
Probability model for single Bernoulli trial: S P 0 q 1 p
How do we model the compound experiment of n independent repeated Bernoulli trials? Sample space: S = {(x1 , . . . , xn ) : xi {0, 1}; i = 1, . . . , n}. Assign probabilities by the product rule:
n
Full Screen Page 24 of 37 Go Back
P (x1 , x2 , . . . , xn ) =
i=1
p q
xi 1-xi
=p
# x's =1 # x's =0
q
.
Close Quit
For example, if n = 4, P (1, 1, 1, 0) =p3 q, P (1, 0, 1, 0) =p2 q 2 , etc
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution
Define N (x1 , . . . , xn ) =
n
Expectation
xl ,
l=1
Title Page
so N = # successes (1) in n repeated trials. What is the probability mass function of N ?
Page 25 of 37 Go Back Full Screen Close Quit
7.
Binomial distribution
Independence.
Consider n repeated Bernoulli trials. We compute pN (k) = P[N = k]. Note the event
n
Random Variables. Discrete RV's pmf
[N = k] ={(x1 , . . . , xn ) S :
l=1
xl = k}
Independence Bernoulli trials Binomial distribution Expectation Title Page
={(x1 , . . . , xn ) S : the list x1 , . . . , xn has exactly k 1's and n - k zeros.} Note, by the product rule, every vector in [N = k] has the same probability, namely pk q n-k . So the probability of the event P[N = k] =
(x1 ,...,xn )[N =k]
Page 26 of 37
P (x1 , . . . , xn ) # elements in [N = k] .
Go Back Full Screen Close Quit
=pk q n-k
The combinatorial problem. How many vectors does the event [N = k] contain? Task: Fill slots of a vector of length n with 1's and 0's in such a way that there are exactly k slots with 1's and n - k slots with 0's. Decompose the Task into two subtasks: 1. Pick k slots and drop 1's into these slots. 2. Pick n - k slots and drop 0's into these slots. The number of ways to do the task is ? = # ways to do subtask 1 # ways to do subtask 2 n = 1. k Conclusion:
Page 27 of 37 Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
P [N = k] = prob of k successes in n repeated trials n k n-k = p q k =b(k; n, p) where p=success probability and n is the number of trials.
Go Back Full Screen Close Quit
Special case n = 3: N b(k; n, p) 0 q3 1 3pq 2 2 3p2 q 3 p3
Independence. Random Variables. Discrete RV's pmf
Why do binomial probabilities add to 1? Formula:
n
Independence Bernoulli trials Binomial distribution
(a + b)n =
k=0
n k n-k a b . k
n
Expectation Title Page
So 1 = (p + q)n =
n
k=0
n k n-k p q = k
b(k; n, p).
k=0
Page 28 of 37 Go Back Full Screen Close Quit
Example: A genetic disease which is rare but fatal is carried by a recessive gene. So with respect to this disease, the following are the possibilities for an individual:
Independence.
Aa AA aa
carrier completely free fatal
Random Variables. Discrete RV's pmf Independence Bernoulli trials
Suppose both the mother and father are both carriers: Mother =Aa, Father =Aa. A child of such a set of parents has the following genetic structure with given probabilities:
Table 3: Genetic structure and probabilities. AA 1/4 Aa 1/2 aa 1/4
Binomial distribution Expectation Title Page
Page 29 of 37 Go Back
Suppose such a set of parents has 4 children. Let N = # of offspring with fatal "aa".
Full Screen Close Quit
From Table 3, p = 1/4, and P [N = i] = 4 i 1 4
i
3 4
4-i
,
i = 0, 1, 2, 3, 4.
Independence. Random Variables. Discrete RV's
The numbers look like this (from Minitab): N pN (i) 0 0.316406 1 0.421875 2 0.210938 3 0.046875 4 0.003906
pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
In particular, the probability of having at least one fatally stricken child is P [N 1] = 1 - P [N = 0] = 1 - 0.316 = .684.
Page 30 of 37 Go Back Full Screen Close Quit
How to get this using Minitab. Open Minitab (duh!) At the top, click Calc. See the drop down menu. Choose Probability distributions and then choose binomial. To get the pmf, click the middle option Probability. If you want to compute b(k; n, p) for various values of k, specify Number of trials. For the previous examples this was n = 4. Probability of success. For the previous examples this was 0.25. The values of k. You need to choose an input column, say C1, and then put the values of k vertically in the chosen column. Then you can tell minitab where to put the output. I chose C2. If you don't specify an output column, results appear in the session window from which you can cut and paste to your heart's content. Press the button; that is, click ok. Voila! The probabilities appear in C2.
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 31 of 37 Go Back Full Screen Close Quit
Sure beats calculating the damn things by hand. Minitab also is willing to calculate the cumulative probabilities P [N j], j = 0, 1, . . . , n. It obligingly gives us: P [N 0] 0.31641 P [N 1] 0.73828 P [N 2] 0.94922 P [N 3] 0.99609 P [N 4] 1.00000
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 32 of 37 Go Back Full Screen Close Quit
8.
Expectation of a discrete rv.
Independence. Random Variables. Discrete RV's
Suppose (S, A, P) is a probability model and let X be a random variable. Imagine Do the experiment. Get an outcome s S. The random variable X gives us a number X(s). What number do we EXPECT before we do the experiment? Definition: The expectation E(X) of the discrete random variable X is E(X) :=
x{ possible values }
pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
xP [X = x] =
x{ possible values }
xpX (x).
Page 33 of 37
In words: To compute E(X):
Go Back
Take a possible value x and weight it by the probability P [X = x] the rv X takes that value. Do this for each possible value and sum.
Full Screen Close Quit
Remarks: E(X) is sometimes called the first moment in analogy with physics. If both S and X are discrete, this is also equal to E(X) =
sS
Independence. Random Variables.
X(s)P {s}
Discrete RV's pmf Independence Bernoulli trials
since by regrouping, the last sum can also be written as X(s)P {s}
x{ possible values } s:X(s)=x
Binomial distribution Expectation
=
x{ possible values } s:X(s)=x
xP {s} xP[X = x].
x{ possible values }
Title Page
=
Page 34 of 37
As with the arithmetic mean of a sample of numbers, the expectation need not be a possible value.
Go Back Full Screen Close Quit
Example 1: X pX (x) -1 1/2 1 1/2
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution
In this example, the possible values are {-1, 1} but 1 1 E(X) = (-1) + (1) = 0, 2 2 and 0 is not a possible value. Example 2: Throw a die: X pX (x) Now E(X) = 1 1/6
6 i=1
Expectation Title Page
2 1/6 i
3 1/6
4 1/6
5 1/6
6 1/6
6
=
21 = 3.5 {1, 2, 3, 4, 5, 6}. / 6
Page 35 of 37 Go Back Full Screen Close Quit
Mean of binomial. Suppose N has a binomial mass function. Notation: N b(k; n, p). Then n k n-k E(N ) = p q = k k k=0
n n
Independence. Random Variables. Discrete RV's
n! pk q n-k k k!(n - k)! k=1
n
n
pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
=pn
k=1
(n - 1)! n - 1 k-1 n-k pk-1 q n-k = pn p q (k - 1)!(n - k)! k-1 k=1
and changing dummy index from k to j = k - 1 gives
n-1
=pn
j=0 n-1
n - 1 j n-j-1 pq = pn j j=0
n-1
n - 1 j n-1-j pq j
Page 36 of 37 Go Back Full Screen Close Quit
=pn
j=0
b(j; n - 1, p) = pn 1 = pn.
Contents
Independence. Random Variables. Discrete RV's pmf Independence Bernoulli trials Binomial distribution Expectation Title Page
Page 37 of 37 Go Back Full Screen Close Quit