3 Chapter Part 1 Probability September 9, 2008
The goals and skill set apply to all of Chapter 3
Goal:
All of inferential statistics rests firmly on probability. So these lectures on probability are designed to give you a good working knowledge of the basics of probability.
Skill Set:
You should be able to solve basic probability problems using the addition and multiplication principles and Bayes' Rule. You should know how to calculate and interpret sensitivity, specificity, predictive value of a positive test, and predictive value of a negative test. You should know how to obtain and interpret an ROC (Receiver Operating Characteristic) curve.
Outline
Part 1 Brief overview of sampling Definition of probability Definition of mutually exclusive events Definition of complementary events Multiplication Principle Definitions of dependent and independent events Addition Principle Problems in interpreting large batteries of tests Definition of conditional probability Definitions of True positive False positive True negative False negative Definitions of Sensitivity Specificity False negative rate False positive rate Predictive value of a positive test Predictive value of a negative test Page 1 Page 5 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 18
Page 19
There exists a
Used to describe Population Population Parameters
A statistician draws a
Used to estimate
Random Sample
Statistics
The sample generates
Used to calculate pertinent Numerical Data
A summary of the manner in which a statistical study is conducted.
Page -1-
Brief introduction to sampling, in particular random sampling
Rosner discusses this in detail in a later chapter but because in some sense probability is the study of randomness, I would like to give just a brief introduction prior to discussing probability. Samuel Johnson is quoted as saying "You don't have to eat the whole ox to know the meat is tough." This is the basic idea behind sampling, you wish to gain information about the whole by studying just a part.
The basic definitions for sampling are:
Population (sometimes called the target, reference or study population) the entire group of objects about which information is desired. Example: The set of all people in the U.S. who are older than 65.
Unit - any individual member of the population. Example: Mr. John Randolph Smith who lives in Virginia is 70 years old, so Mr. Smith is a unit of the above population.
Sample - a part or subset of the population used to gain information about the whole population. Example: 1000 members of AARP who are older than 65 (probably not representative of all people over 65).
Sampling frame - the list from which the sample is chosen. Example: The list of AARP members who are older than 65.
Variable - a characteristic of a unit, to be measured for those units in the sample. Example: Diastolic blood pressure.
Page -2-
Convenience sampling - selection of whatever units of the population are easily accessible. Example: If you selected the first 100 shoppers at a grocery store who said they were older than 65, you'd have a convenience sample.
Convenience sampling frequently gives us results that are not representative of the population from which the sample was drawn. A solution to this problem is to use a simple random sample. The idea is to give each unit in the sampling frame the same chance to be chosen for the sample as any other unit. The formal definition is: A simple random sample of size n is a sample of n units chosen in such a way that every collection of n units from the sampling frame has the same chance of being chosen. Example of what is not a simple random sample: Suppose you wanted to measure the total cholesterol of a sample of Dr. Patel's patients. If you decided to screen everyone who came to the clinic on Wednesday, you would not have a random sample. The people who come to the clinic on Thursday, for example, would have zero probability of being tested. A parameter is a numerical characteristic of the population. It is a fixed number, but we usually don't know its value. Parameters are usually denoted by Greek letters so the mean, for example, is usually represented by .
Example:
The mean of the diastolic blood pressures of each person over 65 in the U.S. is a parameter since it is the mean of our population. We don't usually know the value of parameters.
A statistic is a numerical characteristic of the sample (e.g. the mean of the age of members of the sample). The value of a statistic is known once we have taken a sample, but it changes from sample to sample. We represent the mean of the sample by x .
x
is the sample estimate of
the population parameter.
Example:
The mean of the diastolic blood pressure of the sample of 1000 AARP members older than 65 would be a statistic.
So a parameter is to a population what a statistic is to a sample. Suppose we think of the true value of the population parameter as the bull's-eye on a target, and the sample statistic as a bullet fired at the bull's-eye. Bias means that our Page -3-
sight is misaligned and we shoot consistently off the bull's-eye in one direction. Our sample values do not center about the population value. Lack of precision means that repeated shots are widely scattered on the target; that is, repeated samples do not give similar results but instead give results that vary widely among the samples.
Facts about simple random sampling:
1) Despite the sampling variability of statistics from simple random sample to simple random sample, the values of those statistics have a known distribution in repeated sampling. The Central Limit Theorem, which we will get to eventually, says the sample means follow a normal distribution if the size of the samples is large enough. 2) The precision of a statistic from a simple random sample depends on the size of the sample and can be made as high as desired by taking a large enough sample. This is why we worry about small samples.
Most of the ideas above are taken from David S. Moore's book titled Statistics Concepts and Controversies, 2nd edition, W.H. Freeman and Company, 1979.
Now what we understand is that if we select 2000 samples of size N = 100 from a given population, the means of the samples would not all be the same but would vary from sample to sample. So when we test a drug versus a placebo with the drug tested on one sample of people and the placebo tested on another sample, we have to worry about whether the difference that we see in the outcome of the study is due to the variation among samples (i.e. due to random variation) or is due to the impact of the drug. We would like to be able to make inferences about the effectiveness of the drug versus that of the placebo based on the observations from the samples. The branch of mathematics that allows us to make such inferences is called probability.
Page -4-
Probability
First Principles of Probability
1) If each outcome of an experiment is equally likely, then the probability of each outcome is
1 n
where
n is the
number of outcomes of the experiment.
2)
If an experiment can result in
n equally likely outcomes, n A
of
which have attribute A, then we say that attribute A has probability
nA n
and we write
Pr( A ) =
nA . n
The above principles form the basis for the frequentists version of probability. The other version is Bayesian. When we get to Bayes' Rule, I will talk a little about Bayesian statistics. We are going to set up a small experiment. Suppose that a penny and a nickel are tossed together. What is the probability of getting 2 heads? At first we might think that there are only three possible outcomes - 2 heads, 1 head or no heads and that only one of these has the proper attribute (i.e. 2 heads) so the probability is 1/3. But the problem is that these three events are not equally likely. This is because although there is only one way you can get two heads (both the penny and nickel come up heads) and only one way you can get two tails, there are two ways that you can get one tail and one head - namely, the penny comes up heads and the nickel comes up tails or the penny comes up tails while the nickel comes up heads. So the set of all possible equally likely outcomes (i.e. the sample space) is HH, TT, HT, Page -5-
TH where the first letter represents the outcome for the penny and the second letter represents the outcome for the nickel (i.e. TH = the penny is tails and the nickel is heads). The event of interest is HH and using the first principal of probability we get Pr(HH) = 1/4. One forth is the relative frequency of the event. It also true that Pr(HT) = Pr(TH) = Pr(TT) = Pr(HH) = 1/4 Now we can find the probability of 2 heads, 1 head and no heads. We'll use the second principle of probability. Pr(2 heads) = 1/4 the outcome is HH
Pr(1 head) = 2/4 = the outcome is either HT or TH so Pr(0 heads) = 1/4 the outcome is TT
nA = 2
From the principles of probability we can see that for any attribute A, Pr(A) $ 0 since
nA
is a count (i.e. the number of people who have attribute A) and is therefore greater
than or equal to zero. The number is assumed to be greater than zero since any study that has no outcomes is pretty uninteresting. Also Pr(A) # 1 since the
n
nA
outcomes are a subset of the
n total outcomes, so
n A n (the subset could equal the whole) and hence
So for any attribute A
0 # Pr(A) # 1
nA 1. n
What would be the outcomes of interest be if we had asked: What is the probability of obtaining at least one head or at least one tail in our experiment above (let us call this event Q) ? Well HH, HT, TH all have at least one head. It is also true that TT, HT and TH all have at least one tail. Therefore the set of outcomes that has the attribute of interest is {HH, TT, HT, TT} (we count each outcome only once and the set of outcomes is written in sample notation). So the probability of the event is one (i.e. P(Q) = 1) because = 4 (the total number of equally likely out comes of the experiment) and
n
nQ = 4.
Recognize that the outcomes that satisfy the attribute of at least one head or
Page -6-
at least one tail constitute what we call the Universe or sample space (i.e. all possible outcomes) and this universe is represented by U. So we have Pr(U) = 1. What is the probability of having an outcome of three heads in our experiment above (say outcome W)? Well each of two coins only once). So Pr(W) = 0 where W = (where is the empty set). So the probability of the whole sample space is 1 and the probability of none of the outcomes having the attribute is zero. Let A denote the absence of attribute A (also called the occurrence of "not Suppose the box in Figure 1 below is the Universe.
nW = 0 (i.e. it is not possible to get three heads when you flip
A ").
As we stated above, the Universe for our experiment with the coins is U = {HH, HT, TH, TT}. The means that the entire inside of the box represents those 4 outcomes. If
A
is the event that you get at least one head when you flip the penny and the nickel, So Pr( A ) = 3/4 and Pr( A ) = 1/4. Which
A = {HH, HT, TH} and A = {TT}. means that Pr( A ) + Pr( A ) = 1.
then Figure 1
The probability of A is represented by the textured area and the probability of
A is represented by the white space within the box.
A
A = not A
A and A cover the entire Universe (represented by the box in Figure 1) and there is no overlap in the two sets (i.e. there is no outcome which is in both sets).
So we suspect the following is true:
Pr ( A) + Pr ( A) = 1 and Pr ( A) = 1 - Pr ( A)
Page -7-
Below we use our original definition for probability to prove the statements in the box immediately above and illustrated by Figure 1. If there are attribute
A , then n - n A outcomes have A (i.e. all the rest of the outcomes have attribute "not A "). This means
n outcomes n A
of which have attribute
Pr( A ) + Pr( A ) = and Pr( ) A =
nA n - nA + =1 n n
n - nA n = 1- A n n
= 1 - Pr( A )
So the equations above hold not just for the experiment with the coins, but for all experiments. Events that cannot occur at the same time (like A and "not A") are said to be mutually exclusive events. Still using our coin example above, let D = no heads = {TT} and E = exactly one head = {HT, TH}. Then D and E are mutually exclusive events because they have no outcome in common. Pr(D or E) = P(TT, HT, TH) = 3/4 = 1/4 + 2/4 = Pr(D) + Pr(E). The equation Pr(D or E) = Pr(D) + Pr(E) works because there is no overlap.
Figure 2
D and E are mutually exclusive events
The probability of D is represented by the area labeled D. And the probability of E is represented by the area labeled E. So we see that these two mutually exclusive events do not cover the whole Universe as A and
D = {TT}
A do.
E = {HT, TH}
Figure 2 is definitely not drawn to scale.
Page -8-
The complement of an event B consists of all the outcomes in the sample space that are not in B. So the complement of B is B . This means that an event and it's complement are also mutually exclusive. So complementary implies also mutually exclusive. The implication does not however, go the other direction. D and E are mutually exclusive but not complementary (see Figure 2 above and the fact that the sum of their probabilities = 3/4). Figure 3A In probability (and mathematics) we use the "inclusive or" meaning that when we say A or B occurs, it means that exactly one of the following occurs: only A occurs, only B occurs or both A and B occur. Instead of saying "or" we can use the symbol "^" as in the union of two events.
Figure 3B
In Figure 3A: Pr(A or B)= Pr(A ^ B) = the shaded area in the graph
In Figure 3B The area that is both hatched and solid is called the intersection of A and B and it has to be counted only once in order to get Pr(A or B)= Pr(A ^ B)
Page -9-
Figure 4
AB
The intersection of events A and B consists of the outcomes that are in both A and B. The intersection of A and B is represented by the symbol "_". Pr(A and B) = Pr(A _ B) = the shaded are in the graph on the left.
A
B
If A and B are mutually exclusive, then Pr(A and B) = Pr(A _ B) = 0. A and B are mutually exclusive in Figure 3A. A and B are not mutually exclusive in Figures 3B and 4 If A and B are complementary, then Pr(A or B) = Pr(A ^ B) = 1 (see Figure 1)
Multiplication Principle
If an event can occur in "a" ways and if a second event can occur in "b" ways independent of the ways in which the first event has occurred, then the two events together can occur in ab ways. Example: Suppose a clinical laboratory has available 4 methods for determining a patient's serum cholesterol level and 2 methods for determining his/her blood glucose level. Assume further that any combination of methods for determining cholesterol and glucose level can be used. Then the patient's serum cholesterol and blood glucose levels can be determined in 42 = 8 ways. If we label the four ways of determining cholesterol: I, II, III, and IV and label the two ways of determining glucose: 1 and 2, then the total number of ways of determining cholesterol and glucose are: I1, I2, II1, II2, III1, III2, IV1 and IV2.
Page -10-
1 I 2 1 2 1 III 2 1 IV 2
Cholesterol Glucose
II
The phrase indicating that any combination of cholesterol technique and glucose technique can be used together is an important part of the problem. Suppose for purposes of efficiency or perhaps for cost saving reasons, the laboratory was instructed to always use method 2 for glucose when method IV was used for cholesterol. Now how we measure glucose depends in part on how we measure cholesterol. The number of combinations is now 7 instead of 8 because IV1 is no longer a possibility.
Two events A and B are said to be independent if and only if Pr(A and B) = Pr(A _ B) = Pr(A)Pr(B). Two events are said to be dependent if Pr(A _ B) ... Pr(A)Pr(B).
Page -11-
Addition Principle
If one event can occur in "a" ways and if a second event can occur in "b" ways, then one or the other of the events can occur in a + b ways, provided the two events cannot occur together (i.e. the events are mutually exclusive). Suppose we want to select a single card from a deck of 52 cards. Event "a" is selecting a jack and event "b" is selecting a heart. How many ways can "a" or "b" occur? We know there are four ways of selecting a jack (there is one jack of hearts, one jack of diamonds, one jack of spades and one jack of clubs) and 13 ways of selecting a heart (there are 13 hearts, 13 diamonds, 13 spades and 13 clubs). However, it is not the case that there are 17 (4 + 13) ways of selecting either a heart or a jack because there are only 16 cards that are either hearts or jacks (one card is a jack of hearts and would be counted twice if we add 4 and 13). So Pr(A or B) = Pr(A ^ B) = Pr(A) + Pr(B) [i.e. Pr(A and B) = Pr(A _ B) = 0]. See Figures 2, 3A and 3B above. if and only if A and B are mutually exclusive
If A and B are not mutually exclusive then, as with the cards, we have to worry about not counting the combinations that can occur together more than once. So the more general form of the addition principle is (see Figure 3B above) Pr(A or B) = Pr(A ^ B) = Pr(A) + Pr(B) - Pr(A _ B) = Pr(A) + Pr(B) - Pr(A and B). or Pr(A or B) = Pr(A) + Pr(B) - Pr(A and B)
The probability part of these notes comes primarily from Statistics with Applications to Biological and Health Sciences by Richard Remington and Anthony Schork, PrenticeHall, Inc, 1970. Page -12-
Conditional probability:
Pr(B|A) is read as the probability of B given A and is defined as the probability of event B occurring given that event A has occurred or the probability of event B conditional on event A.
Suppose we let B be the event that on rolling a single die we get a number # 3. Let A be the event that on rolling a single die we get an even number. The universe (i.e. the complete set of possibilities for the outcome) for rolling a single die is {1,2,3,4,5,6} So Pr(B) = 3/6 = 1/2 since there are 6 equally likely outcomes only three of them are # 3, namely 1, 2 and 3. Once the event A has occurred, the universe is changed to {2,4,6}. That is, once A has occurred the universe under consideration is no longer {1,2,3,4,5,6} but is instead {2,4,6}. So Pr(B|A) = 1/3 since knowing that A has occurred means that there are only three equally like events (namely 2, 4 and 6) and exactly one of them is # 3. So knowing that event A has occurred changes the sample space from {1,2,3,4,5,6} to {2,4,6}.
The formal definition of conditional probability is:
Pr ( B| A ) =
Pr ( A and B ) Pr ( A )
For our example above Pr(B and A) = Pr(a number # 3 and an even number) = Pr(of a 2) = 1/6 Pr(A) = Pr(an even number) = 3/6
Page -13-
So Pr(B|A) = (1/6)/(3/6) = 1/3 (fortunately the same answer we got earlier).
5 4 6
A
2
B
1 3
The inside of the box to the right is the sample space for the unconditional probability of B i.e. Pr(B)
The striped area to the left...