3 Chapter Part 2 Probability September 11, 2008
Start reading Chapter 4
The goals and skill set apply to Chapter 3 Parts 1 and 2 combined.
Goal:
All of inferential statistics rests firmly on probability. So these lectures on probability are designed to give you a good working knowledge of the basics of probability.
Skill Set:
You should be able to solve basic probability problems using the addition and multiplication principles and Bayes' Rule. You should know how to calculate and interpret sensitivity, specificity, predictive value of a positive test, and predictive value of a negative test. You should know how to obtain and interpret an ROC (Receiver Operating Characteristic) curve.
Outline
Part 1 Brief overview of sampling Definition of probability Multiplication Principle Addition Principle Problems in interpreting large batteries of tests Conditional probability True positive False positive False negative False positive Sensitivity Specificity Probability of a positive test Probability of a negative test Part 2 Bayes' Rule MI versus non-MI example CT image example ROC curves Stata: roctab
Page 1 Page 3 Page 5 Page 7
Bayes' Rule allows us to define the positive and negative predictive values in terms of sensitivity and specificity of the test and prevalence of the disease. That is, it allows us to consider the patient (has the disease or doesn't have the disease) in terms of the results of the test.
Bayes' Rule:
P(PV + ) = P(D+|T+ ) =
P(D + and T+ ) P( T+ )
Step 1 Definition of PV+ and conditional probability
=
P(D + and T+ ) P(T + and D+ ) + P(T + and D- )
Step 2 See picture below
P ( D + ) P( T+| D + ) = [P(D+ ) P(T+|D+ )] + [P(D- ) P(T+|D- )]
P(PV+) = P(D+|T+) =
Step 3 See explanation on next page
prevalence sensitivity [prevalence sensitivity] + [(1 - prevalence) (1 - specificity)]
Step 4 To get from Step 3 to Step 4 we use the definitions of prevalence, sensitivity and specificity
Proof by picture that P(T+) = P(T+ and D+) + P(T+ and D-) You can also see in the picture that T- is everything outside the circle so that Pr(D-) = Pr(D- and T+) + Pr(D- and T-)
Page -1-
Step 3 uses the fact that P(T+|D+) = P(T+ and D+)/P(D+) by definition. So multiplying both sides of the equation by P(D+) gives you P(D+)P(T+|D+) = P(T+ and D+) so P(T+ and D+) = P(D+)P(T+|D+) P(D-)P(T+|D-) = P(T+ and D-) follows from a similar argument. Step 4: P(T+ | D-) = 1 - specificity because P(T+ | D-) + specificity =
P(T + | D- ) + P(T - |D- ) =
P(T + andD- ) P(T - andD- ) P(D- ) + = =1 P(D- ) P(D- ) P(D- )
(picture above where T- is everything outside the circle)
Similarly,
P(D-|T-) = P(PV -) =
(1 - prevalence) specificity [(1 - prevalence) specificity] + [prevalence (1 - sensitivity)]
The data above is made up. The point is to show you how the positive and negative predictive values shift as prevalence of the disease changes. In considering Scenarios 1 - 4, we can see that as the prevalence increases (while holding sensitivity and specificity fixed) the positive predictive value increases and the negative predictive value decreases. So prevalence has a major impact on positive and negative predictive values. Reverend Thomas Bayes (c. 1702 - April 17, 1761) was a British mathematician and Presbyterian minister, known for having formulated a special case of Bayes' theorem, which was published posthumously. The website said he is buried in a cemetery that contains many nonconformists. One of his major mathematical publications was a defense of the logical basis of Newton's calculus. Page -2-
MI versus Non-MI example:
In this problem we see what happens to sensitivity and specificity as you change the cutoff or referent value (i.e. as you change the definitions of T+ and T-).
Figure A: Frequency distribution of enzyme activity in people with and without MI. 60 is called the referent value because if x 60, the test is considered positive and If x < 60, the test is considered negative.
The vertical line is at 60 enzyme test units. Because 60 is a referent value, all values to the left of the line are considered to be negative for the test (T-). All values to the right of the line plus the value 60 are considered to be positive for the test (T+)
Percent of Total Cases
20
25
30
15
T+ T-
0 0
5
10
20
40
60
80
100
120
140
160
180
Enzyme Test units
The area under the dashed curve on the right hand side of the graph is considered positive for an MI (D+). The area under the solid curve on the lefthand side is considered negative for an MI (non-MI or D-). Notice that some points fall in both the MI and non-MI areas. The hatched area is P(T+|D+ or MI) = sensitivity of the test.
Figure C: Frequency distribution of enzyme activity in people with and without MI. Hatched area = specificity = P(T-|D-)
30 Percent of Total Cases 5 10 15 20 25
Non-MI MI
The hatched area to the left is P(T-|D- or non-MI) = specificity of the test. If we decide the referent value is 80, then the sensitivity decreases and the specificity increases.
180
0 0
20
40
60
80
100
120
140
160
Enzyme Test units
Page -3-
The MI problem above is from Beyond Normality: The Predictive Value and Efficiency of Medical Diagnoses by Galen and Gambino
Below is a table which shows with numbers what we showed above with pictures. What happens to sensitivity and specificity as you change the cutoff or referent value? The values below are derived from the figure above, assuming the prevalence of disease is 50%.
Referent Value 20 40 50 60 80 100
Sensitivity (%)
Specificity (%)
PV+ (%)
PV- (%)
100 95 90 85 70 50
60 75 80 85 95 100
71.4 79.2 82.8 85.0 93.3 100.0
100.0 93.8 88.9 85.0 76.0 66.7
Below we look at another way of changing the referent point.
Page -4-
Table 3.3 (page 65 in Rosner): Ratings of 109 CT images by a single radiologist
CT rating by radiologist
True Disease Status Normal (D-) Abnormal (D+) Total Definitely normal (1) 33 3 36 Probably normal (2) 6 2 8 Questionable (3) 6 2 8 Probably abnormal (4) 11 11 22 Definitely abnormal (5) 2 33 35 Total 58 51 109
Note that Normal = D- and Abnormal = D+. The above data of Hanley and McNeil are ratings of computed tomographic (CT) images by a single radiologist in a sample of 109 subjects with possible neurologic problems. The true disease status is also known for each of these subjects. The question is how can we quantify the diagnostic accuracy of the test. The earlier problems have all been normal or abnormal not shades of grey. Below we see what changing the cutpoint for normal and abnormal does. An ROC (receiver operating characteristic) curve allows us to see the changes in sensitivity (y-axis) that go with changes in 1 - specificity (x-axis) produced by changing the definition of a positive test. Page -5-
I have taken Rosner's Table 3.3 and put it in the format of the tables we used to define sensitivity etc. So D+ is column 1 and D- is column 2. T+ is the upper rows and T- the lower rows.
CT rating by radiologist
Abnormal (D+) Definitely abnormal Probably abnormal Questionable Probably normal Definitely normal Total 33 11 2 2 3 51 Normal (D-) 2 11 6 6 33 58 Total 35 22 8 8 36 109
Page -6-
Table 3.3 Ratings of the CT images
For the true disease status, normal is D- and abnormal is D+, For test status, normal is T- and abnormal is T+.
First Choice:
The definition of T+ (and T-) will change as we change the cutpoint, but the first choice will be T+ = {definitely abnormal} The other 4 categories (probably abnormal, questionable, probably normal and definitely normal) will be considered T-. So the 2 by 2 table for this first choice is as follows:
Disease
Present (D+) Positive (T+) 33 11+2+2+3 = 18 51 Absent (D-) 2 11+6+6+33 = 56 58 Total 35 74 109
Test Result
Negative (T-)
So for the first choice for T+, (TP = 33, FP = 2, FN = 18 and TN = 56.)
sensitivity = Pr(T+|D+) = Pr(T+ and D+)/Pr(D+) = 33/51 = 0.65 and specificity = Pr(T-|D-) = Pr(T- and D-)/Pr(D-) = 56/58 = 0.97 so 1 - specificity = 1 - 0.97 = 0.03. The point on the ROC curve is then (0.03,0.65). Page -7-
Second Choice:
The second choice for T+ = {definitely abnormal, probably abnormal}. The other 3 categories (questionable, probably normal and definitely normal) will be considered T-. So the 2 by 2 table for this second choice is as follows:
Disease
Present (D+) Positive (T+) 33+11 = 44 2+2+3 = 7 51 Absent (D-) 2 + 11 = 13 6+6+33 = 45 58 Total 57 52 109
Test Result
Negative (T-)
So for the second choice for T+, sensitivity = Pr(T+|D+) = Pr(T+ and D+)/Pr(D+) = 44/51 = 0.86 and specificity = Pr(T-|D-) = Pr(T- and D-)/Pr(D-) = 45/58 = 0.78 so 1 - specificity = 1 - 0.78 = 0.22. The point on the ROC curve is then (0.22,0.86).
Page -8-
Third Choice:
The third choice for T+ = {definitely abnormal, probably abnormal, questionable}. The other 2 categories (probably normal and normal) definitely will be considered T-. So the 2 by 2 table for this third choice is as follows:
Disease
Present (D+) Positive (T+) 33+11+2 = 46 2+3 = 5 51 Absent (D-) 2 +11+6 = 19 6+33 = 39 58 Total 65 44 109
Test Result
Negative (T-)
So for the third choice for T+, sensitivity = Pr(T+|D+) = Pr(T+ and D+)/Pr(D+) = 46/51 = 0.90 and specificity = Pr(T-|D-) = Pr(T- and D-)/Pr(D-) = 39/58 = 0.67 so 1 - specificity = 1 - 0.67 = 0.33. The point on the ROC curve is then (0.33,0.90).
Page -9-
Fourth Choice: T+ = {definitely abnormal, probably abnormal, questionable, probably normal} and T- = {definitely normal}. So the 2 by 2 table for this forth choice is as follows:
Disease
Present (D+) Positive (T+) 33 + 11 + 2 + 2 = 48 3 51 Absent (D-) 2 + 11 + 6 + 6 = 25 33 58 Total 73 36 109
Test Result
Negative (T-)
So for the fourth choice for T+, sensitivity = Pr(T+|D+) = Pr(T+ and D+)/Pr(D+) = 48/51 = 0.94 and specificity = Pr(T-|D-) = Pr(T- and D-)/Pr(D-) = 33/58 = 0.57 so 1 - specificity = 1 - 0.57 = 0.43. The point on the ROC curve is then (0.43,0.94).
Page -10-
The two choices for cutpoints that are not so intuitive as Choices 1 - 4 are Choice 0 where all tests negative and Choice 5 where all tests are positive. These are choices that you'd probably never make in clinical practice but they provide the anchor points for the ROC curve.
Choice 0:
For Choice 0 and T- = {definitely normal, probably normal, questionable, probably abnormal, definitely abnormal}
Disease
Present (D+) Positive (T+) 0 51 51 Absent (D-) 0 58 58 Total 0 109 109
Test Result
Negative (T-)
Sensitivity = Pr(T+|D+) = 0/51 = 0 1 - specificity = 1 - 1 = 0
Specificity = Pr(T-|D-) = 58/58 = 1
So the point on the ROC curve is (0, 0).
Page -11-
Choice 5:
T+ = {definitely normal, probably normal, questionable, probably abnormal, definitely abnormal}
Disease
Present (D+) Positive (T+) 51 0 51 Absent (D-) 58 0 58 Total 109 0 109
Test Result
Negative (T-)
Sensitivity = Pr(T+|D+) = 51/51 = 1 Specificity = Pr(T-|D-) = 0/58 = 0 1 - specificity = 1 - 0 = 1 So the point on the ROC curve will be (1, 1)
Page -12-
ROC Curve for Table 3.3 page 65 of Rosner
1 0.94 0.90 0.86
Sensitivity = Pr(T+|D+)
0.65
0
0 0. 03
22
33
0.
I don't recommend that you label axes in this odd manner. I did it so you can clearly see that the 6 points derived from the Table 3.3 are what have been graphed.
In order to be able to graph the ROC curve and obtain the PV+ and PV- I created a small Stata file which is at the end of this handout.
0.
1 - Specificity = 1 - Pr(T-|D-)
0.
43
Page -13-
1
ROC (Receiver Operating Characteristic) curves: The area under the ROC curve is considered an estimate of the overall diagnostic accuracy of the test. It can be shown that this area, when calculated by the trapezoidal rule, corresponds to the probability that for a randomly selected pair of normal and abnormal subjects, the radiologist will correctly identify the normal subject given the CT ratings. It is assumed that for untied ratings the radiologist designates the subject with the lower test score as normal, and the subject with the higher score as abnormal. For tied ratings, it is assumed that the radiologist randomly chooses one patient as normal and the other as abnormal. How you come up with the trapezoids is given in the graph below.
ROC Curve for Table 3.3 page 65 of Rosner
1 0.94 0.90 0.86
Sensitivity = Pr(T+|D+)
Area of Triangle
0.65
+
Area of Rectangle Area of Trapezpoid
0
=
0 03
22
33
0.
0.
The area under the above ROC curve is 0.8932. "The area under the ROC curve measures the probability, denoted by , that in a randomly paired normal and abnormal images, the perceived abnormality of the two images will allow them to be correctly identified." The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve by James A. Hanley and Barbara J. McNeil, Radiology 143 (1): 29 - 36, April, 1982.
0.
1 - Specificity = 1 - Pr(T-|D-)
0.
43
Page -14-
1
The plots above were created using the numbers in the table on page 16.
Page -15-
CT rating by radiologist
Prevalence 0.05
Sensitivity Pr(T+|D+) Choice 0 Specificity Pr(T-|D-)
Prevalence Prevalence 0.50 0.95 PV+ PV0.50 1 0.99 0.98 0.98 0.95 PV+ PV0.05 0.13 0.23 0.27 0.34
PV+
PV0.9500
0 0.65 0.86 0.90 0.94 1
1 0.97 0.78 0.67 0.57 0 0.50 0.17 0.13 0.10 0.05
Choice 1
0.9811 0.95 0.73 0.9908 0.79 0.85 0.9924 0.73 0.87 0.9946 0.69 0.91 0.50
Choice 2
Choice 3
Choice 4
Choice 5
Page -16-
Page -17-
Page -18-
Page -19-
. des Contains data from table3pt3.dta obs: 10 vars: 4 10 Sep 2006 20:04 size: 110 (99.9% of memory free) ------------------------------------------------------------------------------storage display value variable name type format label variable label ------------------------------------------------------------------------------id float %9.0g disease byte %13.0g diseaselbl True status of disease rating byte %19.0g ratinglbl CT rating by Radiologist pop byte %8.0g Number of images ------------------------------------------------------------------------------Sorted by: . label list diseaselbl: 0 1 ratinglbl: 1 2 3 4 5
Normal (D-) Abnormal (D+) Definitely Normal Probably Normal Questionable Probably Abnormal Definitely Abnormal
. list,nolab +-------------...