CHAPTER 11: META-ANALYSIS AND POWER ANALYSIS: EFFECT SIZE,
TYPE 1 ERROR
AND ALPHA, TYPE 2 ERROR AND BETA, AND DETERMINING n
PREVIEW AND INTRODUCTION
To market a new drug for depression or schizophrenia, its developers must show that it is
safe and efficacious. From the point of view of the pharmaceutical company, it would be best if all
these experiments yielded significant results, showing the drug more effective than placebo and/or
competing products. Nonsignificant findings are not, in the long run, good for the company’s
balance sheet. So, pone wants to run a big enough study to find an effect, if it is there.
But neither is it good for the balance sheet to run massive clinical trials with thousands of
research participants. While studies with larger numbers of research participants are more sensitive
and more likely to provide significant results, they are also more expensive. So even in the
industrial sector, where money and resources are often more easily available than they are for
academic research, there are limits on the size of experiments.
One use of power analysis involves designating how many research participants to use in a
specific experiment. Power analysis has other uses as well. For example, it can help us interpret a
study that yields nonsignificant results, helping us decide whether to do a larger, more sensitive
study or to abandon a line of research.
Another use of the principles underlying power analysis involves determining how much
difference an intervention makes. Unless the null hypothesis is absolutely true, a rare event in the
biomedical sciences, if you run a large enough study you will get statistically significant results. But
have you changed anything?
Hypothesis testing statistics can tell us whether an experimental group differs from its
control. But what about the individual? Which interventions make a big difference for many
individuals and which do not? Based on the work of Jacob Cohen, whose most well known book
underlies this chapter
, we calculate an effect size by simply dividing the difference between the
means of each treatment and control groups by the estimated standard deviation, the square root of
. By placing that difference on a normal curve, we can determine how the typical person in the
active treatment group compares to the average person in the control group. (A similar calculation
can be performed on studies using Pearson’s correlation coefficient.) Additionally, one can compute
an average effect size across all relevant studies of an intervention. This technique is called meta-
analysis. Articles reviewing areas of research for such journals as
premier venue for review articles in Psychology, now routinely use meta-analysis as their basic
method of evaluating the effect of an independent variable.
We will begin by calculating effect
sizes and then use effect sizes to perform meta-analyses. Then we will
look at Type 1 and Type 2
error and learn how to calculate the number of participants needed for a good study.
Definition 11.1: Power analysis