Meaningful results require careful use of sampling techniques—the way in which participants are selected to participate. Random sampling occurs when all members of a population of interest have an equal chance of being selected to participate in a study. As a result of random sampling, researchers can then generalize the results of their studies to the larger population. However, random sampling is difficult and costly. It is not always possible to obtain a full list of all members of a population (e.g., all students and faculty at a university). The institution may not wish to release this information, and even if the institution agrees to do so, the financial cost to obtain these data may be prohibitive. Sometimes, researchers settle for convenience sampling, recruiting participants from easiest-to-access members of a population. Undergraduate students are a common convenience sample, but they do not represent the broader population.
Representative and Nonrepresentative Samples
Experimenter Bias and Placebo Effects
Relatedly, the placebo effect occurs when participants think they are in the experimental group and their thoughts, feelings, or behaviors change as a result of this expectation. In drug treatment studies, placebos are drug-free pills made to look like the real medication. Placebos have powerful effects, such as reducing perceptions of pain and fatigue, improving sleep, and boosting mood.
To control for participant expectations, many studies have a placebo group in addition to a no-treatment control group. For example, an experiment about the impact of caffeine on test performance could include three groups. The treatment group drinks 12 ounces of caffeinated coffee, the control group drinks nothing, and the placebo group drinks 12 ounces of decaffeinated coffee. Neither the treatment group nor the placebo group would be told whether their beverage contains caffeine. If the group that drank decaffeinated coffee performs more like the group that drank caffeinated coffee, that suggests that at least some of the benefits of consuming caffeine before a test depend on the belief that caffeine will help. If the group that drank decaffeinated coffee performs like the group that consumed nothing, this suggests that the perceived presence of caffeine has no effect on test performance.To avoid experimenter bias and placebo effects, high-quality treatment outcome studies use double-blind, placebo-controlled designs in which one group receives the treatment (e.g., a medication) and the other receives an inactive substance (e.g., a sugar pill). A double-blind procedure is one in which neither the experimenter nor the participant knows to which group the participant belongs. This guarantees the experimenter treats all participants the same away and minimizes the likelihood that participants will respond to demand characteristics.
Publishing and the Replication Crisis
However, in most scientific fields, publishers tend only to publish studies with new or innovative results. This can leave out research that attempts to replicate or confirm previous findings. A recent emphasis on replication has revealed that some established findings may be weaker than originally thought. For example, in the Reproducibility Project (2015), 270 authors collaborated in an attempt to repeat the results of 100 psychological studies by following the original study designs. Topics included learning, cognition, and personality. Replicators consulted with the original researchers to ensure fidelity. Of the 97 findings that were originally statistically significant, only 35 studies produced statistically significant results when the research was replicated.
Researchers have begun to address these issues as part of what is known as the Open Science movement. Increasingly, researchers are disclosing more details about their methods, even going so far as to share anonymous data so other researchers can verify their analyses. Additionally, researchers can preregister their hypotheses in an online forum. Preregistration involves specifying detailed hypotheses, methods, and analytic procedures before beginning a study. This approach keeps researchers from running endless variations on statistical analyses until they produce a significant result.