Researchers form questions about a population of interest, or target population. For example, they may be interested in testing a hypothesis about mothers with depression or homeless teenagers. It is rarely possible to have everyone in the target population participate in a study. Thus, researchers must select a small subset of the population as participants. This subset is called the sample. The goal is to produce a representative sample, a group of participants whose characteristics match those of the larger target population. Regardless of the research design used, failure to obtain a representative sample will produce inaccurate results because the results will not apply to the broader population of interest.
Meaningful results require careful use of sampling techniques—the way in which participants are selected to participate. Random sampling occurs when all members of a population of interest have an equal chance of being selected to participate in a study. As a result of random sampling, researchers can then generalize the results of their studies to the larger population. However, random sampling is difficult and costly. It is not always possible to obtain a full list of all members of a population (e.g., all students and faculty at a university). The institution may not wish to release this information, and even if the institution agrees to do so, the financial cost to obtain these data may be prohibitive. Sometimes, researchers settle for convenience sampling, recruiting participants from easiest-to-access members of a population. Undergraduate students are a common convenience sample, but they do not represent the broader population.
Representative and Nonrepresentative Samples
Experimenter Bias and Placebo Effects
Research findings can also be biased by the way a study is conducted. Experimenter bias occurs when a researcher who knows the intended outcome of the study influences the procedure or outcome in some way. For example, in an experiment, a researcher who believes (or hopes) a treatment is effective may interact more frequently with the treatment group than the control group. Members of the treatment group may behave differently under more frequent oversight from the researcher compared to members of the control group. Demand characteristics are cues leading participants to behave in a way they think is desirable in a study instead of behaving naturally. For example, a participant in a treatment study may choose not to mention lingering symptoms to avoid disappointing the researcher.
Relatedly, the placebo effect occurs when participants think they are in the experimental group and their thoughts, feelings, or behaviors change as a result of this expectation. In drug treatment studies, placebos are drug-free pills made to look like the real medication. Placebos have powerful effects, such as reducing perceptions of pain and fatigue, improving sleep, and boosting mood.
To control for participant expectations, many studies have a placebo group in addition to a no-treatment control group. For example, an experiment about the impact of caffeine on test performance could include three groups. The treatment group drinks 12 ounces of caffeinated coffee, the control group drinks nothing, and the placebo group drinks 12 ounces of decaffeinated coffee. Neither the treatment group nor the placebo group would be told whether their beverage contains caffeine. If the group that drank decaffeinated coffee performs more like the group that drank caffeinated coffee, that suggests that at least some of the benefits of consuming caffeine before a test depend on the belief that caffeine will help. If the group that drank decaffeinated coffee performs like the group that consumed nothing, this suggests that the perceived presence of caffeine has no effect on test performance.To avoid experimenter bias and placebo effects, high-quality treatment outcome studies use double-blind, placebo-controlled designs in which one group receives the treatment (e.g., a medication) and the other receives an inactive substance (e.g., a sugar pill). A double-blind procedure is one in which neither the experimenter nor the participant knows to which group the participant belongs. This guarantees the experimenter treats all participants the same away and minimizes the likelihood that participants will respond to demand characteristics.
Publishing and the Replication Crisis
A researcher's ultimate goal is to share results publicly by publishing them in a scholarly journal. To be published, a research article or manuscript must undergo thorough evaluation. Peer review in academic publishing involves having experts (generally two or three) in the field review a research article for clarity, quality of research design, and accuracy of statistical analyses. Reviewers provide feedback to the researcher, who must then revise their work to meet publishing standards.
However, in most scientific fields, publishers tend only to publish studies with new or innovative results. This can leave out research that attempts to replicate or confirm previous findings. A recent emphasis on replication has revealed that some established findings may be weaker than originally thought. For example, in the Reproducibility Project (2015), 270 authors collaborated in an attempt to repeat the results of 100 psychological studies by following the original study designs. Topics included learning, cognition, and personality. Replicators consulted with the original researchers to ensure fidelity. Of the 97 findings that were originally statistically significant, only 35 studies produced statistically significant results when the research was replicated.
Researchers have begun to address these issues as part of what is known as the Open Science movement. Increasingly, researchers are disclosing more details about their methods, even going so far as to share anonymous data so other researchers can verify their analyses. Additionally, researchers can preregister their hypotheses in an online forum. Preregistration involves specifying detailed hypotheses, methods, and analytic procedures before beginning a study. This approach keeps researchers from running endless variations on statistical analyses until they produce a significant result.