Proceedings of the 2005 Winter Simulation Conference
M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines, eds.
WORK SMARTER, NOT HARDER:
GUIDELINES FOR DESIGNING SIMULATION EXPERIMENTS
Susan M. Sanchez
Operations Research Department and
Graduate School of Business & Public Policy
Naval Postgraduate School
Monterey, CA 93943-5219, U.S.A.
We present the basic concepts of experimental design, the
types of goals it can address, and why it is such an important
and useful tool for simulation. A well-designed experiment
allows the analyst to examine many more factors than would
otherwise be possible, while providing insights that could not
be gleaned from trial-and-error approaches or by sampling
factors one at a time. We focus on experiments that can cut
down the sampling requirements of some classic designs by
orders of magnitude, yet make it possible and practical to
develop an understanding of a complex simulation model and
gain insights into its behavior. Designs that we have found
particularly useful for simulation experiments are illustrated
using simple simulation models, and we provide links to
other resources for those wishing to learn more. Ideally, this
tutorial will leave you excited about experimental designs—
and prepared to use them—in your upcoming simulation
The process of building, verifying, and validating a simula-
tion model can be arduous, but once it is complete, it’s time
to have the model work for you. One extremely effective
way of accomplishing this is to use experimental designs
to help explore your simulation model.
Before undertaking a simulation experiment, it is useful
to think about
this the experiment is needed. Simulation
analysts and their clients might seek to (i)
develop a basic
of a particular simulation model or system, (ii)
decisions or policies, or (iii)
compare the merits
of various decisions or policies (Kleijnen et al. 2005). The
goal will inﬂuence the way the study should be conducted.
The Feld called Design of Experiments (DOE) has been
around for a long time. Many of the classic experimental
designs can be used in simulation studies.
We discuss a
few in this paper to explain the concepts and motivate the
use of experimental design (see also Chapter 12 of Law
and Kelton 2000).
However, the environments in which
real-world experiments are performed can be quite different
from the simulation environment.
Table 1, adapted from
Sanchez and Lucas (2002), lists some of the assumptions
made in traditional DOE settings, as well as features that
characterize many simulation settings.
Three fundamental concepts in DOE are control, repli-
cation, and randomization.
means that the experi-
ment is conducted in a systematic manner after explicitly
considering potential sources of error, rather than by using
a trial-and-error approach. This tutorial should give you a
good understanding of controlled experiments.
can be viewed as a way to gain enough data to achieve