Refers to the consistency of a set of measurements when a testing procedure is
repeated on a population of individuals or groups
o
Random or unsystematic errors: occurrence or degree of the error does not appear
to be predictable
o
Systematic errors: the errors may be made in a consistent, or predictable fashion
o
Systematic errors
do not affect the accuracy of the measurements but rather the
meaning or interpretation of those measurements
Interpreting reliability coefficients
o
Another way to think of reliability is in terms of the variability of a set of scores
o
Reliability can be thought of as the ratio of true score variance, Var(T), to
observed score variance, Var(X)
o
o
is the reliability coefficient, the degree that observed sores, which are made on
the same stable characteristic, correlate with one another
Measurement error
o
Measurement error can be thought of as the hypothetical difference between an
individual’s observed score on any particular measurement and the individual’s
true score
o
Measurement error, whether systematic or random, reduces the usefulness of any
set of measures or the results from any test
o
It reduces the confidence that we can place in the score the measure assigns to any
particular individual
o
The
standard error of measurements
is a statistical index that summarizes
information related to measurement error
o
This index is estimated from observed scores obtained over a group of individuals
o
It reflects how an individual’s score would vary, on average, over repeated
observations that were made under identical conditions
Factors affecting reliability
The factors that introduce error into any set of measurements can be organized into three
broad categories.
o
Temporary individual characteristics
o
Factors such as health, motivation, fatigue, and emotional state introduce
temporary, unsystematic errors into the measurement process
o
Lack of standardization
o
Changing the conditions under which measurements are made introduces
error into the measurement process
o
Can be random or systematic
o
Chance
o
Factors unique to a specific procedure introduce error into the set of
measurements
o
Change factors produce unsystematic or random measurement errors
Methods of estimating reliability
o
To measure reliability, we have to estimate the degree of variability in a set of
scores that is caused by measurement error
o
We can obtain this estimate by using two different, but parallel, measures of the
characteristic or attribute
o
Over the same set of people, both measures should report the same score for each
individual
o
This score will represent the true score plus measurement error
o
Both measures reflect the same true score; discrepancies between the two sets of
scores suggest the presence of measurement error
o
The correlation coefficient based on the scores from both measures gives an
estimate of r
xx
, the reliability coefficient
o
