are Surveys systems for collecting information
to describe, compare, and predict attitudes,
opinions, values, knowledge, and behavior.
Matt Campbell, Richard Couchon,
Robert Garland, Rheuben Herbert
April 9, 2009
Planning of the Questionnaire
Question Writing
Question Order
Questionnaire Format and Appearance
A questionnaire is a research instrument
consisting of a series of questions and other
prompts for the purpose of gathering
information from respondents. The
questionnaire was invented by Sir Francis
Galton.
Consider the advantages and disadvantages
of using questionnaires
Self-Administered Surveys vs.
Interview Surveys
Issues Relating to Costs (Money, Time,
Manpower), Response Rate, and
Quality of Information
Prepare Written Objectives for
the Research
Sources of error in surveys: Sampling
error, coverage error, measurement
error, and non-response error
Establish context for the instrument
Decide the purpose of the survey
What is to be measured?
Decide on the target population of the survey
and who is to be included/excluded.
Certain goals require experimental and
control groups
Measurement error (quality of response) and
non-response error (quantity of response) are
often overlooked and can be addressed with a
quality questionnaire
All decisions that go into survey design need to
fit together and support one another in a way
that encourages most people to respond and
minimize inaccurate or inadequate answers
1
Goals to Elicit Response
Make Questionnaire Interesting
Avoid Inconvenience
Make Task Seem Important
Make Questionnaire Seem Short and Easy
What types of decisions do you think need to be
made in the process of designing a survey?
Survey medium
(email, mail, phone)
Question Wording
Question Type
Questionnaire
Length
Questionnaire and
Page Format
Rewards
Does the question require an answer?
Do survey recipients have an accurate, readymade answer for the question?
Can people accurately recall past behaviors?
Is the respondent willing to reveal the requested
information?
Will the respondent feel motivated to answer
each question?
Is the respondents understanding of response
categories likely to be influenced by more than
words?
Is survey information being collected by more
than one mode?
Are all of the variables to be measured
addressed by the questions used in the
questionnaire?
Is changing a question acceptable to the survey
sponsor?
Knowing question types is important to be
able to address your goals and to make
sure questions fall into one of the following
categories:
Open-Ended
Closed-Ended with Ordered Responses
Closed-Ended with Unordered Responses
Open-Ended Questions
Can lead to incomplete answers compared to
those obtained in interview surveys
Ex: Why did you purchase a new car?
Could still be appropriate if the researcher does
not know reasonable responses to make a
closed-ended question or may be eliciting
responses to use on future close-ended
questions.
2
Open-Ended Questions
Some open-ended questions are useful both
because of the information they provide and the
simplicity they bring to a survey
Ex: What make and model of car do you drive?
_____(make) _____ (model)
Providing labels can focus responses to be more
consistent and accurate and thus more useful
Open-Ended Questions
Breaking questions into multiple parts can
also produce more specific and useful
responses
Ultimately, open-ended questions should
be limited because they are difficult to
analyze
Closed-Ended Questions with Ordered Responses
Closed-Ended Questions
A closed-ended question is one which can
be answered by a specific, simple piece of
information. This includes, but is not
limited to, yes/no answers and multiple
choice. There are two main types, those
with ordered responses and those with
unordered responses.
Possible Scales:
Strongly Agree to Strongly Disagree
Very Favorable to Very Unfavorable
Excellent to Poor
Extremely Satisfied to Extremely Dissatisfied
High Priority Goal to Low Priority Goal
Complete Success or Complete Failure
Scale of 1-5,7,10, or 100
Scale of -3 to 3
Closed-Ended Questions with Ordered Responses
Closed-Ended Questions with Unordered Responses
Issues to Consider
Needing to extend the scale in order to spread
out any clustering of answers
Issues to Consider
Require more effort and comprehension yet often
provide the most useful information
Need limit to number of options to be ranked and
the amount of detail in options to ease the
demand on respondents
Partially close-ended questions
Though fewer categories is more common for
telephone interviews for ease of comprehension
Number scales are easier to comprehend and
interpret but strip away much of the meaning
behind the responses
Scale needs to convey the same concept.
3
Questions should be simply worded
Avoid lengthy questions
Use complete sentences to ask questions
Present ideas generally to allow for
respondent to give ready-made accurate
answer
Use equal numbers of positive and
negative categories for scalar questions
Avoid bias
At this point we have a handout for
the class to review as groups of 3-4
and decide on question
discrepancies.
Should be like a conversation and not jump
from topic to topic
Group items by content, providing subtitles
for each group
Within each group of items, place items with
same format together
Focuses respondent on content and
question instructions
Eliminate check-all-that-apply question formats
Develop response categories that are mutually
exclusive
Questions should be technically accurate
Avoid double negatives
Avoid double-barreled questions, or questions
that contain more than one question within
them
Soften the impact of objectionable questions
with wording
Start questionnaire with questions with widest
appeal to gain interest:
Should be easy, interesting and be connected
to the surveys purpose
Should not start with demographic information.
In fact, such questions should be put toward
the end of a questionnaire since personal
information could be sensitive.
Objectionable questions should be placed
near the end of questionnaire
Vary question direction to avoid
respondents from answering only with
positives or negatives
Ordering can have effect on response on
opinion questions
4
Adequate spacing and use of white space
Crowded papers seem longer and more
overwhelming
Adequate space should be given for responses,
especially open-ended questions
Spacing should also be considered for questions
with a scalar response to give the impression of
equal intervals
Be consistent with where answer spaces are
placed (i.e. to the left or right of the question)
Consider the length of the questionnaire and time
to complete
Appearance counts!
Paper and print quality (or online equivalent)
Use consistent font style and size
Changes should be used exclusively for important
highlights
Questions should be numbered consistently
Letters can be used to signify different sections (e.g.
A1, A2 B1, B2)
List answer categories vertically instead of horizontally
Spacing should also be considered for questions with
a scalar response to give the impression of equal
intervals
This is an
example of a
poorly
formatted
questionnaire.
Rejoin your
groups and
discuss some
of the
discrepancies.
Writing Directions:
Should be simple, specific
and complete
Emphasize the need for
honest responses
Write new directions for
every format change
Build trust.
Focus, in regard to sources of error, has
been on sample design and size as
opposed to the effect of the design of a
questionnaire.
Good questionnaire design is a cost
effective way to ensure high quantity and
quality of responses.
Cox, J. & Cox, K. B. (2008). Your
opinion, please! How to build the
best questionnaires in the field of
education, Thousand Oaks, CA:
Corwin Press.
Dillman, D.A. (2000). M ail and
internet surveys: The tailored
design method (2nd Ed.). New
York, NY: John Wiley & Sons.
Fink, A. (1995). How to Design
Surveys. Thousand Oaks, CA:
Sage Publications.
Fowler, Jr., F.J. (1988). Survey
research methods (Revised edition).
Newbury Park, CA: Sage
Publications.
Fowler, Jr., F.J. (1995). Improving
survey questions: Design and
evaluation. Thousand Oaks, CA:
Sage Publications.
Checking for Alignment
Instrument Validity
Will my questions be
answered by the
questionnaire?
Instrument Validity
Part of the quality of any
survey study is the
reliability and validity of
the questionnaire
Pretesting
http://www.sysurvey.com/tips/arsha
m.htm#rssss
http://www.cc.gatech.edu/classes/c
s6751_97_winter/Topics/questdesign/
http://www.tardis.ed.ac.uk/~kate/q
mcweb/q7.htm
http://www.tardis.ed.ac.uk/~kate/q
mcweb/q8.htm
http://www.tardis.ed.ac.uk/~kate/q
mcweb/q9.htm#Long%20questions
http://en.wikipedia.org/wiki/Questio
nnaire
5