Agenda
E-MAIL 4/14/2009
Presentation AND INTERNET
SURVEYS
April 16, 2009
Brief Introduction
Construction of E-mail and Internet Surveys
Response Rate and Other Problematic Issues
Innovations in E-mail and Internet Surveys
Privacy Issues in E-mail and Internet Surveys
Niko Hales, Kenny Nguyen, Christina Oxendine, and
Jeff Rice
Introduction
E-mail and Internet Surveys are the most profound
innovation in survey methodology since the
introduction of telephone surveys in the 1970s
Definitions
E-mail Survey: Essentially: little more than a simple text
message, and its construction may require computer
skills no greater than those needed for composing and
sending a message to a friend (Dillman, 2000).
Internet or Web Surveys: A survey answered with a
web browser.
Discussion Question
What are some major advantages and
disadvantages of E-mail and Internet Surveys?
4/14/2009
Unique Aspect of Internet Surveys
Internet surveys are self-administered
Internet surveys are computerized
Internet surveys are (mostly) interactive
Internet surveys are distributed there is a lack of
uniformity in the hardware/software used by
respondents to take the survey
Internet surveys are rich visual tools
Advantages of Web/Email Surveys
Cost savings stemming from eliminating
printing costs, mailing costs, and the
need for someone to take the time to
record the results (returned web surveys
are likely already in the desired format)
Response speed
Reaching a different demographic
Respondents can fill out the survey at
their leisure
Interactive without drawbacks of having
an interviewer
Eliminates time zone constraints
Disadvantages of Web/Email Surveys
Computer and E-mail Access
Lack of uniformity in the
hardware/software used to
complete the survey can affect the
feel or look of the survey
Response rates harder to gauge,
but generally lower
Sample coverage and sample bias
Data editing/entry is still required
(for email surveys only)
Hard to ensure intended
respondent is the one completing
the survey (for email surveys only)
E-mail vs. Web Surveys
E-mail
Web
Client-side executed on a
respondents machine
Technical limitations in regards
to interactive features of the
survey reduces the number of
e-mail surveys in use
Transmission of information
back and forth to the server
isnt happening in real-time
which makes
prompting/seeking additional
information difficult
Server Side executed on the
survey organizations web
server
Typically involves the
respondent completing the
survey while connected through
a browser with the answers
being transmitted to the server
as each answer is submitted or
next button is pressed
Internet connection is on
Most prevalent form of internet
survey
4/14/2009
Web Survey Design Types
Separate browser windows aka pop-up surveys
vs. the main browser also referred to as paging
vs. scrolling surveys
Single question (e.g. question of the day polls) or
many questions
Questions can be included in a single HTML form or
distributed across many forms
Scrolling Surveys
Advantages
Similar to a paper survey
Respondents can readily
determine survey length and
see upcoming ?s
Can answer ?s in the order
they prefer
Easy to program
Quick to download
Less interaction with the server
so less technical difficulties
Scrolling vs. Page Designs
Scrolling
Scroll Down to answer
questions below
Entire survey is on one
page
Page
Multiple pages to the
survey
Each question presented
on a separate HTML form
Paging Surveys
Disadvantages
Typically, survey must be
completed in one sitting
All responses may be lost if
respondent forgets to hit submit
at the end of survey
Can see all ?s may influence
order of questions answered
and answers themselves
Having to scroll may increase
missed items, especially on
longer surveys
Use When: Survey is short, everyone needs to answer all questions,
respondents missing ?s isnt a concern, providing context for later ?s is
desirable, question completion order is of little concern, offering the web
survey as an alternative to paper in a mixed-mode design
Advantages
Minimal or no scrolling
necessary
Respondents can complete the
survey in multiple sessions
Skips and routing are
automated
Immediate feedback can be
provided for missing data, etc.
Feedback can be used to
provide help
More interactive
Disadvantages
Greater interaction with the server
is required
Respondents may have trouble
knowing where they are in the
survey (context)
Respondents have less control over
?/answer order
Harder to code
Raises concerns about
confidentiality once answers are
submitted, they cant be undone
even if the survey is abandoned
Use When: Survey is lengthy (allows respondent to complete survey in
more than one session), survey contains skips, edits, etc., survey contains
many graphs, its desired for respondents to answer questions in sequence,
survey contains key screening items
4/14/2009
Discussion Question
Inconclusive Response Results
Given what weve learned about questionnaire
construction and mail surveys, discuss some potential
pratfalls in questionnaire design for Internet
surveys. How are these similar/different than for
mail surveys?
How do you think response rates for E-mail/Internet
Surveys compare to Mail Surveys?
Cho and LaRose (1999)
Inconclusive Response Results
Cobanolgu et al. (2001): Mail 26.27%, Fax
17%, Web 44.21%.
Leece et al. (2004), Survey of Surgeons: Web
45%, Mail 58%.
Cook, Heath, and Thompson (2000)
Mean response rate over 49 studies and 68 surveys
was 39.6%.
Significant predictors of response rate: Number of
contacts (ceiling effect), personalized letter, salience,
incentives (negative correlation), and education.
E-mail Vs. Web Based
Type
Response Rate
Browser Issues
Smart Resend
HTML Based Email
Typically Higher.
Survey is
viewable in email
inbox. Higher
responses when
displayed
opposed to
having to click
link for survey.
Some e-mail
programs cannot
respond to inline
email surveys.
Alternate link is
included for
those.
Automatically
resend email to
those that have
not responded.
Those that have
are removed
from list.
Web Based
Much lower than
inline surveys.
Screen resolution,
plug-ins
No resend. Not
sent out first
time.
4/14/2009
Problems with Web Surveys
Sample Quality will a representative of a sample have the chance to
respond? Internet users are usually younger and have higher education level.
Web surveys provide high quality samples if the intended audience is those
that are frequent internet users. If a target audience is wanted that is older or
non-white, then it may not be a well represented sample.
Some technical problems are also prevalent while filling out an online survey
Freezes and crashing servers crashing and web browsers freezing up make
finishing surveys difficult. For this reason, they should be relatively short or
spread across several pages where the information is submitted at the end of
each page. Slightly higher cost would allow for the continuation of a survey
from any point.
Error messages If not properly programmed, some responses can cause error
messages making the responses skewed and it hard to finish. Messages that
notify you when a question is skipped is also important to be user friendly.
Potentially take you to the question that was skipped when clicked on.
Web Survey Question Design
Problems cont.
Double Entry without the use of user ids and passwords, the
responses can be skewed by double entry. A specific list of people
should be developed for the survey. A general survey online has
skewed responses with the number of times each person can fill out
the survey online.
Responses that are intentionally entered falsely will also skew
responses.
Sampling frame since online surveys to not normally have a
defined sampling frame, it is impossible to calculate a response rate
Drop out rate most of the people that begin surveys do not finish
them. Only a very small percentage finish the survey.
Cost web surveys are cheap to create but can have a high cost if
companies are having to provide monetary or coupons to entice
users to complete a survey.
Skewed Scores
Categorical questions use questions that allow respondents to
place themselves into exactly one category
Double negative questions type of question that asks for a level
of agreement with a statement. Example: Teachers should not be
required to supervise students during recess. If respondent
disagrees, i.e. does not think teachers should not supervise
students in other words, they should supervise students.
Web survey users typically have private access to computers, hold
more responsibility, and better paid.
Double-barreled questions type of question about more than
one issue in a single question. May result in the inaccuracies in
the responses being measured for the question.
Why? Difficulty accessing survey, the inability to move forward and
backward through questions, interruptions in completing survey make
it tough to continue survey (would have to restart), confidentiality
concerns.
Leading questions are questions designed to lead your
response in a certain direction.
If a company offers surveys in an online and paper format, high
level employees generally choose the online version with the low
level employees choosing the paper format. Providing results.
Opting out response rates for web surveys can be as much as
80% lower than that of printed surveys.
Sugarcoating poorly designed web surveys produce favorable
responses. Employees reluctant to complain because of fear of
identity being released.
4/14/2009
E-mail response rate influences
Survey Length one of the main reasons for nonresponse in business surveys
Respondent Pre-notification leads to increased
response rates. Reminder emails also on average
increased response by 25%.
Design Issues
Research Affiliation and Compensation research
cost of survey can get expensive to companies.
Internet Users Americans 50% Europeans 23%
Asians 37%
Innovations in Internet Surveys
Currently used for:
Evaluations (classeval.ncsu.edu)
Market and scientific research
Online job applications
Polls
Used to take tests and quizzes
Creates tables, charts and graphs instantly
Password features
Other Issues
Speed benefits mail surveys take on average 11.8
days to respond with email taking 7.6 days
Cost of the survey itself only costs 5-20% that of a
paper survey.
E-mail surveys can also accommodate for some non
response by knowing the number of undeliverable
email addresses. Cannot count them as non-response
if they were not delivered.
Survey Creation Software
Survey Creation Software is used for creating online
surveys and web polls.
Some Websites that use Survey Creation Software:
SurveyMonkey
SurveyGizmo
SurveySite
Zoomerang
Perseus
4/14/2009
Survey Creation Software
Consultation throughout survey research process,
including:
method design
questionnaire creation
data collection and analysis
interpretation of result
Survey Creation Software
Access to tailored email lists and multisource
recruiting for sampling
Allows researchers to target specific demographic
groups within a population of interest
Tracking of respondents email
Email response notification
export survey responses to statistical software
packages such as SAS and SPSS
Survey Creation Software
Help researchers collect data by advertising the
survey on certain websites.
Pop-up advertising to aid in recruiting
participants.
Some companies offer other types of features to
aid with the survey research process.
For example, EZ Survey offers a free sample size
calculator
unsubscribe respondents from an email list after they
have completed a survey, which may help to reduce
multiple responses from the same participant.
Mobile Surveys
Merge computer technology with traditional survey
methods.
Telephone surveys with touch-tone responses.
Online focus groups where individuals can view the
same audio, video and text from different locations.
Researchers can interact with individuals using web
chats or via teleconference.
4/14/2009
Mobile Surveys
Uses wireless handheld devices such as Palm pilots,
Smartphones, BlackBerry,etc.
Data is sent to a server (similar to other online survey
forms) where the information is posted to a database
file.
Advantages: Using a wireless device researchers can
bring a survey to inaccessible populations:
Demo
http://classevals.ncsu.edu
those without personal computers.
Individuals in healthcare settings
rural environments
socioeconomic groups that do not have access to computers
or the Internet.
Issues in Privacy
1.
2.
Discussion Questions
What are some common threats to online
privacy?
How might these threats affect E-mail
and Internet surveys?
The Appeal of Online Surveys
Can address surveys to specific individuals
Establishes asynchronous contacts with respondents
on the move
Can screen low incidence populations
Can theoretically improve the quality and quantity
of response
Can extend contacts across national boundaries
4/14/2009
But Privacy Poses a Serious Challenge
Three Types of Privacy Issues
Security Issues regarding web site encryption and data
storage and security
Online Community Privacy Issues
Personal Privacy Issues
Other Security Issues
Security of data
Anonymity
Personal Anonymity
Multiple Respondents
Security Issues
Common Issues
Cookies
SPAM
7/10 respondents to an online survey about privacy
said that they worry about privacy more on the
Internet than through mail and telephone surveys
(Cho & LaRose, 1999)
27% provide false information (Cho & LaRose,
1999)
< 10% of Web Sites have stated polices protecting
against private information (Federal Trade
Commission, 1998)
Online Community Privacy Issues
Risk of privacy through trolling for email
addresses
E-mail addresses usually collected from online
communities
Problem is exacerbated by closed
communities such as the Facebook and the
MySpace
Banners placed in search engines, e-mail lists,
and other Internet advertisements
4/14/2009
Personal Privacy Issues
Three Types of Personal Privacy Issues
Physical Privacy
Informational Privacy
Psychological Privacy
Informational Privacy
Control over the conditions, release, use, and
retention of personal data
Affects all medium and, ultimately, trust between
the researcher and respondent
E-mail can be copied and forwarded with ease
False Identities: Problem for both researchers and
respondents
Private subscriber lists
Physical Privacy
Intrusion of an individuals space
Because e-mail is personally addressed, it is
considered more invasive than generic postal
questionnaires
Home computers can be considered private
sanctuaries
Internet fees
E-mail accessible at work
Psychological Privacy
The degree of control over personal information:
i.e. informed consent
Personal choice and voluntary prudence
4/14/2009
Addressing Privacy
Anonymous form of response if trolling is used
Provide Users with Personal PIN
Use a personification e-mail to obtain consent
Use E-Incentives
Run Data Collection through Web Pages (i.e.
Zoomerang)
Gives respondents multiple response options (i.e.
mail, telephone, and web)
Conclusions
The Response Rate Issue
Research questions should drive methodology
Do your research before implementation
Follow best Practices for the medium
Protect User Privacy Using Best Practices
Uniqueness of Web and E-mail Surveys Can Be
Positive or Negative
Be Aware of New Innovations
Other Suggestions
No Cookies
Provide Disclosures
Privacy Certifications
Encryption
Community Consent
References
Batinic, B., Reips, U., & Bosnjak, M. (2002). Online Social Surveys. Seattle, WA:
Hogrefe & Huber Publishers.
Cho, H., & LaRose, R. (1999). Privacy issues in internet surveys. Social Science
Computer Review, 17(1), 421-434.
Cobanoglu, C., Warde, B., & Moreo, P. J. (2001). A comparison of mail, fax, and
web-based survey methods. International Journal of Market Research, 43(4), 441452.
Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in
web- or internet-based surveys. Educational and Psychological Measurement, 60(1),
821-836.
Couper, M. P. (2008). Designing Effective Web Surveys. New York, NY: Cambridge
University Press.
Creative Research Systems. (2009). Web/Internet Survey Software. Retrieved from
http://www.surveysystem.com/web-surveys.htm#samples
Dillman, D. (2000). Mail and internet surveys: The tailored design method. New York:
John Wiley & Sons, Inc.
4/14/2009
References
Federal Trade Commission (1998, June). Privacy on-line: A report to Congress.
[Online]. Available: http://www.ftc.gov/reports/privacy3/toc.htm
Kaplowitz, M. D., Hadlock, T. D., & Levine, R. (2004). A comparison of web and mail
survey response rates. Public Opinion Quarterly, 68(1), 94-101.
Leece, P., Bhandari, M., Sprague, S., Swiontkowski, M. F., Schemitsch, E. H., Tornetta,
P., et al. (2004). Internet versus mailed questionnaires: A randomized comparison
Journal of Medical Internet Research, 6(3). Retrieved from http://www.jmir.org
Top Ten Reviews. (2009). Survey Software Review 2009-Top Ten Reviews.
Retrieved from http://survey-software-review.toptenreviews.com/
Wright, K. B. (2005). Researching Internet-based populations: Advantages and
disadvantages of online survey research, online questionnaire authoring software
packages, and web survey services. Journal of Computer-Mediated Communication,
10(3), article 11. http://jcmc.indiana.edu/vol10/issue3/wright.html
Yun, G. W., & Trumbo, C. W. (2006). Comparative response to a survey executed
by post, e-mail, & web form. Journal of Computer-Mediated Communication, 6(1).
Retrieved from http://jcmc.indiana.edu/