{[ promptMessage ]}

Bookmark it

{[ promptMessage ]}

chapter_6_public_opinion - STATMEISTER How to Spot a Bogus...

Info iconThis preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Background image of page 2
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full Document Right Arrow Icon
Background image of page 4
Background image of page 5
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: STATMEISTER How to Spot a Bogus Poll Opinion surveys can look convincing and be completely worthless. But asking four simple questions of any poll can separate the good numbers from the trash. weapons in campaign ads. Journa- oliticians use opinion polls as verbal P HIS AND HER NEWS lists use them as props to liven up in- (percent of Americans who follow selected types of news stories fotainment shows. Executives are more like- "yew closely," 1989-95, by sex) 1y to pay attention to polls when the numbers support their decisions. But this isn’t how polls are meant to be used. Opinion polls can be a good way to learn about the views Amer- icans hold on important subjects, but only if you know how to cut through the contradic- IT IS DIFFICULT tions and confusion. . - Conducting surveys 15 dlfficult. It is especially difficult to take a TO TAKE A meaningful survey of public opin- ion, because opinion is a subjective MEANINGFUL thing that can change rapidly from day to day. Poll questions some- military/ natural busi- sports crime interna- legal . . . terrorism dis- ness/ tional SURVEY OF PUBLIC times produce conflictlng or mean- aster financial politics ingless results, even when they are OPINION, BECAUSE carefully written and presented by professional interviewers to scien— OPINION IS A tifically chosen samples. That’s why Source: Pew Research Center for The People & The Press Men are more likely to be fascinated by stories the best pollsters sweat the details about wars and politics.Women are more interested in SU BJ ECTIVE THIN G 01’] the order and wording Of ques- tions, and the way data are coded, anal zed, and tabulated. THAT CAN CHANGE Y Pollsters other than the best can also set up sur- veys that deliberately shade the truth. They do this RAPID LY. tions, or they restrict their questions to people likely to give by acting like trial lawyers: they ask leading ques- the desired response. In fact, pollsters can use dozens of obscure tricks to intentionally push the results of a survey in the desired direction. So the next time a poker-faced person tries to give you the latest news about how Americans feel, ask some pointed questions of your own. Did You Ask the Right People? In 1936, the editors of Literary Digest conducted a Presidential preference poll 10 American Demographics October 1996 disasters and crime. of more than 2 million Americans. The poll predicted that the Republican candidate, Alf Landon, would defeat Franklin Roosevelt. Landon’s loss made the Digest history’s most famous victim of sample bias. The Digest mailed more than 10 million ballots to households listed in telephone books and automobile reg- istration records. This method might create a relatively representative sample today, but in 1936, it substantially biased the sample toward those affluent enough to own cars and phones. The magazine’s disgrace was made com— plete by young poll-takers like George Gallup and Elmo Roper, who used samples of a few thousand people to pre- keeping force.” CBS’s harsher wording may have contributed to its respondents’ harsher judgment of the Clinton decision. Sometimes words are problematic because they are too vague. In April 1996, the Pew Research Center asked which Presidential candidate was best described by the phrase: “shares my values.” By this measure, Clinton beat Dole by 47 percent to 37 percent. But when CBS and the New York Times asked whether each candidate “shares the moral values most Americans try to live by," 70 percent said that Dole did, but only 59 percent said so of Clinton. One way to sharpen the meaning of a poll on “values” is to ask an open-ended question, such as: “What ‘values’ do you share with the candidate?” The long list of responses generat— ed by such a question can be entered and coded to provide the sample’s average defini— tion of the term. This step obviously takes more time and money. It produces a survey that is more precise, but harder to explain. If you were a television reporter and you had 20 seconds to describe the question and give the result, what would you do? These are some of the most common rea- sons why polls that appear to be authoritative are, in fact, total trash. Other pitfalls are described in Polls and Surveys: Understanding What They Tell Us, a layperson’s guide by Norman M. Bradburn and Seymour Sudman, (San Francisco: Iossey—Bass, Inc., 1988). The Pew Research Center’s Poll Watch newsletter does an excellent job of spotting and explain— ing poll gaffes: for more information, call Pew at (202) 293-3126. When you’re presented with a new survey or a used car, it helps to ask a few key ques- tions before you buy. But for all their flaws, surveys are essential to the work of politi— cians, journalists, and businesspeople. “Yes, there are too many bad polls,” writes Richard Morin, director of polling for the Washington Post. There are also “too many polls that report what people think but not why they think it.” At the same time, with lots of competing polls, it’s easier to see errors and rapid shifts in public opinion. “Polling is a robust methodol- ogy,” writes Morin.“A lot of little things can go wrong and the final result can still be right, or at least close enough.” —Brad Edmondson “Health Care Data You Can Trust” A bold, new magazine uniting marketing and technology. To subscribe call: 800-838-1133 CA“- TODAY To advertise call: 800-832-1486 1'800-939-7768 http://www.marketlngtools.com CHECK US ON THE WEB http://www.20010n|ine.com/hsdf MAHKlllNfilflll|§ A Publitotion of Dovlones 1. Company, Int. HSDF, Inc., Louisville, KY e-mail: [email protected] Pssku anything you want about the Wyay marke Really, anythin "closet. For the first time Gay consumers are coming out oft ever, Simmons is conducting a co , . "ehensive, in—depth study of the gay and lesbian market in th ‘ l Mulryan/Nash, an ad agency that Produced in conjunction w: specializes in reaching ga consumers, the survey will include all kinds of valuable infor ‘ tion about this influential market niche: Which products and se‘ ices gay men and lesbians use. How they make purchasing deci Ons. Even how they feel about advertising that’s directed specifica y toward them. This landmark study ll establish a comprehensive database with information about ga consumers—men and women who boast some of the most desirable demographics in the U5. Subscribe today Just , ony Incalcaféra at 1-800-291-8890, ext. 8927. Mulryaiyflash American Demographics Octoberi996 15 It’s TRUE... All the data/software you need for demographics and analysis for under $100. The Right Site” Demographics made easy! The secret of our high quality and low price. At EASI (Easy Analytic Software, Inc.) we have decades of experience working with the most complex systems. Now we’re taking advantage of the mass market for CD ROMs, by incorporating the most important features and benefits of those systems into The Right Site. That means you can now get a powerful demographic tool without a major investment. Get 505 variables for any area. The Right SiteTM is a CD-ROM for WindowsTM that gives you the most important demographic data — including population, income, age, employment, family characteris- tics V plus Quality of Life information. 7 ways to use the Right Site . Demographic analysis. . Site selection. . Site evaluation and analysis . Demographic reference for consultants. . Cluster creation. . Market evaluation. 7. Sales territory creation. An important bonus. We’ll supply data you can use for personal and business decisions—so you can find your ideal neighborhood! Only The Right Site lets you fac- tor in additional quality of life facts!like crime rates, weather, culture, recreational activities, and relative quality of schools. Money-back guarantee. Order today! For a limited time, our introductory price isjust $99.95, plus $6.95 for postage and han- dling". There’s no risk since we offer a 30-day money back guarantee. Call 1-800-HOW- EASI (1-800-469-3274) for quick shipment and answers to any questions. Easy Analytic Software, Inc. (EASI) The Right Site, P.O. Box 363, 891 East Oak Rd., Vineland, NJ 08360 Fax: 1-609-696-2130 We’re on the Web at www.easidemographics.c0m. 'Add sales tax in NY State 1 2 American Demographics October 1996 STATMEISTER dict a Roosevelt win. Gallup and Roper care- fully chose their samples to reflect a demo- graphic cross-section of Americans, just as they do today. The most amazing thing about this story is that some journalists and businesses in the 19905 still make the mistakes the Literary Digest made 60 years ago. Any journalist with half a pencil knows know that only a scientifically chosen survey sample will represent the country’s opin— ions. But the temptation to take a biased poll is great if you have a tight deadline and a small budget, as many news organizations do. SAMPLING ERROR WILL ALWAYS EXIST UNLESS YOU SURVEY EVERY MEMBER OF A POPULA- TION. IF YOU DO THAT, YOU HAVE CONDUCTED A CENSUS. The 2 million who responded to the Liter- ary Digest poll in 1936 were even more likely than the total sample base to be wealthy and Republican, typifying a common survey problem—nonresponse bias. Even when you start out with a representative sample, you could end up with a biased one. This is a risk all pollsters take, but some particular methods lend themselves to greater error. For example, readers of women’s magazines are frequently asked to fill out surveys on weighty subjects like crime and sexual behavior. Not only do such polls ignore the opinions of nonreaders, they are biased toward readers who take the trouble to fill out and return the question- naire, usually at their own expense. Television news and entertainment shows get into the act by posting toll—free or even toll numbers that viewers can call to “vote” on an issue. These samples are not only biased, they are prone to “ballot—stuffing” by enthusiasts. In other words, viewers who call 12 times get 12 votes. Poll results based on “convenience” samples can be wildly misleading, even if the sample sizes are huge. A call-in poll conducted by a television network in 1983 asked: “Should the United Nations continue to be based in the United States?” About 185,000 calls were received. Two-thirds said that the UN. should move. At the same time, the network conduct- ed a random-sample poll of 1,000 people, and only 28 percent said the UN. should move. Between 1989 and 1995, the Pew Research Center for The People & The Press in Wash- ington, D.C., monitored the public’s interest in 480 major news stories. Almost half of Americans paid little or no attention to these stories, and only one in four followed the aver- age story very closely. Stories about wars and disasters were followed most closely, while those about celebrity scandals and politics fin— ished last. When Pat Buchannan announced that he was running for President in 1991, for example, only 7 percent of Americans paid close attention to the story. Conflicts make news. When journalists are trying to liven up a boring political story, they need angry, well-informed citizens like a fish needs water. This is one reason why older men may be quoted more often than other groups. Those aged 50 and older are more likely than younger adults to follow news stories “very closely,” according to the Center, and men are more likely than women to follow stories about war, business, sports, and politics. In the last decade, angry white men have dominated media programs designed to give ordinary people a chance to speak out in pub- lic. Two—thirds of regular listeners to political talk radio programs are men, according to a 1996 poll taken by Roper Starch Worldwide for the Media Studies Center in New York City. Republicans outnumber Democrats three to one in the talk-radio audience, and 89 percent of listeners are white, compared with the national average for voters of 83 percent. Three in five regular listeners to political talk radio perceive a liberal bias in the mainstream media, compared with one in five nonlisteners. A multitude of reputable surveys have shown that most Americans generally believe that the country is headed in the wrong direc— tion and that political leaders can’t be trusted. But those who respond to convenience polls and call in to talk shows probably don’t speak for most Americans. What’s the margin? A statistician and two friends are hunting for deer. They spot a buck. Friend number one takes a shot and hits FRAMING THE QUESTION (question wording and results of three polls taken on November 27, 1995) CNN/USA Today/Gallup: Now that a peace agreement has been reached by all the groups currently fighting in Bosnia, the Clinton Administration plans to contribute U.S. troops to an interna— tional peace-keeping force. Do you favor or oppose that? favor ............................ 46% oppose ......................... 40 don’t know ...................... 14 CBS News: Do you favor or oppose sending up to 20,000 U.S.troops to Bosnia, as part of a NATO peace-keeping force, to enforce this peace agreement between Bosnia, Serbia, and Croatia? favor ............................ 33% oppose ......................... 58 don’t know ....................... 9 ABC News: Clinton said now that a Bosnian peace treaty has been signed, he’s sending 20,000 U.S. troops there as part of an international peace-keeping force. Do you support or oppose sending 20,000 U.S. troops to Bosnia as part of an international peace-keeping force? favor ............................ 39% oppose ......................... 57 don't know ....................... 3 Source: Poll Watrh, Pew Research Center for The People & The Press Words like "enforce”and numbers like "20,000” can direct respondents’ thoughts. a tree five feet to the left of the animal. Friend number two fires and hits a tree five feet to the right. The statistician exclaims,“We got him!” No matter how carefully a survey sample is chosen, there will still be some margin of error. If you selected ten different sets of 1,000 people using the same rules and asked each group the same question, the results would not be identical. The difference between the results is sampling error. Statisticians know that the error is equally likely to be above or below the true mark, and that larger samples have smaller margins of error if they are prop- erly drawn. They are also able to estimate the margin of error, or the amount by which the result could be above or below the truth. Sampling error will always exist unless you survey every member of a population. If you do that, you have conducted a census. Sampling error is one reason why two pro— fessionally conducted polls can show different results and both be correct. For example, the CNN/ USA Today/Gallup Poll of January 5-7, 1996, showed that the proportion of Americans who approved of President Clinton’s perfor— mance had dropped to 42 percent, from 51 per— cent on December 15—18, 1995. Polls conducted that same week by Washington Post/ABC and New York Times/CBS showed that his approval rating was 53 percent and 50 percent, respec— tively. This made for a few wild days at the White House, until the next Gallup survey showed a sudden rebound. Reputable surveys report a margin of error—usually of 3 or 4 percentage points— at a particular confidence level—typically 95 percent. This means that 5 percent of the time, or 1 time in 20, the poll’s results will not be reliable. The other 95 percent of the time, it is within 3 or 4 percentage points of the “truth.” This sort of inevitable statistical problem explains the blip in the January Gallup poll. Which came first? The order in which questions are asked can have a big effect on the results. In late May 1996, the CNN/USA Today/ Gallup poll reported that 55 percent of Americans believe taxes can be cut and the federal deficit reduced at the same time, com— pared with 39 percent who do not believe this. The same week, New York Times/CBS report- ed a dead heat of 46 percent who believe and 46 percent who do not. This variance was way beyond the margin of error. The questions were almost identical in their wording. But the order of questions in the Gallup poll may have biased the results, according to Poll Watch, a publication of the Pew Research Center. In the CBS poll, questions before the tax COMPLETE COUNTY ECONOMIC AND DEMOGRAPHIC DATA I Population by Age, Race and Sex I Household Data by Income I Per Capita Income I Personal Income by Source I Retail Sales I Employment by Industry I For Each Year 1970 to 2020 FORECASTS FOR ALL I Counties I Metro Areas (MSA/PMSAs) I Regions and States USED IN I Regional Market Analysis I Long Range Planning I Site Location Studies I Research and Reference I Presentation of Data and Mapping AVAILABLE IN I Print I Floppy Disk I Magnetic Tape I CD—ROM “ Woods 8c Poole has produced county projections since 1983. Wooos&Poou~: E C O N O M I C S Q WASHINGTON D. c. a: Call or write for more information Woods 8c Poole Economics, Inc. 1794 Columbia Road N.W Suite 4 Washington, DC 20009-2805 1-800-786-1915 American Demographics October1996 13 Jamerican Since ; E Associates -1964- . ThgoglfstorQPE Off-snore Data Capture ' AttEgi’Eég 11119111111111, -SGML / HTML -Full Text -Coding 'Database Superior Quality, Key & Key Verify Servicing Fortune 100 Companies Exceptional Turnaround and Flexible Capabilities Fax Broadcast & Fax on Demand 3 E V ' ‘ Services 800‘828‘1133 us. ornce (540)898-0238 /(540)898-3762 Fax g - For a free copy, press 21 pr “131;,1erricl 13111311111 1'11: :le -:1- e13 Crvlwss f1:1r11::ff ' g dish 11'111l1r1 1$',1ll (lies ilk/Git; QGQEIS IOIOOIflQ a More and analyze geographic patterns: . 130::Lc1iéc111‘1s lroim" counts carve 11:11:11 shop-1:1: mg cerien 5 01:011. 0111 00.0%.. I Large and small formats I Digital products I Rush orders occomodoled IZ‘r::-~::1iir.::-l Elfgl 1’13 cfiso l r 1‘11"er" unang some are data produc ts sé‘e Rulerslrcn our 121 Wei rest—‘1”:hs 11’1—1 Put your business on the map. .. coll Spo‘riol insights Tel: 800-347-5291 Fax: 703-827-7037 1 4 American Demographics October 1996 STATMEISTER cut/deficit question were not related to the subject. But Gallup first asked respondents if they favor a tax cut. Then it asked those who did if they would still favor it even if it meant no reduction in the deficit. Then it asked all respondents if they believed both could be done at the same time. By that point,“some of Gallup’s interviewees may have felt invested in the idea of a tax cut,” says Poll Watch. Most people want to appear consistent to others and to be consistent in their own minds. When a pollster asks a series of related ques— tions, this desire can lead people to take posi- tions they might not have taken if they were asked only one question. Neither way pro— duces an obviously “correct” response, but the results are different. One way to handle this problem is to rotate the order of questions. Then the degree of differences due to question order can be described and interpreted. But not everyone pays heed to such fine distinctions. What was the question? “Do you want union officials, in effect, to decide how many municipal employees you, the taxpayer, must support?” Well, do you? This question, taken from an actual survey, is obviously biased. The results might make good propaganda for an anti-union group, but they are totally bogus as a poll. 80 before you pass a survey finding on to others, or even believe it yourself, be sure to look at the actual question. Question wording is extremely subtle. In the hours after President Clinton’s November 27, 1995, speech announcing that 20,000 U.S. troops would be sent to Bosnia as part of a NATO peace—keeping mission, three major news organizations took reaction polls. CNN/ USA Today/Gallup found that 46 per- cent of Americans favored Clinton’s plan, while 40 percent were opposed. CBS found that only 33 percent were in favor, and 58 per- ...
View Full Document

{[ snackBarMessage ]}