False Economies Article

False Economies Article - False Economies in Survey...

Info iconThis preview shows pages 1–5. Sign up to view the full content.

View Full Document Right Arrow Icon
Background image of page 1

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 2
Background image of page 3

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
Background image of page 4
Background image of page 5
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: False Economies in Survey Research Randall G. Chapman This article details seven of the “sins "committed by both marketing research clients and suppliers in survey implementation. Overcoming these false onomies, as suggested b y this author, may .jttire higher research standards and, therefore, higher costs. There are scores of opportunities for false economies in designing and executing surveys. Many false economies arise when a client and a researglier focus their primary efforts on mini— mizing out—of—pocket costs when designing and conducting surveys, while ignoring opportunity cost and data quality issues. These false economies can be subtle in form. A naive marketing researcher or client may not even know these economies could be operating in a specific marketing or survey research undertaking. An emphasis on minimizing direct, up front, easily observable, out-of—pocket survey research costs is not without redeeming merit. Obviously, it is rational to choose the lowest cost surveying option, other things being equal. To an outside marketing research supplier, cost minimization in a proposal may improve the chance ofwinning the survey contract. Within the client organization, a low cost survey research proposal may make research of any kind more acceptable. However, in the long run, neither may be well Served by focusing primarily on observable costs, while neglecting more difficult to quantify unobservable costs and data quality considerations. —_—__—__.______ Most marketing researchers have encoun- tered the “I want to do a survey” form of problem definition. Thoughtful research professionals do not accept this at face value. -—-—-———.._._..—________—' This article describes seven false economies in survey research efforts. To the thoughtful reader and experienced marketing researcher, other false economies will undoubtedly come to mind. These particular false economies may not be “The (Only) Seven Deadly Sins ofSurvey Research.” However, they should certainly be among the practices avoided by thoughtful marketing research profes- ———-——-—-—-——_.__.____—_ Randall G. Chapman (Ph.D., Carnegie-Mellon University) is an associate professor of marketing at Boston University. His research ' ‘rests include: the design and execution of efficient and effective :eting research studies: the development ofimproved marketing .._vsis techniques: and the analysis ofjudgmcnt. perception and preference formation. and choice behavior. % l 6 sionals. To Obtain better value for their marketing research dollar, decision makers and marketing research clients must be watchful about these “gins” so that they, too, can be alert to avoid these practices —- and to avoid marketing researchers who don't avoid these practices! False Econorny #1: Inadequate Effort Devoted to Problem Definition All client marketing researchers have encoun« tered the “My sales are down, I need some marketing research” situation. Unfortunately, declining sales are only a symptom of some larger problem or problems. Decreased sales are not a problem in and of themselves, except to the manager whose sales are down! More details are needed before a marketing research effort can be designed. A wide range of underlying problems might be contributing factors to a sales decline: mpg— naia-tc, Huffman: orunco mpetitive marketing-mu; elements; changing: competitive marketing pro— grams; changing customer preferences or sensi~ tivities to marketing program elements; changing channel membgrs’ behavior, preferences or sensi— tivities to marketing program elements; or changes in other relevant exogenous market factors (for example, economic conditions). This broad array of possible causes for decreased sales implies that substantial efforts are required to isolate a few probable causes, so meaningful marketing research can commence. Poor communications are, ofcourse, a tw0-way street. Two participants are involved — a research supplier and a client -— and both should be held accountable for poor problem definition. However, the case can be made that the professional market— ing researcher must bear the greater share of the burden in ensuring that the communications links Operate effectively and openly. The research sup- plier knows more about the capabilities and limitations of marketing research. Presumably the supplier also has more experience with the problem definition process, especially with its subtleties. Finally, the client might not truly know what the problem is. Rather, there may be an uneasiness as to how to proceed and a call for a survey may seem reasonable to the client. Multiple conversations with a client are normally required to ensure that the problem at hand is well English defined to permit efficient, effective mar- keting research design and implementation. Dis— cussions with knowledgeable “experts” (within and outside the client organization), in addition to the client, are normally a prerequisite to adequate problem definition. Review of any available secondary data may help to shed light on the nature ofthe problem. E Research should not be conducted for ego, power, security, fun or timefilling reasons. m Most marketing researchers have encountered the “I want to do a survey” form of problem definition. Thoughtful research professionals do not accept this at face value. After all, some careful thinking, exploratory research, a focus group interview or secondary data might adequately resolve the issue, instead of an expensive surveying effort. Rather, professional marketing researchers will push a client as far as possible to define the need for the survey, to specify the precise uses in which the survey results will be applied and to indicate how certain survey results will differen- tially influence forthcoming marketing decisions. Thoughtful marketing research professionals are careful not to fall it; Q glgeétmof conducting a survey (or any ot er substantial marketing research study) to provide evideitgpg decision. mantel??? professional marketing researchers also learn to recognize those situations where marketing research is not worthwhile, where a business decision can and should be made based only on the current available facts and where an additional investment in marketing research would not materially reduce the uncertainty associated with the marketing decisions to be made. Clients must be watchful of marketing re— searchers who try to conduct research for its own sake, because others are doing similar research, or tojustify their own continuing existence within an organization. Surveying in particular, and market— ing research in general, is done to shed light on specific marketing issues and to reduce the uncer— tainties associated with marketing decision making. Research should not be conducted for ego, power, security, fun or time-filling reasons. False Economy #2: Ignoring Exploratory Research EXploratory research takes time and costs money. The careless researcher may ignore this l7 1Q stage ofthe marketing research process to get right i with surveying. The danger of ignoring the exploratory research phase is simple: the wrong problem may be investigated or inappropriate procedures may be used, or both. ‘- - - «and to Wile ag. = is .2 Ignoring the exploratory research stage involves taking risks: the risk in assuming that all of the relevant issues associated with the given problem definition are fully known, and the risk that the problem definition statement may actually be imprecise or incorrect. Thoughtful professional, marketing researchers realize the value of explor- atory research efforts, prior to going into the field with expensive and time—consuming survey research missives. One possible mate-of exploratory research is a rWth-tpmhm statement. Such an outcome should be viewed positively by all concerned, not negatively. It is much better to recast the problem statement before going into the ’eld with a surveying effort, rather than after, or at, the data analysis stage. This leads to the “Well, it’s interesting, but . . kind of response when results are presented to the client at the completion of a large—scale marketing research study. False Economy #3: Not Anticipating the Eventual Data Analysis ' engagement-tie ._ If not, how can one be sure that Just the right data are collected to answer the research questions which motivated the sur— veying effort in the first place? In the absence of such anticipation, the careless analyst must collect all manner of relevant and irrelevant data, to ensure that enough data are available at the analysis stage so that “interesting” relationships will be discovered. Of course, with enough data, and with enough univariate and multivariate statis— tical analysis, something is bound to pop out. Harem—:- The construction of“jt§“: ; ‘is a crucial tool to ensure that the iii ”' i"eiihg“re'sEarch analyst and the client have fully thought through how the data will ultimately be analyzed. True, such anti- :ipation requires a lot ofcareful thought up front. However, afterthe field work is done, it is too late to discover the omission of several key questions that would have facilitated the data analysis. Serendipity has its place in survey data analysis, to be sure. However, the thoughtful marketing researcher relies as little as possible on luck, and as much as possible on having thought through the problem and the necessary analysis ahead oftime. in some large—scale surveying efforts, conducting a modest pilot study may be appropriate to provide some data for pilot analysis efforts, as well as to test out survey research administration proce~ dures. False Economy #4: Failure to Pretest Survey Questionnaires Some researchers lament that there is rarely time to pretest survey questionnaires with real respondents. Most survey research these days seem to come with the timing parameterization “I want it yesterday.” The only answer to the “There’s no time” statement is the firesp make thetime.”Pretestingis‘{ u - inanimate, especially if the study involves an unfamiliar target population, administration method, sampling frame, type of questioning or questionnaire style, format or length. To look at most surveys, one would have to assume that potential respondents have nothing to do but sit and wait with bated breath for the next survey participation occasion. Also, it is importantat pretesting exercise is . ' nu timates of potential survey response rates. Such respOnse rate estimates may have important implications for sample size assessments, inter— viewer scheduling and survey cost estimates. False Economy #5: Providing Respondents With Insufficient Participation Incentive To look at most surveys, one would have to assume that potential respondents have nothing to, do but sit and wait with bated breath for the next survey participation occasion. There has been- things . 7“‘_W_e must a administration procedures can also provide es— much empirical research to document that many ’- ‘7 i L as " . (More personal — 'm expensfv "‘“ El'epl‘i'one and personal inter- viewing procedures also seem to be major deter- minants of survey response rates.) And yet, the typical surveying effort seems not to employ these valuable tools on a regular basis. True, they cost some money and take some time . I H a; _ h" ercher w; l Nth p dent (with pre— notification and follow—up) and make it worth the reSpondent’s while to participate in surveying efforts (with incentives or, if necessary, with compensation.) False Economy #6: Using Substitutions Instead of Callbacks What should be done with a reSpondent who does not answer on the first contact occasion? Whether the administration procedure is mail, telephone or personal contact, it Seems natural to just substitute the next available respondent and get on with the study. However, many empirical studies have documented how detrimental this practice can be. ., r ., .... ,. __ .Highquahtysu’Ey research requires the use of extensive callback procedures. Professional marketing researchers build such callback features into their survey research designs, and into their survey research budget proposals. False Economy #7: Assuming That Non-response Bias Does Not Exist In the write-ups of survey research results, the response rate is usually reported along with a phrase something like, “This is a typical response rate for surveys of this kind.” Unfortunately, this does not really provide any proofthat peOple with more time on their hands or greater interest in study participation were the ones who tended to participate in this particular survey. If these respondents are fundamentally different from the non-respondents, then the survey results may be substantially or fatally flawed. Remember, we estimate the overall mean ofthe target population on a survey question by the corresponding reSpondents‘ mean. Only with lOO percent response can we be sure that no n0n~ responSe bias exists. (Of course, even with 100 percent response, other forms of non-sampling error may exist.) Why not try to deal directly and forcefully with the non-response bias problem? Again, monetary considerations come to the fore. There is obvious merit in being concerned with controlling survey research expenses, since non- respondent bias analysis efforts are costly. How- ever, what are the costs assooiated with having unknown levels of non—response bias in survey research results? The difficult-to-reach people may well be very different from the easy-to-reach people. Callbacks or multiple mailings in mail surveys, may help alleviate the non—response problem to y ay be sugg v‘ , oug ,4. €3_ . u , - s 0 potential differences between reSpondents and non-respondents. Of coure, wave , _ I ' " tis’etween respon— dents and all members of the target population may be reassuring. But, demographic comparisons do not, of course, determine whether the reSpon— dents and non-respondents behave in the same way with regard to critical issues. C lients must become more demanding about survey research and marketing research standards. It is eXpensive, admittedly, to deal with the non- response bias problem correctly. Subsample measurement requires that a sample of the non— respondents (or, the whole population, if non- respondents cannot be identified) be administered a few of the key survey questions by abnormal methods, if necessary (perhaps involving “infinite” callbacks and significant monetary compensation). the surv _ Then, a comparison of the sample means of spondents and non-respondents provides conclu— ,.ve proof as to thc representativeness of the respondents‘ data. if differences are detected between the respondents and the non-respondents, all is not necessarily iost. Weighted averages ofthe respondent and non-respondent means may be deveIOped to provide estimates ofthe true overall mean of all population members (for those ques— tions asked of both). In such a case, suitable caveats about the existence and apparent direction of non~re3ponse bias would have to be highlighted in the associated written research report. Conclusions These false economies involve short»sighted and misguided efforts to minimize what is easily observable (out‘of-pocket monetary costs assou ciated with surveying efforts), while ignoring what is difficult to see (poor data quality). Overcoming these false economies requires changes in practice, attitude and perspective on the parts of both marketing researchers and end users ofmarketing research. Marketing research suppliers must guide clients toward making appropriate trade—offs between explicit out—oprocket costs and implicit opportunity costs (and data quality) in survey research undertakings. Clients must become more knowledgeable with respect to the strengths and limitations of marketing research in generai, and survey research in particular. Clients must become more demanding about survey research and mar“ keting research standards. And, most importantly, clients must be prepared to pay for better marketing research. ...
View Full Document

This note was uploaded on 01/18/2012 for the course MKTG 4100 taught by Professor Armentashchian during the Spring '12 term at Kennesaw.

Page1 / 5

False Economies Article - False Economies in Survey...

This preview shows document pages 1 - 5. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online