and Qualitative Quantitative Methods: Can They Be Joined? (Not By Causal Process Observations!)
Nathaniel Beck Draft of August 28, 2006. Prepared for the 2006 Annual Meeting of the American Political Science Association, Philadelphia, Sept., 2006
ABSTRACT Brady, Collier and Seawright have argued that causal process observations can be adjoined to data set observations. This implies that qualitative methods can be used to add information to problematic quantitative data sets and hence can solve quantitative research design issues. In a recent symposium in Political Analysis I argued that such qualitative information cannot be adjoined in any meaningful way to quantitative data sets. In that symposium the original authors oered several defenses. Here I consider those defenses. In particular, I argue that the causal process observations are useful either for the explanation of specic events, or to help in the theory building process (or in any other way that qualitative analysts nd them useful); but, they are not capable of being adjoined to standard quantitative data. Furthermore, there appears to be ambiguity in how BCS use the term causal process observation, and whether the BCS usage covers items that look more like data set observations.
Department of Politics; New York University; New York, NY 10003 USA; [e-mail]. Thanks to Henry Brady and David Collier for many extremely civil conversations, and to Gary King, Bernard Manin, Adam Przeworski and Jas Sekhon for comments on this paper. None of these people are to be blamed for my thoughts, but I hope I faithfully transcribed some of their thoughts.
1
In an important recent book, Brady, Collier and Seawright (Brady and Collier, 2004) argue for a unied methodology for both qualitative and quantitative social science that respects both traditions. While the collection contains many interesting contributions, I argued in a symposium on the book (Beck, 2006) that the key contribution, the joining of causal process observations to data set observations is chimerical. Causal process observations (CPOs) are the bread and butter of the qualitative analyst; data set observations (DSOs) serve the same role for the quantitative analyst. If my argument were correct than the admirable task of Brady, Collier and Seawright would be less successful than they have argued. In a rejoinder to my comment, Brady, Collier and Seawright (2006) argued, using examples from natural science, epidemiology and political science that CPOs can be combined with DSOs in ways that I argued were impossible. In this paper I continue the argument.1 BCS (Brady and Collier, 2004, 227-8) dene a CPO as [a]n insight or piece of data that provides information about context, process or mechanism, and that contributes distinctive leverage in causal inference. A causal-process observation sometimes resembles a smoking gun that conrms a causal inference in qualitative research, and is frequently viewed as an indispensable supplement to correlation-based inference in quantitative research as well. The tie of CPOs to the qualitative analysts standard method is strengthened by adding to the denition a reference to process tracing. CPOs are distinguished from DSOs; the latter are the quantitative researchers typical measures on a set of variables for each subject or case in the study. Obviously quantitative analysts nd DSOs to be of value, and qualitative analysts likewise nd CPOs to be of value. There is no need to debate this here. Clearly if we do two separate analyses, and each sheds some light, then the two together must shed more light than either one alone. The only debate is whether the two types of observations can be meaningfully combined.2 But I take it the latter claim is what is novel in BCS. Let me stress what is not at stake in the current paper. Researchers use a combination of quantitative and qualitative methods, in diering proportions. Some nd one or the other type of method of more use, but few if any readers of BCS would rule out either method a priori. Thinking primarily of comparative politics (broadly dened, so that it includes any types of comparisons across political units, and so includes almost all international relations and a good deal of the area study of American politics), the standard dissertation or book length study has a mix of quantitative and qualitative chapters (with additional purely theoretical chapters). Each of these chapters provides insight into some issue. And these analyses can be used to inform the others in an iterative manner (whether formally, as
As with all controversies, the reader familiar with the earlier pieces will be better able to evaluate the current eort. For simplicity, I refer to for both the earlier Brady, Collier and Seawright argument and the rejoinder (unless a specic citation is needed) and I use I to refer to my published comment as well as the current argument. I trust this will not confuse the reader. 2 Brady, Collier and Seawright (2006, 326) are willing to drop the term observation from CPO, noting that [g]iven this potential misunderstanding as to whether researchers directly observe causal-processes[, i]f some readers nd it more helpful to think of this as causal-process information, that is a useful alternative label. However, to reiterate, we deliberately called these pieces of data causal-process observations to emphasize that this kind of evidence merits the same level of analytic and methodological attention as data-set observations in quantitative research. They continue to use the terminology of adjoining these alternative kind of data (Brady, Collier and Seawright, 2006, 368).
1
2
in Lieberman 2005 or more informally). What is at stake here is whether the two types of observations can be adjoined in a single analysis. The basic claim I am making is that the correct part of the BCS argument is simply what good quantitative researchers have always been doing, and provides no new synthesis of quantitative and qualitative methods. There is no doubt that what BCS call CPOs are an important component of any qualitative analysis; they are also vital for purely quantitative analysis, but here their role is in developing some basic understanding of how the world works. We expect students of comparative politics to know their cases. To take the most straightforwardly quantitative subelds of political science, our best scholars of the legislative process have spent a year as Congressional Fellows, and our survey researchers pretest their instruments using, amongst other things, focus groups. In my own work on the politics of monetary policy, which in the end used only DSOs, my rst step was to talk with various people inside and outside the Federal Reserve. While this does not guarantee that one will not still say some foolish things, clearly it does help to guard against some stupidities. Similarly, any American who has grown up outside Chicago would probably realize, as did Brady, that it would be very hard for any Floridian living in the panhandle counties to fail to vote based on an early call of the Florida outcome if they had already voted. Martians could be very good at running regressions but they would need some eld experience before we would believe their quantitative analyses of Earth politics. The examples BCS use are from astronomy and paleontology, epidemiology and political science. While epidemiology looks a lot like quantitative social science, astronomy and paleontology do not. Hence I deal with the latter examples very briey. All the various astronomical and paleontologic observations discussed look, to me, like straightforward hard data. (I would not call them DSOs since in these elds there is no data spreadsheet.) The observation of moons of Jupiter or various chemicals in the earths crust hardly look like what comes from process tracing. In many areas of science, where the claim would be that stochastic variation is irrelevant, scientists proceed by saying if theory A is correct we must observe X, and so if we do not observe X, theory A must be incorrect. So errors in the positioning of planets (from a Ptolemaic perspective) allowed for rejection of Ptolemaic theory (though of course the history of science is a bit more complicated than this). In this elementary philosophy of science textbook view, astronomers would subject Copernican theory to a similar set of tests, looking for implications of the theory and then seeing if they are upheld. This overly simple but not incorrect view is how we all learned about the Einsteinian theories overthrowing the Newtonian theory, with the famous Michaelson-Morley experiment measuring the speed of light under various conditions, or observations made during total eclipses of the sun. We have similar stories about the implications of the structure of DNA for what characteristics various X-ray crystallographs should have. So if BCS are simply saying that we should test the implications of our theories by examining whether their consequences are consistent with what we observe, it would be hard for anyone with any positivist view of science to disagree. But what this has to do with causal process observation (or information) is beyond me. Epidemiology is more similar to social science. Compared to astronomy, these areas have much weaker theory, and theory testing must take into account a lot more randomness. So 3
the two examples cited by BCS, Semmelweis work on puerperal fever and Snows on cholera, may be more relevant to our own discipline. What is to be said about BCSs claim that both analysts combined CPOs with DSOs? Both Semmelweis and Snow are known for is their careful testing of theories rather than theory development. Thus Hempel (2006, 16970) states [o]ther people before Semmelweis ... had correctly hypothesized about the nature of puerperal fever, and how it was being spread, although their message, like Snows on cholera, was largely ignored. Semmelweis is famous amongst epidemiologists then, not so much for the originality of his ndings, as for the statistical, scientic approach that he brought to investigating the problem. It was not his observation of one doctor with symptoms similar to puerperal fever, but rather his disinfection experiment, that made him an icon of epidemiology. The experimental data were pure DSOs and are completely consistent with purely quantitative (experimental) social science. Semmelweis work presents no challenge to standard quantitative methods. Similarly, Snow is known for his carefully analyzed quasi-experiments showing that cholera was a water borne illness, caused by drinking water that had been contaminated by infected sewage. Snow clearly did make theoretical as well as empirical progress, advancing the contagion theory of cholera to argue that it was spread by something ingested via a water borne. Snows belief that cholera was a water borne illness was based on careful observation, particularly that the disease rst aected the alimentary canal and that its rapid spread made simple person to person transmission an unlikely route for spreading the infection. We can argue about whether observing that the initial symptoms of a disease are related to the digestive system bears any relationship to the qualitative analysts process tracing, but these CPOs were not part of the crucial empirical work that Snow did to show that cholera was a water borne disease. It is also interesting to note how misleading these CPOs can be. Brady, Collier and Seawright (2006, 364) argue that CPOs helped Snow make an initial evaluation of alternative mechanisms of transmission, such as the hypothesis that it was carried by miasma or bad air. Snow documented sequences of infections in specic individuals, who successively had close personal contact with each other.... For example, one individual died from cholera after staying in the room in a boarding house previously occupied by a sailor, newly arrived in London, who had cholera. Snow used this and similar information to infer that cholera could be transmitted from person to person. Such CPOs helped Snow to discard environmental explanations, such as miasma, and to focus instead on other vectors by which the disease might travel. Why infection seemingly caused by occupying the same location shows that the miasma (bad air) theory to be incorrect is beyond me, and what this shows about cholera being water borne is even more beyond me. CPOs can be misleading, though apparently less so after the fact! Many of us now of Snows work through the work of David Freedman. Freedman (1991, 299) concludes Snow did some brilliant detective work on nonexperimental data. What is impressive is not the statistical technique but the handling of scientic issues. He made steady progress from shrewd observation through case studies to analysis of ecological data. In the end, he found and analyzed a natural experiment. (Of course, he also made his share 4
of mistakes: For example, based on rather imsy analogies, he concluded that plague and yellow fever were also propagated through water....) Fortunately epidemiologists studying the plague and yellow fever could rely on DSOs rather than CPOs to assess how plague and yellow fever were spread. Snow convinced people that cholera was a water borne illness by two famous quasiexperiments. In one, he showed that users of a water company that drew from the contaminated portion of the Thames River suered from cholera at a much higher rate than users of another company that drew from upstream, even though users of the two companies were so mixed that many neighbors used dierent companies. In the second, he showed that users of the contaminated Broad Street pump were much more likely to get cholera than users of other water sources. Again, we need to be careful about CPOs here. Brady, Collier and Seawright (2006, 364) state that the cholera epidemic came to a close shortly after Snow convinced the authorities to remove the pump handle, thereby preventing people from using the contaminated water .... This sequence of specic, localized CPOs, most dramatically the consequences of removing the pump handle, were diagnostic markers that linked the cause (contaminated water) to the eect (cholera). Alas, Freedman (1991, 2956) gives a dierent account. As the story goes, removing the handle stopped the epidemic and proved Snows theory. In fact, he did get the handle removed and the epidemic did stop. However, as he demonstrated with some clarity, the epidemic was stopping anyway, and he attached little weight to the episode. CPOs seems to speak much more clearly in hindsight. Note that it is very dicult to answer the question why did the London epidemic end? whereas standard methods make it possible to ask what are the eects of using contaminated water? Snows empirical work was brilliant, but shows only excellent quantitative analysis rather than any challenges to such analysis. For example, he could not observe who used the contaminated Broad Street pump, so he plotted cholera outbreaks versus distance to the pump. But rather than using Euclidian distance, he took the distance it would actually take to travel to the pump over available streets. (He also famously found some brewery workers near the pump who did not get cholera, since they had no need for water.) In the analysis of users of the two water companies, he had to do incredibly hard (and clever) work to gure out which company serviced a given house. Armed with his list of addresses where deaths had occurred, he travelled to south London and, like a detective solving a crime, he began weeks of pounding the streets and knocking on doors, asking the same question from which company did the household get its water? (Hempel, 2006, 172) Both Freedman and Hempel use the metaphor of the detective. But in both cases, they are referring to careful observation. Snows detective work or shoeleather epidemiology consisted of the gathering of hard, but very standard, DSOs. While the work was hard, it consisted simply of gathering, for a large number of households in the cholera aected area, the presence of absence of cholera and the water source for the household (measured either by which water company was used or closeness to the Broad Street pump). The modern day social science analogue of Snow as detective is the household researcher, carefully gathering information on household behavior. Hard work, careful work, but no challenge to standard quantitative analysis and no challenge to making inferences based on (good) DSOs. 5
Brady, Collier and Seawright also use the metaphor of the detective. At the simplest level, CPOs are diagnostic pieces of information that provide key insights in assessing explanations. A standard metaphor employed in discussing CPOs involves the parallel to criminal detective work. Detectives make their diagnoses on the basis of dogs that dont bark (as in Sherlock Holmes famous Silver Blaze story), missing suicide notes, other clues that are or are not found at crime scenes, and stories that just dont add up. As will be shown in the examples below, this type of analysis goes beyond a simple model of cause and eect and recognizes that a causal process typically involves complex mechanisms, mediators, and markers that can provide alternative ways to test theories and to develop explanations. Paying attention to these mechanisms, mediators, and markers can reveal causal processes, and they are the foundation for CPOs. (Brady, Collier and Seawright, 2006, 360) This use of the metaphor is dierent from that of Freedman and Hempel. To see the dierence, we must distinguish attempts to nd law like generalizations from attempts to explain specic events.3 In my previous critique I have argued that the former is what social scientists do. But obviously there is disagreement here. As Mahoney and Goertz (2006) note, qualitative researchers are often interested in explaining single events while quantitative researchers are more interested in generalizable statements. Returning to the detective metaphor, I think that social scientists resemble criminologists, that is, seekers after general principles. These would be dogs generally bark in the presence of strangers or suicides generally leave notes. Establishing whether these principles are consistent with the behavior of dogs or suicides is the stu of standard quantitative analysis using only standard DSOs (though perhaps our criminologist rst thought about dogs not barking by observing his own dog). Now our detective uses these generalizations to help explain a specic case. Did Mr. X commit suicide? Unlikely, since no note was found. But specic cases admit to complicated causal stories. So perhaps Mr. X was a suicide but, alas, illiterate. Or maybe the dog that did not bark in the night was a Basenji. Explanations of individual events will also be complex and involve complicated causal stories. Why the did US invade Iraq? Clearly there are systematic explanations, but we can also nd very specic stories related to highly non-systemic matters (say family relationships). My own take is our interest is in systematic relationships (laws), but it is evident that there is interest in explaining specic events. And it is clear that people who explain specic events often use the types of qualitative information that BCS call CPOs.4 My disagreement with BCS relates to whether CPOs
I do not think we need to be precise about these generalizations or what the term law like actually means. Some generalizations are largely empirical (and have survived empirical testing) while others are theoretical statements (that have survived empirical testing). Thus for present purposes it does not matter much if we think of Duvergers Law as largely an empirical generalization (with a plausible story) or a theoretically derived law which has been subject to serious empirical scrutiny. What is relevant is that either version can used to answer the question why does the United States have a two party system? and that our interest is in the general proposition, not the specics of the United States. 4 While not the subject of this paper, it is not obvious to me that social scientists should proceed in this
3
6
can somehow be adjoined to as dataset containing DSOs. With this, we can move on to the political science examples. BCS discuss four examples, three from their previous work and a new one, Liebermans (2003) comparative study of the determinants of a nations ability to raise taxes.5 I have little to add on what I said previously on the Tannenwald (1999) and Stokes (2001) studies. Tannenwald, having four cases and no variation on the dependent variable, turns to documents to see the accounts that decision makers gave of why they failed to use nuclear weapons in a crisis; Stokes supplemented her cross-national regressions with accounts of policy-makers as to why they chose neo-liberal policies. I suppose that reasonable people can dier on the utility of accounts given by decision-makers as to why they did what they did; sometimes they tell stories we like, and we are happy, and sometimes not. Tannenwald simply had no useful data and turned to a qualitative analysis; Stokes admirably provides qualitative and quantitative evidence, but her qualitative evidence is about specic cases, not general processes. Are we shocked that her policy-makers gave her accounts in terms of general economic theories? Imagine a study of why members of Congress facing serious scandals choose not to run for re-election. Imagine previous decades where there are very few such scandals, and the three or four scandalized all chose not to seek re-election. Would we be as happy with the conclusion that members of Congress facing a scandal chose to spend more time with their families as with the CPOs of Stokes and Tannenwald? But the only claim I really wish to make here is that CPOs cannot be adjoined to DSOs; certainly many scholars nd CPOs to be of great interest on their own. Lieberman rst ran a standard cross-country regression and found that tax revenues were well predicted by economic development. He then looked closely at two cases, South Africa and Brazil, with similar levels of economic development but very dierent tax revenues. He then hypothesized that elites are willing to pay taxes if they believe that they will receive a non-trivial portion of the subsequent government spending. Lieberman then tested this with another regression. The moves from case analyses to regression is not uncommon; as noted above, Martians do not typically run sensible regressions and students of Congress
way. Mahoney and Goertz (2006, 230) note that quantitative scholars want to nd the eects of causes while qualitative scholars are more interested in the causes of eects, that is, in explaining outcomes in important cases. They then argue that the causes of eects approach is consistent with normal science as conventionally understood and that most natural scientists would nd it odd that their theories cannot be used to explain individual events[,] using the example of explaining the space shuttle Challenger disaster. My view is that engineers (or in this case Nobel Prize physicists) could show that O-rings become brittle at low temperature, which looks like a law to me. This, combined with very cool temperatures at launch time, explains the explosion. This is a standard syllogism based on a law, where the critical law like component is either a theoretical or empirical generalization that has been empirically tested. So, as with our detective, there is nothing unusual about using law like statements to explain specic events, but this does not mean that our work as social scientists is not in coming up with the law like statement. We are quite good at guring out that smoking causes cancer (or how smoking changes the relative risk of cancer), but much less good at coming up with a full understanding of why John Smith died of cancer (or even what proportion of cancer deaths are due to smoking). While in a court of law it might be relevant that he also worked with asbestos, it is the eect of smoking on cancer that should be of interest to epidemiologists. 5 As in my earlier comments, my issues here are purely methodological and I have no interest in critiquing any of these works, which were chosen because BCS discussed them. I have tried to only deal with issues raised by BCS.
7
rst observe it before they model it. But to say that Lieberman adjoined CPOs to DSOs seems to be stretching the idea of adjoining. While I am sure that there are some Martian political scientists out there mindlessly running regressions on comparative issues they do not understand, and while obviously Lieberman devotes relatively more eort to case study than do most quantitative political scientists, there is nothing in his work to challenge the standard ideas of inference (as in King, Keohane and Verba, 1994). Finally, let us turn to Bradys reanalysis, with out interest, as usual, being whether CPOs can be adjoined to DSOs. Brady seeks to make a causal inference about a singular outcome i.e., whether the early media call of the election yielded a substantial vote loss by Bush in the Florida panhandle. Brady argues against the large-N, regression-based study carried out by Lott, which played a critical role in the national debate on the election, given Lotts conclusion that Bush did indeed suer a major vote loss due to the early media call. Brady disputes Lotts ndings, basing his analysis on CPOs rather than DSOs, as in Lotts analysis, utilizing a form of quasi-detective work focused on a sequence of steps that would have had to occur for the vote loss to be plausible. Far from oering an informal analysis that impressionistically suggests the linkages between the putative cause (the early media call) and the supposed eect (lower turnout), Brady presents a carefully structured evaluation focused on a sequence of necessary conditions. These necessary conditions are not met, and the causal claim presented by the large-N study is eectively ruled out. Contra Beck, this is as instance of successfully adjoining CPOs and DSOs to address the same problem of inference. (Brady, Collier and Seawright, 2006, 3556) As noted in my earlier piece, there is no doubt in my mind that Bradys analysis is superior to that of Lott. Lott simply regressed county turnout in Florida in four national elections on xed county and year eects, and, based on that simple two way analysis of variance, found that turnout in the Panhandle counties was approximately ten thousand voters fewer than predicted by the simple analysis of variance. This is a totally atheoretical regression, of the type that our hypothetical Martian political scientist might choose to run. Also, this is a discussion of a specic case. While Brady, most likely correctly, stresses the role of Democratic mobilization in south Florida, it also might have rained in the Panhandle, or there could have a major trac accident, or any number of other idiosyncratic factors leading the Lott result. While obviously turnout in Florida in 2000 was a politically (and perhaps legally) important issue, our interest as political scientists should be the determinants of turnout (including factors like party mobilization). These can then be applied to specic cases should interest warrant.6 But, as I noted previously, each of Bradys necessary conditions comes from, or could come from, a purely quantitative analysis. He starts by noting that people who voted before
Thus it is clear that Brady was interested in Lotts claim about Panhandle turnout rather than general theories of voting behavior. But it is the generalizations about turnout that Brady uses for this explanation. Thus, while more complicated, the structure of Bradys argument is no dierent from the argument about the cause of the Challenger explosion discussed above.
6
8
the election call could not have been aected by it. Now while we do not have good studies of how many people vote right as the polls close, Brady does come up with a plausible number. But surely this is not a DSO. Then, based on prior purely quantitative work on media attention, turnout, and the eect of previous early calls, he shows that the maximal eect of the early call on turnout in the Panhandle must have been small. Now what of the claim that the carefully structure sequence of necessary conditions makes the Brady study the successful adjoining of CPOs to DSOs. If this simply means that Brady has thought about what it would take for the early call to have mattered, then we all agree that CPOs are important. But what this has to do with smoking guns or process tracing is beyond me. To make the issue simpler, let us return to epidemiology. Imagine we are interested in how many lives will be saved by a new screening process. So we do a standard experimental study (surely no CPOs here), randomly giving some people the screening process, some a placebo process (let us not worry about any details here). We then can compute both the false positive rate (how many unnecessary operations will be done) and the dierence between the true positive and false negative rates, telling us how many lives will be saved. We then compute how many deaths are caused by unnecessary operations, and perhaps do another quantitative study on how many people will choose to be screened. We can then, using purely quantitative tools, and only DSOs, compute how many lives the screening program will save (or cost!). Now if by CPO we mean an understanding that in life many will not choose to be screened, or that false positives are costly, then of course we need CPOs. But this is not the usage of CPO as dened by BCS. Similarly, this analysis is multiplicative, not additive, and based on necessary conditions (one cannot die from a procedure not undertaken). But what does this have to do with CPOs? So while Brady did an admirable study, I fail to see how this is an instance of successfully adjoining CPOs to DSOs. I hope that being committed to DSOs and quantitative analysis does not mean that I also must be committed to being a Martian political scientist! What then can we make of the general argument. Brady, Collier and Seawright ask, and then answer why a...