LaunchingTheSpaceShuttleChallenger - IEEE TRANSACTIONS ON...

Info iconThis preview shows pages 1–2. Sign up to view the full content.

View Full Document Right Arrow Icon

Info iconThis preview has intentionally blurred sections. Sign up to view the full version.

View Full DocumentRight Arrow Icon
This is the end of the preview. Sign up to access the rest of the document.

Unformatted text preview: IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 38, NO. 1 , FEBRUARY 1991 63 Launching the Space Shuttle Challenger: Disciplinary Deficiencies in the Analysis of Engineering Data Frederick F. Lighthall Abstract-This paper analyzes published and archival testimony of participants in the decision to launch the space shuttle Challenger and extracts new lessons from the decision process for engineering training and engineering managers. Examination of interview testimony, published hearings, and tabular data examined by the decision participants at the time of the Challenger launch show not only that analysis of data and reasoning were flawed, but that the flaws are attributable in large measure not to personal or even organizational failings but rather to a professional weakness shared by all participants. The professional weakness pointed to is either curricular or instructional: a gap in the education of engineers. Staff engineers and engineering managers arguing for and against the launch, all of whom agreed they had insufficient quantitative data to support an argument against the launch, were unable to frame basic questions of covariation among field variables, and thus unable to see the relevance of routinely gathered field data to the issues they debated before the Challenger launch. Simple analyses of field data available to both Morton-Thiokol and NASA at launch time and months before the Challenger launch are presented to show that the arguments against launching at cold temperatures could have been quantified, but were not quantified, t o the point of predicting degrees of component failure beyond those held by decision participants to be safe. The weakness in engineering education, in turn, is taken to be of a pervasive genre: An overemphasis in contemporary universities and research centers on specialization and analysis and an underemphasis on synthesis of knowledge across fields. A larger lesson of the accident, then, is that professional narrowness, leading to false diagnosis of cause-effect relations, can he fatal. Keywords: Decision making, data analysis; case study, statistical reasoning; Challenger; organizational processes; curriculum, engineering education. INTRODUCTION 0 NE essential ingredient of an effective education is the imparting of ideas and methods necessary to test and integrate the range of realities the student is likely to face. This holds for the specialist in professional life as for the citizen in everyday life. Similarly, organizational effectiveness depends on the organization’s capacity to test and integrate the realities with which it is faced, day by day, year by year. The flawed decision making that led up to the launch of the Challenger is a case where the issues of educational effectiveness and organizational effectiveness were joined: Could those flawed decision processes have resulted from an inadequate professional education that all participants shared? The decision on January 27, 1986 not to further delay launching the Challenger the following day took place over several levels of review and authority and involved technical questions of causality, questions requiring participants to disentangle the effects of different factors and conditions on joints in the shuttle’s Manuscript received November 6, 1989; revised May 11, 1990. The review of this paper was processed by Editor D. F. Kocaoglu. The author is with the Departments of Education and Psychology, The University of Chicago, 5835 South Embark Avenue, Chicago, IL 60637. IEEE Log Number 9039019. booster rockets. I start the argument of this paper from a widely accepted premise, that the decision process was “flawed,” but point to causes others ignore, fail to probe, or discount entirely. Feynman [6] focused on the failure of those at lower levels in the process to convey information to higher levels, a point whose validity rests importantly on the assumption that information at lower levels represented an adequate testing and understanding of the realities participants faced. I adduce evidence in this paper that at the level most directly dealing with the basic facts of cause and effect, where key decision-making participants discussed the functioning of rocket joints, data and analyses were not valid but deficient. Had their data and analyses been as effective as I shall show they might have been, a firm decision to delay would probably have been made by the contractor and, being thus binding, would have been passed up the line. McConnell’s [ 113 review of those same decision processes characterizes the Morton-Thiokol engineers’ presentation as “meticulous” and “convincing” (p. 194) but (a) makes no analysis of the engineers’ tabular data or causal arguments and (b) ignores the fact that neither the data nor the arguments were convincing to three NASA managers and three active Thiokol managers, including R. K. Lund, Vice President of Thiokol’s Engineering. Boisjoly and colleagues [2] point to weakened responsibility among managers of large organizations as an important part of explaining the decision. In such a system of authority, based on rules rather than on individual integrity, individual judgment is said to be replaced by a mere following of rules and fulfilling of categories in order to avoid unpleasant truths. Their analysis, like those of Feynman, of McConnell, and of the Presidential Commission itself, portray engineers as heroes and managers as villains-or at best as unwitting victims of a system that subverts individual intelligence and responsibility. Jackal1 [7] offers detailed support for that line of analysis. But multiple lines of inquiry are required to comprehend the complexity of such organizational decision processes. The Presidential Commission [ 161 presented two findings which in an important respect are contradictory: . . . Thiokol Management reversed its position and recommended the launch of 51-L [the Challenger] at the urging of Marshall and contrary to the views of its engineers in order to accommodate a major customer (p. 104, emphasis added), and A careful analysis of the flight history of O-ring performance would have revealed the correlation of O-ring damage and low temperature. Neither NASA nor Thiokol carried out such an analysis; consequently they were unprepared to properly evaluate the risks of launching the 51-L mission in conditions more extreme than they had encountered before (p. 148, emphasis added). 0018-9391/91/0200-0063$01.00 O 1991 IEEE 64 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 38, NO. I , FEBRUARY 1991 The contradiction lies in the weight given in the first finding to the “views” of Thiokol’s engineers as right thinking while focusing in the second finding on a deficiency at both NASA and Thiokol in the analysis of data centrally related to the assessment of risk. The first finding identifies managers at Thiokol and Marshall Space Center with untruth leading to a disastrous decision, and identifies Thiokol engineers with the truth that should have prevailed. Implicit in this finding, and in the analyses of Boisjoly, et al. [ 2 ] , McConnell [ l l ] and at points Feynman [6] is the metaphor of Robber Barons, the powerless ones in the right, the powerful ones in the wrong. That metaphor is useful for arousing righteous indignation and solidarity in action. But it must not displace analysis of cause and effect, where a more useful metaphor is that of effective adaptation. The adaptation metaphor draws our attention to an unanticipated, threatening condition arising in a particular situation, leading to effective, or ineffective, testing of that situation’s realities and to successful, or unsuccessful, integration of those realities into action The Commission’s second finding focuses precisely on reality testing: “careful analysis of the flight history . , . .” It found absent an analysis that would have revealed a relation between O-ring failure and temperature. The Commission was silent about why “neither NASA nor Thiokol carried out such an analysis.” The contradiction between the Commission’s two findings lies in the fact that the analyses it found wanting in the second finding would be primarily the responsibility of the engineers, identified as right thinking in the first finding. The Commission’s suggestion of carelessness (‘‘A careful analysis . . . ”) is overwhelmingly contradicted by testimony of engineers and managers at Thiokol regarding participants’ presentations of tabular data at NASA Headquarters in Washington, at a professional conference, and in the January 27 teleconference. I shall argue, in contrast, that the Commission’s identification of both NASA and Thiokol as analytically deficient is supported by testimony from both managers and engineers of both NASA and Thiokol. The present paper analyzes the extent and probable source of those analytical deficiencies. TECHNICAL BACKGROUND self-sealing in that the very gap that initially allowed a small escape of the highly pressurized gas was the means by which the nearest O-ring was forced into that same gap. If for some reason the nearest O-ring (the “primary”) failed to seal the opening, the second O-ring was there as back-up. A central question in discussions leading up to the Challenger launch was whether temperatures predicted for launch time would allow normal O-ring functioning. A year earlier O-rings of booster rockets of the shuttle launched on January 24, 1985 (designated Space Transportation System 5 1-C) had been found both damaged and breached: black soot was found between the primary and secondary O-rings. The only conclusion engineers could reach after examining the recovered rocket’s joints was that the unusually cold weather leading up to and at the time of that launch had caused the O-rings to harden, thus to move more slowly, and thus to allow more gas to blow by and to erode the O-rings. O-ring temperatures for that launch had been estimated at 53 OF, the lowest temperature of any launch up to that time. Both NASA and Thiokol engineers, therefore, had come to regard low temperature as dangerous to proper O-ring sealing. The booster rockets had two kinds of joint: field joints (assembled at Cape Kennedy, i.e., in the “field”) in the main body of the rocket’s cylindrical case, and a nozzle joint where the nozzle was joined to the aft segment or “case”. The nozzle-to-case joint was structurally different from and known to be subject to greater combustion heat than, the field joints. But field joints became bent open (joint “rotation”) under pressure of ignition such that the secondary O-ring could lose contact with the surface of the tang between an estimated .17 and .33 seconds after ignition. If the primary O-ring had failed to seal successfully before this rotation gap opened up, the secondary O-ring, subject to this rotation gap, might also fail, causing total joint failure which all knew to be disastrous.’ CONDITIONS, EVENTS, ARGUMENTS AND The central issue worrying engineers at Thiokol from about noon the day before the scheduled launch was whether the predicted temperature in the joints of its two booster rockets of ’ A second complication was that malfunction of the joints was signaled by The immediate cause of the accident was found to be a breach different kinds of evidence. After post-flight disassembly of a rocket, secwhere two segments of a booster rocket were joined (a “field ondary O-rings could show evidence of heat without damage to either primary or secondary O-ring, gas having blown by the primary O-ring joint”). The immediate cause of that breach was found to have briefly before sealing. This “blow-by” was taken to be benign, resulting been malfunctioning of the two rubber-like O-rings in the joint, from only momentary opening of the joint, sealed within a few milliseconds. rings designed to be forced into the small gap between the two Blow-by taken to be malignant was evidenced by gray or, worse, black soot segments by the pressure of expanding gas immediately upon between primary and secondary O-ring. Blow-by of either kind might, or might not, be accompanied by actual erosion of the primary (rarely the ignition of the rocket’s fuel. secondary) O-ring material. Furthermore, two types of erosion were distinEach of the spacecraft’s two 150-ft long booster rockets was guished: “impingement” erosion and “by-pass” erosion, the latter resulting assembled from four cylindrical segments some twelve feet in in sooted blow-by. Impingement erosion was erosion that was terminated by diameter and 26-ft in length, tipped by pointed front-end assem- successful seating of the O-ring. It occurred as pinpoints of hot gas burned blies and tailed by a nozzle assembly from which burst the away O-ring material in the milliseconds taken by O-rings to move to a seal. It is not clear whether or when participants distinguished these different rocket’s thrust. Each cylindrical segment’s edge or lip fit its meanings of “blow-by” and “erosion” in their own minds, nor whether in adjoining segment’s lip by a tongue-in-groove configuration of communications they meant the same things by ”blow-by,” ”erosion,” or the two joined lips such that the front lip of each segment was ”damage. ” Boisjoly described blow-by, erosion, and joint rotation in these terms: actually U-shaped (a “clevis”) into which the rear lip of the “O-ring material section segment ahead would fit as the tongue (the “tang”). The inner much, faster whengets removed from the crossblow-by, of the O-ring much, as people have been you have bypass erosion or projection of each segment’s U-shaped clevis contained two terming it. We usually use the characteristic blow-by to define gas past it, grooves inside the “U” running around the entire circumference and we use the other term [bypass erosion] to indicate that we are eroding at of the segment (about 37 ft). In each of these grooves was the same time. And so you can have blow-by without erosion, you can have inserted a .280-inch diameter O-ring. The O-rings normally blow-by with erosion.” ( R P C , pp. 784-85) “At the beginning of the transient cycle [initial ignition rotation, up to .17 responded to the pressure of motor ignition by following the seconds] . . . [the primary O-ring] is still being attacked by hot gas, and it is direction of the pressure and extruding into the very small gap eroding at the same time it is trying to seal, and it is a race between, will it between the tang-clevis joints. The joint, thus, was normally erode more than the time allowed to have it seal” ( R P C , p. 136). 65 LIGHTHALL: SPACE SHUTTLE CHALLENGER: DEFICIENCIES IN ANALYSIS OF ENGINEERING DATA the Shuttle expected to be 29 “ F at launch time, would cause both O-rings in any one of those joints to fail to seal, causing destruction of the rocket and Shuttle. Extensive post-accident analysis [16] of recovered Challenger rocket debris and of standard rocket motors, revealed that, indeed, the triggering cause of the accident was the failure of one of the rocket’s field joints, a weakness that both engineers and managers at both NASA and Thiokol had known about for more than a year, and that a special task force of engineers had been working seriously to correct for some months. A telephone conference the day before the accident, among 16 NASA engineers and managers at Marshall Space Flight Center in Huntsville, AL and at Cape Kennedy in FL, and 14 Thiokol engineers and managers gathered at the rocket plant in Utah, reviewed data assembled by Thiokol engineers in which the latter, in unanimous agreement, argued three points: 1) the lowest previous launch temperature (of flight # 15 the previous January, designated “STS-51-C”) had been 53 OF, 2) Thiokol’s booster rockets of STS-51-C had experienced the most serious O-ring damage and evidence of seal failure of any flight, and 3) taking all “bench” test data and flight data into account, the impending launch of the Challenger should be delayed until its booster O-ring temperatures reached 53 Key NASA managers countered in this teleconference that the Thiokol engineer’s data, presented via both their 13 telefaxed charts and their commentary and argument, did not support their recommendation to delay launching, since they failed to demonstrate that temperature was a factor in O-ring performance or damage. NASA managers argued for example, that earlier, qualifying rocket motors had been tested (only static tested) below 53 “ F without incident and that the O-rings of a recent rocket motor launched at 75 “ F had shown a good deal of sooted blow-by. Close analysis supports the Presidential Commission’s conclusion that NASA managers’ commitment to launch the Challenger overreached both caution and data. The whole decision process of that launch is studded with flaws up the chain of command. At issue here, however, is the capacity of all participants to collect, present, and analyze data such that managers’ sometimes overreaching commitments to production schedules and career advancement do not prevail over detection of dangerous realities. Thiokol managers, crediting the NASA managers’ criticisms with some validity, interrupted the teleconference to hold a separate caucus among the 14 Thiokol engineers and managers gathered in Utah. After some 30 min of caucus discussion, Thiokol managers returned to the teleconference with a complete reversal: Unable to sustain a case that temperature affected O-ring blow-by or erosion, they now recommended launching. Analysis of the charts of data and arguments telefaxed to all participants by Thiokol, of hearing transcripts, and of lengthy interviews by Presidential Commission investigators of all key * Boisjoly summed up his worry about effects of cold temperature on O-ring sealing thus: “ . , , the issue is can you stand a longer period of time in an attempt to seal [with cold, sluggish O-rings] before the erosion eats you alive and you don’t have a seal, period?” ( R P C , p. 796) Decision flaws due to “linear processes” (Lighthall [8])-passing information “up the line” or “down the line”-are important, were much in evidence in the full launch decision, and will be addressed in a subsequent paper. While Morton-Thiokol managers reversed their earlier recommendation to delay launching and passed on to the NASA managers a decision to proceed with the launch, NASA managers could have, but chose not to, pass up the line the debated “concerns” and the stance reversal of the contractor (see especially Presidential Commission chair Roger’s interchanges with Mr. Mulloy, a NASA manager, RPC, I: 97-98). ’ -1 participants, and of many peripheral and silent participants, reveal an absence of elementary statistical ideas and analytical methods-ideas and methods critically relevant to the focal argument that temperature was, or was not, a significant cause of O-ring blow-by or damage. THECHARTS The Thiokol engineers who were closest to booster rocket functioning responded to the threatening cold temperatures by assembling data supporting their argument that: 1) the predicted temperature of 29 “ F was too cold for a safe launch, and 2) the launch should be delayed until O-rings warmed up to at least the temperature of the coldest previous launch, 53 OF. The logic of such an argument required that they show how the temperatures of previous launches related to O-ring damage, or blow-by of previous launches, showing in particular how lower temperatures of launches matched up with higher incidence of O-ring damage or blow-by and how higher temperatures matched up with lower incidence of O-ring damage or blow-by. The element r form of conveying a relationship like that between two ay variables, temperature and damage-a schema learned in any first course in statistics-is the bivariate plot. Such a plot would cast one variable, say temperature, on the horizontal axis and the other, some index of O-ring damage, on the vertical axis, and would place one dot in that two-dimensional space for each of the launches to date.4 An alternative to such a bivariate graph would be to list all pairs of temperature and damage readings in order, say, from high to low temperature, showing how as temperature dropped, numbers representing O-ring erosion increased. Neither of these types of data were conveyed via the charts by the Thiokol engineers either that day or in formal review of data in the previous year. Of the 13 charts circulated by the Thiokol managers and engineers to the scattered teleconferees, six contained no tabled data about either O-ring temperature, O-ring blow-by, or O-ring damage (These were primarily outlines of arguments being made by the Thiokol engineers). Of the seven remaining charts containing data either on launch temperatures or O-ring anomaly, six of them included data on either launch temperatures or O-ring anomaly but not both in relation to each other. The remaining chart contained results of full pressurizing of two small, sub-scale facsimiles of rocket motors at 75 “F and 30 “ F without O-ring leakage. The data of this chart, however, were explicitly discounted by its discussant: the gas used to pressurize had inappropriate properties, invalidating the results. None of the tables in any of the charts contained more than seven independent data points. For example, the first chart presented in the teleconference reported the damage sustained to O-rings in field joints of the rocket motors of seven flights by erosion depth, perimeter affected, length of maximum erosion, total heat-affected length, and clocking location around the circumference of the rocket. But no temperature data appear in that chart. Similarly, another chart laid out in seven even data points of temperature from 70 to 10 “F corresponding values of O-ring hardness measured in another sub-scale motor simulation-without any corresponding simulation or test of O-ring erosion or blow-by. Several charts mentioned both temperature and sooted blow-by from two flights in particular: STS-51-C (flight # 15, at 53 “ F on January 24, Booster rockets of the fourth flight were lost at sea, so no damage aSSeSSmentS were possible. Also, since the 24th flight had taken place only two weeks before the Challenger, data from that flight would not have been available for analysis. IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 38, NO. 1 , FEBRUARY 1991 66 1985) and STS-61-A (flight # 22, at 75 “F, on October 30, 1985). STS-51-C suffered severe, black-sooted blow-by and O-ring damage in its field joints. STS-61-A also experienced sooted blow-by, but noticeably grayer, with less carbon than STS-51-C. NASA managers at the Cape and at Marshall Space Center argued pointedly that the flight at 75 degrees with blow-by vitiated the argument by the Thiokol engineers that STS-51-C’s severe blow-by and accompanying erosion was probably caused by that launch’s unusually cold temperature. The NASA manager’s focus on only two data points to argue against a relationship between two variables definable by the 22 points of data available was fallacious. Nevertheless, no one in any role at any location asked for any data from other flights besides these two flights relating O-ring damage to temperature. In sum, neither the charts nor the participants’ commentary and argument focused on a relationship between the 22 available launch temperatures and the 22 degrees of O-ring damage for those launches. Nor did anyone at any of the three teleconference locations, from either NASA or Morton-Thiokol, engineer or manager, ask for or make reference to either 1) temperature data on non-damaged flights or 2) the full set of pairs of temperature-damage data points from all flights to date. Preparation of the charts used in the teleconference was th rushed. Several were telefaxed from U a handwritten. Some were sent only after the teleconference had begun, and the conclusions and recommendations arrived only after much of the presentation. At the Thiokol plant, participants were in and out, doing separate analyses and then brining them to the conference room where Lund and others outlined charts on the board. Were they all so rushed with the deadline of decision for the following day’s launch that their normal collective capacities to look for and to see bi-variate relationships escaped them? PARTICIPANTS’ EXPLANATIONS The answer to the question, why, can be found in testimony given before the Presidential Commission and in the lengthy interviews by Commission staff members of all who participated in the teleconference the day before the launch. They indicate not only widespread, explicitly admitted inability to ‘‘quantify” the effects of temperature on O-ring blow-by or damage but also a general incapacity to use data to test whether patterns of relationship were random or revealed systematic effects. Both active and silent participants revealed these two analytic weaknesses, but only testimony from the key participants need be cited here. Roger Boisjoly, one of Thiokol’s two most informed engineers about O-ring properties and functioning, spoke at greatest length and in greatest detail about the charted data in the initial teleconference. Both Boisjoly ’s conviction and his admitted inability to quantify are evident in his testimony before the Commission: Boisjoly [describing his arguments during the teleconference with NASA officials]: “I expressed deep concern about launching at low temperature . . . a lower temperature than current data base results in changing the primary O-ring sealing timing function, and I discussed the STM-15 [Solid Rocket Motor 15 of flight 51-C, January 24, 19851 observations, namely, the [left rocket] motor had 80 degrees arc black grease between the O-rings, . . . black like coal . . . And [the right rocket motor] had 110 degree arc of black grease between the O-rings . . . ” I was asked to quantify my concerns, and 1 said I couldn’t, I couldn’t quantify it, I had no data to quantify it, but I did say I knew that it was away from goodness in the current data base. Someone . . . commented that we had soot blow-by by on SRM-22 [flight 61-A, October 30, 19851 which was launched at 75 degrees . . . I then said that SRM-15 [the flight at 53 degrees] had much more blow-by indication and that it was indeed telling us that lower temperature was a factor . . . I was asked again for data to support my claim, and I said I have none other than what is being presented, and I had been trying to get resilience data . . . since last October . . . ( R P C , p. 88-89, emphasis added).5 Jerry Burn, an engineer in the Applied Mechanics section of Morton- Thiokol, had visually inspected the disassembled joints of flight 22 (designated STS-61-A, launched on October 30, 1985) and described finding only “very spotted” soot beyond primary O-rings of two joints: . . . they were trying to compare apples and oranges, that [flight] 15 was so much worse than 22 . . . there really was some correlation in the fact that . . . 22 was warmer than 15 . . . [and] 22 was far less severe than 15. Whether you can take a correlation there or not . . . it is speculation but as far as I’m concerned it may very well have some correlation. . . .one comment was we’ve had soot blow-by at the 53 degrees as well as 75 degrees on 22, and we’ve had motors in between that didn’t have soot blow-by, so therefore we didn’t have a correlation. . . . They didn’t believe our data, and I agree, some of our stuff may be speculation but that’s all we have (Int.N.A., pp. 16-17, emphasis added). Larry Mulloy, Manager of the Solid Rocket Booster Project Office at NASA, most actively argued that Thiokol’s charted data and commentary did not support the Thiokol engineers’ conclusion. Later, in an exchange with General Kutyna, a member of the Presidential Commission, Mulloy revealed three aspects of his thinking: 1) a focus on relatively small variations in temperature, 2) an equal weighting of data from simulated, sub-scale tests with data from actual launchings, and 3) a failure to make any distinction between O-rings in field joints and O-rings in nozzle joints: General Kutyna: You said the temperature had little effect? Mr. Mulloy: I didn’t say that. I said I can’t get a correlation between O-ring erosion, blow-by and O-ring, and temperature. General Kutyna: 51-C was a pretty cool launch. That was January of last year. Mr. Mulloy: It was cold before then but it was not that much colder than other launches. General Kutyna: So it didn’t approximate this particular one? Mr. Mulloy: Unfortunately, that is one you look at and say, aha, is it related to a temperature gradient and the cold. The temperature of the O-ring on 51-C I believe was 53 degrees . . . We have fired motors at 48 degrees . . . [ RPC, vol. IV, pg. 290, emphasis addedI6 George Hardy, Deputy Director of Science and Engineering of NASA at Marshall, was the other most active NASA manager reacting to Thiokol’s presentations, including a comment that he was “appalled” at Thiokol’s recommendation. He. like others, focused on blow-by, ignoring erosion, and ignored both erosion ’The five-volume Report of the Presidential Commission on the Space Shuttle Challenger Accident, will be referred to as RPC, and page numbers are continuous in volumes four and five. Interviews of participants conducted by Presidential Commission staff members, copied with permission from the National Archives, Washington, D.C., are indicated by 1nt.N.A. See A. J . McDonald’s rebuttal at the time, in response to Mulloy’s use of a firing at 48 degrees ( R P C , IV: 741-742). Note also in that account the absence of any suggestion to examine paired temperature and damage data from other flights. LIGHTHALL: SPACE SHUTTLE CHALLENGER: DEFICIENCIES IN ANALYSIS OF ENGINEERING DATA and blow-by in nozzle joints: Mr. Hardy: . . . the temperature data was not conclusive. In fact, we had had blow-by on primary O-rings with joints at 75 degrees. So it was obviously not conclusive . . . that the temperature induced O-ring blow-by Mr. Hotz [a member of the Commission]: Mr. Hardy, we heard some testimony yesterday that the character of the damage in the low temperature blow-by and soot was of a much different nature and much more severe than any of your higher temperature experiences. Mr. Hardy: This is correct. Mr. Hotz: How did you evaluate that factor in reaching your conclusions? Mr. Hardy: . . . as some evidence of the fact the temperature did have an effect on the duration of blow-by. Z do not believe that temperature in and of itself induces the blow-by, and 1 think that is kind of obvious because we have occasions f o r blow-by at all temperatures . . . ( R P C , vol. V, pp. 861-862, emphasis added). Allan McDonald, Director of the Space Shuttle Solid Rocket Motor Project of Thiokol’s Space Division, contributed important comments just before the teleconference was interrupted for Thiokol’s caucus. McDonald described to a Commission interviewer the reasoning of Thiokol’s engineers five months earlier at NASA Headquarters, in their presentations regarding the severe O-ring erosion on flight STS-51-C, January 24, 1985: Mr. McDonald: We didn’t have any real data, but the presentation says that we know that durometer of the O-ring gets harder with temperature. . . . it got colder than we had seen before, therefore the O-ring got harder, and maybe that made it more difficult to seat . . . We had no hard data.’ (Int.N.A., p. 39-40, emphasis added). In the caucus at Thiokol’s plant in U a in which Mulloy’s th counter-arguments were examined, Jerald Mason, Senior Vice President at Thiokol in charge of all Wasatch plant Operations, initiated and guided the discussion. Mason had been persuaded by Mulloy’s counter-arguments that the temperature data were inconclusive, that sub-scale tests had shown O-rings sealing when they had three times the erosion experienced by O-rings in previous flights, and by McDonald’s comment that a certain testing procedure (the “leak check”) would have placed the secondary O-ring in a favorable position to extrude and seat. He, too, focused on blow-by, ignoring erosion data. Dr . Ride [a member of the Commission]: You know, what we’ve seen in the charts so far is that the data was inconclusive and so you said go ahead. Mr. Mason: . . . I hope I didn’t convey that. But the reasons for the discussion was the fact that we didn’t have enough data to quhntify the effect of the cold, and that was the heart of our discussion . . . we have had blow-by on earlier flights. We had not had any reason to believe that we couldn’t experience it again at any temperature . . ( R P C , vol. IV, p. 764, emphasis added). Arnold Thompson actually changed seats in the caucus room to diagram for Mason and the two other Thiokol managers (Joe ’By “real” or “hard” data, McDonald referred to “the fact that we had not conducted any hot fire tests or run any analysis that had isolated temperature by itself as a key contributor to O-ring erosion or blow-by” (personal communication). His eyes were turned, as were those of his colleagues, toward the laboratory, not toward the field, where the “real” variables were at work in their complex interaction. ---1 61 Kilminster and Calvin Wiggins) the effects of temperature. Later Thompson bemoaned the lack of sub-scale test data: . . . I think that had I had some more support, I would have had better data that Monday night to explain to the people what I was attempting to explain. (Int.N.A., pp. 56-57, emphasis added). Robert Lund, Vice President of Engineering at Thiokol, presented the charts on conclusions and recommendations. It fell to Lund, whose authority spanned both engineering and management, to give his judgment last, after Mason and two other managers had become persuaded by Mulloy’s counter-arguments against temperature as a variable in O-ring anomalies. Staff Interviewer: After the initial presentation was made to Marshall using the charts, . . . the conclusion was not to fly . . . Thirty minutes later you came to a point . . . where you said fly . . . Reconcile for me, if you will, how you got from one to the other. Mr. Lund: Okay. I think that chart there [the conclusions from the teleconference] pretty well says it. Number one, it was pointed out that the conclusion we had drawn that IOW temperature was, you know, an overwhelming factor was really not true. There was really not, it was not an overwhelming correlation that low temperature was causing blow-by, okay? . . . we have blow-by both at IOW temperature and high temperature. . . . So it is inconclusive . . . So I said, well, that’s right. The second thing, we . . . had some . . . data that was brought out, we have experienced erosion on 0rings, and we have also run lots of tests having erosion on O-rings where we really put a lot of erosion on [i.e., purposely cut material from] the O-ring and we could experience nearly a factor, somewhere between a factor of three and a factor of four more erosion on the O-ring and still have it work just fine, so we had a very large margin on erosion. If the erosion went up some . . . we still have a safety factor of fdur for the worst thing we had ever seen [referring to the cuts of material from O-rings in tests on small simulation motors without temperature as a variable]. (1nt.N.A. pp. 45-47, emphasis added). It should be clear from the foregoing sample of testimony, as it is overwhelmingly clear from the entire corpus of interviews, that the absence of systematic comparison of erosion data, blow-by data, afld temperature data reflected not a lapse of normal functioning but normal functioning itself, normal for NASA and Morton-Thiokol, and for managers, and engineers alike (all formally trained as engineers). Also clear is that participants focused on the two flights with black soot behind the primary O-ring, ignoring degrees of erosion as a variable. The tangible evidence of combustion is perhaps more dramatic than measures of erosion depth.* But a focus only on flights with soot behind an O-ring limited attention to two data points only, flights # 15 and 22, and ignored erosion as an indicator of vulnerability. To ignore measures of erosion (depth, length, arc) was to ignore data from the other 20 flights, data every bit as relevant to potential joint failure as sooty grease. Mackie [lo] and Einhom & Hogarth [5] note the power of the intrusive event and the abnormal, more than background conditions or normal rates and qualities of events, to be taken as causal. Both the failure to examine erosion data as well as the more meager blow-by data, and the failure of all participants to consider data from all 22 flights reflect the findings of psychological studies that the dramatic and atypical tend to distort judgments of cause-effect relationships. 68 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 38, NO. I , FEBRUARY 1991 Distribution o 0-Ring Temperatures f For All Flights (Temp’s in 5-degree groupings) Frequency 12r ~ 101 ’ 8 - Average ~ 68.4 - 10.8 “25 Std Dev. 6’ 4 - F l i g h t 51 - C 1 55 60 65 70 30 75 80 85 Tem p e r a t u re O n l y u p p e r b o u n d of t e m p b i n shown Fig. 1 . 0-Ring temperatures for all flights. QUANTITATIVE ANALYSES POSSIBLE THE EVEOF THE ON CHALLENGER LAUNCH To assess the degree of absence of relevant analytical schemas it is necessary to show what their presence would have looked like. What number and kind of analyses would have been proper and possible at the time the decision was being made, given relatively elementary understanding of statistical concepts and methods? Then, beyond an elementary level of statistical analysis, what further light could have been thrown on the arguments by concepts and methods at a more intermediate level of knowledge? Assume, then, the presence of statistically oriented engineers and managers who possess contemporarily available statistical software. By the time the January 27 cold temperatures were announced for the Challenger launch, these members could easily have had in hand some revealing statistical analyses. The Field Joints: All participants could have had available the temperature and the degrees of damage to O-rings on all 22 flights for which data were available: 22 pairs of data. One of those flights, 5 1-C, the fifteenth, had drawn immediate attention by its unusually severe degree of damage and its unusual O-ring temperature. A first quantification appropriate to the question of “normal” functioning would be computation of the means and dispersions (standard deviations) of temperature and of degrees of damage. How far away from average temperature and aver- age damage was this flight a year earlier, and how far from that average temperature were the Challenger’s O-rings predicted to be? These are important questions in any argument in which confidence in the known and caution in face of the unknown is at issue. If a new danger has arisen, it is important to break up tendencies to think that conditions are really unchanged, that business can proceed as usual, that the case at hand is like all the others. So we array all available temperatures and degrees of damage in order to force quantitative, visual attention to what is, and what is not, usual. Statistically oriented engineers could have argued that the Challenger was about to be launched far outside engineer’s knowledge base.’ How far? Fig. 1 shows the distribution of all launch temperatures that could have been available at the time of the teleconference the day before the launch. The temperature of the Challenger’s O-rings predicted for launch time (29 O F ) stands isolated in the lower tail of the distribution, 3.6 standard deviations below the average temperature, 68.4 OF.-a starkly deviant condition. In other words, they could have argued, to the extent that temperature is a factor, not necessarily the factor but a factor in causing erosion, the launch temperature about to be Boisjoly and Thompson had argued successfully at first that the impending launch’s expected O-ring temperature took them as Boisjoly put it, “outside of our data base.” But it occurred to no one, apparently, to ask or show how far outside that base 29 degrees was. I I 69 LIGHTHALL: SPACE SHUTTLE CHALLENGER: DEFICIENCIES IN ANALYSIS OF ENGINEERING DATA Distribution o Total Erosion Depth f For 22 Flights (in 1,000ths of a n inch) Frequency 20- _ _ _ _ ~ ___ - I _____ ~~ _____ ~~ - These v a l u e s a l l zero 15 I 10 I I I I n v 5 10 15 20 25 30 35 40 45 50 55 Sum of Maximum Erosion Depth O n l y u p p e r b o u n d o e r o s i o n b i n shown f Fig. 2. Erosion depth for 22 flights. experienced by Challenger’s O-rings is both dangerous and far beyond the range of our knowledge of the boosters under normal conditions. If temperature is a casual factor, the extreme atypicality of the expected O-ring temperature of the Challenger may render current modeling or understanding of field joints under normal conditions simply inapplicable. Reasoning by statistically oriented members could easily have proceeded thus: We cannot know the severity of erosion for 29 degrees, but we do know the comparative damage of the flight most like the one we face with respect to temperature, namely, STS-51-C. It is like the impending launch in one striking way: the degree to which it’s O-ring temperatures were below average. Fig. 1 also shows how atypical STS-51-C’s O-ring temperature was in relation to all other flights for which complete data were available. The position of 51-C in Fig. 1 (3.4 standard deviations below the mean calculated without the temperature of the impending flight 51-L) is also extremely atypical. How much erosion, then, did 51-C suffer in relation to all other flights for which data could have been available? Fig. 2 located 51-C again distant from most other flights in degree of erosion. Under the abnormal temperature of 51-C’s launch, an abnormal amount (.048-inch depth) of erosion occurred. Was it mere coincidence that those two improbable characteristics, lowest temperature and deep erosion, occurred on the same flight? Fig. 2 shows, however, that 51-C was not the only flight at the high end of the distribution of erosion, nor was its erosion the deepest. Does the absence of a perfect ordering of temperature and erosion negate a causal connection between them? -1 Statistically oriented members would argue no, that temperature could be, and likely was, one important variable among others, and that the simultaneous effect of many variables would result in a trend of a relation across all 22 data points even if a few temperatures and degrees of erosion showed up out of order. They could easily then present the graph of Fig. 3, which plots the linear relation between O-ring temperature and depth of field joint erosion. lo An appropriate index of relationship between two variables with skewed distributions indicates a non-random connection between O-ring temperature and depth of erosion: Kendall tau-B (corrected for ties) = - .54, Chi Squared = 11.0, df = 3, p < .005.” Physical theory relates temperature and the responsive action of elastomers in the same way: Cold causes lo Blow-by (presence-absence of some sooted grease) occurred in field joints on only two flights, #’s 15 and 22 (51-C and 61-A); in nozzle joints on five flights. Chi squared analyses of nozzle blow-by showed a marginal relation to depth of erosion on nozzle O-rings (X2 = 5.33, df = 2, p = .07, Kendall’s Tau .39), and no reliable relation to leak check pressure, to temperature, or to field joint blow-by. The sum of instances of blow-by across joints within flights shows a strong relationship to depth of erosion across joints within flights (X2 = 8.25, df = 1, p < ,005, Kendall’s Tau = .61). These results are produced by a contingency table of two levels of erosion (zero and > zero) against three levels of temperature (equal to or <60, 60-70, and >70). The abnormal temperature of 53 degrees should not be considered a random “outlier.” It is a single temperature in just that range of temperatures precisely at issue, sampling more than any other flight the at-issue conditions to be experienced by the impending Challenger launch at 29 degrees. The magnitudes of linear correlation between temperature and both length and volume of O-ring erosion were similar. + + 70 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 38, NO. 1 , FEBRUARY 1991 TABLE I TEMPERATURES, PREDICTED AND ACTUAL DEPTHS (INCHES) FIELD OF JOINTEROSION, AND UPPER % CONFIDENCE 95 LIMITS PREDICTED OF OF FOR SHUTTLE FLIGHTS DEPTHS EROSION 22 SPACE 1 I ! ! ; I I Maximum Erosion Depth Summed For Each Flight 1 .os+ p < .GO5 = - . 5 4 Tau-B : t + 1 ~ .02+ I j .01+ , I 1 .00+ 1 3 1 1 2 11- 1 _ _ _ _ _ +-------+-------+-------+-------+-------+-------*-------~ 50.0 55.0 60.0 65.0 70.0 75.0 80.0 85.0 PROJECTED O-RING TEMPERATURE A T LAUNCH Fig. 3. Linear correlation between O-ring temperature and summed maximum depths (inches) of O-ring erosion for each flight for which complete data were available on January 27, 1986. Frequencies of observations at each bi-variate point in graph are indicated by namber. hardness, hardness slows responsiveness, and slow response allows more time for hot gases to surge by and erode O-rings. Therefore, temperature is linked to O-ring erosion both by physical theory and by observation of O-ring performance under real-world (not laboratory) conditions. Since temperature can be shown to be related to O-ring erosion, the extreme atypicality of the Challenger’s predicted temperature means that the impending launch would move into a causal environment that (a) held many more unknowns except that it (b) tends to bring greater O-ring erosion. The importance of O-ring temperature as a causal factor together with the extremely deviant O-ring temperature expected for the Challenger launch render questionable or irrelevant the safety margin (the “factor of three”) that gave Mason, Lund, and other managers comfort. That margin had been estimated from subscale bench tests, conducted at room temperature, and had never been tested under varying temperatures. Quantification could then proceed further, addressing the extent of erosion likely on the Challenger. If one can compute Pearson’s r between temperature and depth of erosion for 22 flights (r = - .56), one can also compute an estimate of depth of erosion expected at any given temperature. The O-rings of the Challenger were expected to be 29 degrees at launch time: What corresponding depth of erosion would an r of - .56 predict for that temperature? Table I presents predicted and actual values of erosion depth for all flights of the shuttle for which data were available, along with temperatures and probability estimates of upper confidence limits for the predicted depth of erosion.’* The middle column of Table I shows clearly the four flights whose O-rings experienced measurable depth of erosion. Notice the values in the last row of the last two columns of this table: Predicted depth of erosion for the Challenger (.066) and the upper limit of likely erosion depth (. 107). A statistically oriented member would point out that the .066 predicted depth is a little over twice the depth predicted for the worst previous case of erosion, flight 15 (.032). Since the crude straight-line relation depicted in Fig. 3 quite under-predicts erosion for flight 15, it might be seriously under-predicting erosion for flight 25. Statistically oriented members could further point out that the upper limit of erosion depth predicted for the Challenger (.107) ex- ’’ Indices of length and volume of O-ring erosion show similar relation- ships. night No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 51-C 16 17 18 19 20 21 22 23 2551-L ’ Temperature of O-rings 66 70 69 80 68 61 12 73 70 57 63 70 78 67 53 67 75 70 81 76 79 75 76 29 Actual Depth of Erosion (in thousandths) Predicted Depth of Erosion (in thousandths) 0 53 0 0 0 0 0 0 40 0 28 0 0 48 0 0 0 0 0 0 0 0 - - 13 8 9 6 11 12 5 4 8 26 18 8 4 12 32 12 1 8 8 1 - 5 1 - 1 66 Upper 95 % Confidence Limit (in thousandths) 21 14 16 5 17 19 12 11 14 41 27 14 7 19 50 19 9 14 5 8 6 9 8 107 ‘Data from Flight 24 are omitted because they would not have been available for analysis by the eve of the Challenger launch. ceeds the margin of safety that engineers and managers had set for erosion depth, the “factor of three” of .092. The possibility of exceeding even that factor-of-three margin would tend, if these data and analyses were believed, to give pause to anyone favoring launch on ~ c h e d u l e , ’ ~ given the expected O-ring temperature of 29 degrees. Statistically oriented members might strengthen a tendency to delay the launch by pointing out that the simple estimate of linear relation between temperature and erosion depth is too gross and under-predicts depth of erosion at 29 degrees. The relationship in question must be curved, since the (statistically significant) linear relations predicts negative erosion at higher temperatures. A best fitting curve for these data would (a) eliminate predictions of the physically impossible negative erosion and (b) predict much deeper erosion at 29 “F. The Nozzle Joints: Statistically oriented engineers at Thiokol or NASA would have had sufficient time to discover that erosion of O-rings in nozzle joints was a function not of temperature but of other factors. While the correlation (r) between O-ring erosion in field joints and temperature is - .56, the same index of that relationship for nozzle O-rings is .18, or effectively zero. While temperature threatens O-rings of field joints, therefore, it is unlikely to threaten O-rings in nozzle joints. That O-rings in these two kinds of joint are quite different with respect to causes of erosion is reflected in the fact that the correlation between erosion depth in field joints and in nozzle joints is also nil (r = - .06). Knowing the degree of field joint erosion of a given flight, therefore, tells us nothing about that flight’s depth of erosion in its nozzle joint. ’ ’ Primary and secondary alternative launch “windows” had already been planned for and were available, two on January 29 and two others on January 30. A delay for warmer temperatures was therefore possible. 71 LIGHTHALL: SPACE SHUTTLE CHALLENGER: DEFICIENCIES IN ANALYSIS OF ENGINEERING DATA The implication of the pattern of three correlations is simple and stark: The causes of O-ring erosion in the two kinds of joint differ in general, but also most particularly with respect to temperature. Of the seven flights with O-rings at 75 degrees or higher, all but one of them (flight # 17) suffered O-ring erosion only in nozzle joints: Those six flights experienced no erosion in their field joints. Thus, the failure of all participants to distinguish evidence from the two kinds of field joints may have led them to confuse irrelevant evidence with relevant. More important, perhaps, they were not led to ask, “If temperature is not a factor in the more frequent incidence of nozzle erosion, then what are those different causes of erosion in nozzles?” Engineers at Thiokol most responsible for analyzing O-ring performance were aware that by late April 1985, specifically after the seventeenth shuttle (STS-51-B), O-rings showed increasing anomalies. As Thompson put it, “I really don’t know what the correlation was [between various changes in rocket motor preparation and O-ring erosion]. I just knew the incidence was rising, and that the nozzle events were worse” (1nt.N.A. p. 40). When Boisjoly was asked after the accident if he could account for an apparent increase in erosion and blow-by after about eight flights, he replied: “That has been a topic of much discussion over the years, and there was a putty change . . . we have been struggling with that because the data appears just totally random. If you look at the degree locations. . . . at the temperatures, . . . it is random with regards to erosion per se” ( R P C , p. 798). According to Thompson, someone did apply some statistical analysis to the frequencies of O-ring anomalies: . . . we had people do statistical analysis of the data to try and determine like the frequency of the field joints versus frequency of the nozzles . . . it showed the nozzle on a frequency basis was a much higher incidence . . . we were doing some statistical work at that time trying to gather the data and try and understand it clearly” (Int.N.A., p. 38). While differences in frequency may have been subjected to statistical treatment, the testimony and documentary evidence on all sides offer clear evidence that no statistical analysis was made of the relationship between O-ring erosion in either joint and any other variable. Members had become aware at various points, then, that the incidence and severity of erosion in nozzle O-rings had risen, particularly after STS-17 at the end of April 1985. McIntosh reported that speculation about causes of erosion in the nozzles became focused, and he was far more confident than Boisjoly about having reasoned out a likely cause: “We looked mainly at the leak check as the culprit” (Int.N.A., p. 77). We will examine the “leak check” presently. Given the foregoing awareness, statistically oriented members could have, and likely would have, explored how depth of erosion covaried with other variables. One variable immediately suggested by the foregoing awarenesses is simply flight number: As flight number increased did depth of erosion of O-rings increase in either field joints or nozzle joints? A first approximation of an answer is provided by arrays of basic data in Table 11. Perusal of the first three columns shows no progression of erosion in field joints corresponding to flight number, but a marked increase is evident in nozzle erosion after flight 9. A scan of the last column reveals at least some basis for McIntosh’s focus on the leak check as “the culprit. ” The O-rings of each rocket’s joints were tested before each flight for their capacity to hold a seal. Each joint had a valve 1 TABLE 1 DEPTH EROSION OF (INCHES) AND LEAK CHECK PRESSUREFIELD IN JOINTS AND NOZZLE JOINTS 22 SPACE FOR SHUTTLE FLIGHTS Flight No. Field Joint Erosion Depth (in thousandths)’ Nozzle Joint Erosion Depth’ (in thousandths) Field Joint Leak Check psi Nozzle Joint Leak Check psi 0 0 0 0 50 50 50 50 0 0 0 0 0 40 0 38 0 0 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 53 0 0 48 0 0 0 0 0 0 0 0 0 0 39 34 46 0 0 0 19 208 45 0 94 0 75 56 50 50 50 50 50 100 100 200 200 200 200 200 200 200 200 200 200 200 200 200 200 50 50 50 50 100 100 100 100 100 100 100 200 100 200 200 200 200 200 200 ‘Depth is summed over all sites of erosion within each flight’s rockets within type of joint. through which nitrogen was blown, much like inflating a tire. The gas was blown through the valve into the space between the primary and secondary O-rings. At about the time of flight 7 engineers discovered that the rocket’s normal insulating putty, between the O-rings and the solid fuel, could withstand more than 50 psi pressure by itself. At only 50-lbs pressure, the test gas could blow by a faulty primary O-ring seal, enter the space between the primary O-ring and the putty, and be contained by the putty itself. The technician would be thus led mistakenly to interpret the gauge pressure reading as a solid seal of O-rings, since he or she would not be aware that the putty, not the faulty O-ring, was holding the pressure. To correct for the putty’s capacity to mislead, the pressure was changed, first to 100 psi and then to 200 psi when it was discovered that occasionally putty would hold at 150 psi. The change was instituted first in field joints, from flight 10 onward, then in nozzle joints.14 The increased pressure brought side affects, however: blowing nitrogen at 200 psi into the joint caused the gas to penetrate the putty through “blow holes.” Once created, these blow holes then became ready conduits through which, later, the hot gases of the rocket at ignition could immediately flow outward toward the O-rings. The blow holes focused the hot gases at concentrated points, thus turning the blow holes into small blow torches for milliseconds before the O-rings sealed. Despite suspicions that the leak check might be “the culprit, ’ ’ they remained suspicions only. Statistically oriented engineers faced with the data of Table I1 would have run quick tests to see whether the pattern made by the third, fourth and fifth columns (nozzle erosion versus leak l4 Neither interviews nor the extensive detail of the five-volume report explains why nozzle tests were shifted to 200 psi consistently after flight 17 while 200 psi was used in the field joints consistently after flight 9. IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 38, NO. 1, FEBRUARY 1991 72 checks) were as “random” as Roger Boisjoly believed them to be. The linear correlation (r) between depth of erosion and flight 1 number for the data in Table 1 is - .22 for field joints (well within chance variation) and .48 for nozzle joints (expected by So chance alone only 25 times in a th~usand).’~ McIntosh’s culprit is confirmed in the case of nozzle joints, but not for field joints. So Boisjoly’s sense of a random pattern of leak check effects is confirmed for field joints, but not for nozzle joints. These results strengthen evidence that erosion in the two kinds of joint was produced by different causes. Apparently, a tacit, impressionistic pooling of observations from the two different joints led many NASA and Thiokol members, not having gathered the above evidence about the two joints’ differential sensitivities, to see merely randomness where a systematic pattern is to be found (leak check effects in nozzle joints), and actually to disconfirm a valid hypothesis (temperature as a cause in field joints). l6 One pattern is very clear: With 50 lb of pressure, there was no measurable erosion in nozzle joints, and soon after the first change in pressure the first nozzle erosion appeared.I7 The connection these simple analyses make between nozzle erosion depth and leak check procedures poses a dilemma which, if faced by NASA and Thiokol managers, could easily have resulted in suspension of flights for redesign of joint, O-ring, and putty configurations. On one hand, the nitrogen test pressure had to be 150 psi or greater to be strong enough to breach the insulating putty. When gauges showed that pressure was being contained, engineers had to know it was the O-rings that were holding the pressure, not the putty. But on the other hand, the very breaching of the putty necessary to allow a valid test of the sealing capacity of the O-rings also provided channels through which the flow of hot gases could be concentrated, thus constituting a hazard. Ironically, then, the increase in pressure designed to provide a more valid, secure safety check could be the means by which hot gas became concentrated with power to erode the O-rings. No such dilemma was likely to be posed forcefully to the managers, however, without quantitative connections of nozzle erosion to leak check pressure. Statistically oriented members could easily have connected nozzle erosion (A) with the progression of flights (B). Searching for what it was about the progression of flights that might be related to O-ring performance, they could easily test the connection between flight number (B) and nozzle leak check pressure (C). It would then be only a matter of normal curiosity to examine the relation of A to C, nozzle + This positive relation appears also by Chi squared test of nozzle erosion, cross classifying extent of erosion (zero and greater than zero) with psi leak check pressure for nozzle joints. 16 Explicit pooling of depth of erosion from field joints and nozzle joints within flights completely washes out any effect of temperature ( r = - .Ol), and yields a correlation of +.46 with field joint leak check pressure (P$ ,025). Chi squared comparison of erosion depth for nozzle pressure of 100 psi versus 200, omitting flights with 50 psi’s, with a near-median break on nozzle depth of above and below ,040yields a Chi squared value of 3.23, df = 1, p <: .lo. Table I1 focuses on the shift in leak check pressure only. But other changes were made-e.g., in type of putt-at about the same time. Engineers statistically oriented enough to array the data of Table I1 would also have tested hypotheses regarding these other changes. But the major point here is not that the leak check was the culprit. Rather, it is that one or more “culprits” were at work rendering laterflights more dangerous, not less dangerous, than earlierflights. The import of this fact is that, unwittingly, a change was made somewhere in flights 7, 8, or 9 that were in the wrong direction of safety, and that fact, itself, was proof that dangerrelated properties of nozzle O-rings were not only not understood, but were misunderstood. False theories of the nozzle joint were afoot. erosion to nozzle leak check pressure. But note: these relationships would have been obscured unless the data for nozzle erosion and for field joint erosion had been kept separate. PREVENTIVE KNOWLEDGE The foregoing presents analyses that statistically oriented engineers might have introduced the day before the Challenger launch. They could have quantified (a) the degree to which the Challenger O-ring temperature was atypical, beyond the range of members’ knowledge of the rockets’ normal functioning; (b) the direction of strength of relationship between temperature and field joint erosion; and (c) the extent of possible erosion in the Challenger’s field joints, where the upper limits of likely erosion exceeded the safety “factor of three.” But what might they have known earlier, with fewer flights and therefore less data? What early warnings might they have provided, with time for discussion, deliberation, and confirmation? Enough worrisome erosion had occurred by flight 17 to have mobilized statistically oriented engineers. Could they have detected any systematic patterns by the eve of flight 18, six months earlier? Specifically, could statistically oriented members have had evidence before flight 18 that field joint erosion depth could be predicted by temperature? Yes. Not only would depth of field joint erosion have shown up related to temperature (Kendall’s Tau = -0.56, Chi Squared = 7.47, df = 2, p < 0.025), but erosion in nozzle joints shows up as unrelated either to temperature (Kendall’s Tau = - 0.13) or to erosion depth in field joints (Kendall’s Tau = 0.23). These data reveal a pattern of relations that would have been confirmed for every flight thereafter, flight by flight. The suggestion in these data that erosion in the two kinds of joint might be caused by quite different factors is confirmed in other data available just before flight 18. Erosion in nozzles, but not in field joints, is related to leak check pressure used to test both nozzle joints and field joints. Simple cross tabulations of above and below zero on erosion and at the three levels of pressure show that the pattern of no erosion appearing until the shift to a higher pressure would happen by chance alone fewer than five times in a hundred.18 But the same kind of analysis reveals that the coincidence of field joint erosion and field joint leak check pressure was random, occurring by chance alone 46 times in a hundred. AND DISCUSSION CONCLUSIONS Seven conclusions are warranted from the foregoing analyses. First, while a strong norm of quantification operated at both NASA and Thiokol and was accepted by all participants, many admitted explicitly that they could not quantify the relation suspected between O-ring temperature and O-ring erosion. Second, focus of all parties was clearly upon that specific relation, temperature and O-ring malfunction, not merely on the question of temperature and O-ring hardness (a relation all accepted as strong and negative). The issue in explicit focus, therefore, was one precisely implicated in causing the accident. Third, the data presented in chart form during the teleconference were essentially irrelevant to the casual question that Thiokol engineers themselves were attempting to answer. At no point did the charts bring into juxtaposition paired data on temperature and erosion ’* Chi squared for zero-above zero nozzle erosion depth cross classified with 50, 100, and 200 psi pressure in the nozzle joints is 6.69, df = 2, p < .05; Kendall’s tau-B = .62. + LIGHTHALL: SPACE SHUTTLE CHALLENGER: DEFICIENCIES IN ANALYSIS OF ENGINEERING DATA for shuttle flights. In this regard, NASA managers were justified in concluding that Thiokol’s charted data did not support Thiokol’s conclusion that temperature was a factor. Fourth, the data that were presented in the charts, however, clearly demonstrate that detailed data on erosion and temperature for each flight could have been presented on the eve of the Challenger launch. The absence of such data and of analyses of the relationship in question, and the kinds of explanations given by participants after the accident, warrant a fifth conclusion: that none of the participants had ever learned, or had long since forgotten, elementary ideas and methods of statistical analysis and inference. Sixth, simple analyses of variation and of co-variation show not only that such analyses were possible but also that they would have quantified a strong, negative relationship between the two variables whose relationship was not quantified and in the end was doubted and discounted. Specifically, such analyses were capable of showing 1) that to pool erosion data for the two types of joint would confound two different causal systems; 2) that O-ring erosion in field joints was indeed sensitive to temperature, but not to leak check pressure, and that O-ring erosion in nozzle joints was not sensitive to temperature, but was sensitive to leak check pressure; 3) that the launch temperatures of both 5 1-C and 5 1-L were extremely atypical, rendering questionable the safety “factor of three” computed for room temperature only; and 4) that amounts of erosion predictable for the Challenger flight were severe and might reach or exceed even the margin of three that managers held as safe. Seventh and finally, these results could have been reported not only on the eve of the Challenger launch, but also six months earlier, with data from the first 17 flights. It might be argued that the Challenger accident was merely another instance of what Perrow [15] has called a “normal” accident. Normal accidents result from a great complexity of technological systems (nuclear reactors or booster rocket systems) that renders normal the interactions of many parts and subsystems that lead, sooner or later, to “unexpected” and “mysterious” failure from rare concatenations of events. Surely the booster rockets comprised hundreds of minute parts and conditions which, in peculiar combinations, could lead the whole system to malfunction. But Perrow [15] makes a clear distinction between “component” failure accidents and “system” failure (normal) accidents. Component failures are those in which “any interaction of two or more failures is anticipated, expected, or comprehensible to the persons who designed the system, and those who are adequately trained to operate it” (p. 70-71). The full report of the Presidential Commission points to factors, like ice in the joints and high humidity interacting with putty, that were not understood before the accident. But the number and complexity of the factors finally evident as causal were few and clearly subject to mastery and correction.’’ Finally, we must raise the question why none of the participants asked for or presented the kind of analyses shown in this paper to have been possible and crucially relevant to the very 19The report of the “STS 51-L Data & Design Analysis Task Force Accident Analysis Team Solid Rocket Motor Workmg Group” (RPC, pp. L-50-L-240) narrows its causal factors to five. Examination of its data and reasoning, however, shows that temperature swamps all other variables in the temperature range in which the Challenger was launched. See esp. Fig. 29, p, L-82, Vol. II of RPC. That report makes it just as clear, however, that with slightly higher temperatures much more complicated interactions could have caused an accident which then would have been much more baffling and beyond any expectation that engineers or managers could have understood it. 73 questions they were pointedly examining. To raise the question is to direct our attention away from rocket joints and launch decisions, back much earlier in the careers of the participants: to their professional training and, specifically, to the participants’ engineering curriculum. So uniform was the participants’ absence of elementary ideas and methods of statistical analysis that we must consider the likelihood that that uniformity came from a single source, one common to all decision participants, and the likelihood that that source was their professional training as engineers. Perusal of current courses of study of several engineering programs and conversations with engineering educators involved in engineering curricular reforms provides uniform support (admittedly from incomplete and non-random sampling) for the hypothesis that the specialties of engineering represented among these two dozen decision participants either did not include even elementary ideas and methods of covariance analysis (e.g., linear correlations, multiple regression) or included them with such weak emphasis or weak instruction that they made no impression. I propose, then, that these failures of thought and perception were not from a lack of sophisticated expertise but from lack of simple, elementary understandings and methods. Others argue for more sophistication in experimental design [14] and in Bayesian approaches to risk assessment [3], [4].20 No doubt more experts are needed. But identified here was a lack of an elementary, working knowledge of descriptive and inferential statistics of covarying relationships between variables of the kind one gains in a semester course with laboratory practice. The solution to this circumscribed problem in the future is simple to state but no doubt would be difficult to implement: a semester or two-quarter course and integrated laboratory in computing and interpreting descriptive and inferential statistics, regarding covariation between and among variables, including simple, partial, and multiple correlations and multiple regression. 21 If my hypothesis is correct that the flaw of cause-effect reasoning I have shown in the decision process was due to a curricular or instructional deficiency, due to the exclusion of *‘A caution must be advanced regarding statistical sophistication: The more sophisticated the analytical technique, the fewer persons who will understand or master it, requiring the participation of statistical experts in decision processes such as the one under review here. But that does not address the problem of integrating statistical thinking into decision processes because it is managers in these organizations, not statisticians, who decide what the statisticians will work on. Morton-Thiokol did have a statistical expert who had worked on modeling of the field joints’ functioning in relation to simplified data. But at no time was he asked to address the simple, direct question of estimating the extent or probability of damage to O-rings as a function of temperature drawn from actual launch-time data. If working level engineers and managers do not have the statistical ideas and methods necessary to ask the right questions, even the most sophisticated statistical experts in their organizations will not provide answers. At issue in the flawed segment of organizational functioning here is not high sophistication -not the kind Dalal, et al. report to fellow statisticians in the Journal of the American Statistical Association [4]. At issue, rather, is an understanding of simple, basic ideas and methods of analyzing probable associations of causal conditions and effects under complex field conditions in order that more relevant and needed questions might be asked by both working engineers and their more removed, but organizationally more powerful, managers. The lack of a statistical orientation at NASA and Morton-Thiokol was far greater than Penzias [14]apparently believes to be true for engineering generally. More importantly, since it is the functioning of the complex product in its own environment, not in the laboratory, that is crucially at issue, the statistics to be emphasized are not those dealing with controlled experiments but those allowing exploration of the complicated effects of factors in the product’s ‘‘live’’ environment: multivariate, correlational concepts and methods. *’ 14 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 38, NO. 1, FEBRUARY 1991 covariance analysis from engineering curricula (or weakness of it in instruction), it would not be an isolated case of professional narrowness. It would be but a single instance of a much more pervasive pattern in the production and transmission of knowledge in industrial societies: the pattern of ever-increasing specialization of research, teaching, academic degrees, and professional certification. It is a pattern that gives a privileged place to analysis and to work confined to circumscribed problems calling for circumscribed patterns of perception and thought. This pattern gives a decidedly underprivileged place to synthesis and to work and research that crosses the boundaries of specialization. By directing our attention to a pervasive pattern of research and teaching in our society, the Challenger accident may have a much more important lesson to teach than that engineers and engineering managers need to know and to use better statistical analyses. Of greater importance may be that in a world where causes and events operate interdependently, as complex wholes, a highly technological society is vulnerable that gives unmitigated privilege in its research, teaching, and professional development to specialization and analysis over integration and synthesis. Professional narrowness in this case led to dramatic, fatal failure. In a world of nuclear reactors and nuclear arguments professional myopias of the kind revealed here may leave us ruinously vulnerable to the normal tendencies of managers to prove themselves and to produce results. Complex technologies like space shuttles require capacities to analyze complex realities, including hidden cause-effect relations, analytical capacities that must transcend narrow specialization. H. J. Einhorn and R. M. Hogarth, “Judging probable cause,” Psychological Bulletin, vol. 99, pp. 3-19, 1986. R. P. Feynman, What Do You Care What Other People Think? New York: W. W. Norton, 1988, pp. 113-237. R. Jackall, Moral Mazes: The World of Corporate Managers. New York: Oxford University Press, 1988. F. F. Lighthall, Local Realities, Local Adaptations. London: Falmer Press, 1989 R. Lund, Commission Interview Transcript, April 1 , 1986, National Archives, Washington, DC, 1986. J. L. Mackie, The Cement of the Universe: A Study of Causation. Oxford, England: Clarendon Press, 1974. M. McConnell, Challenger: A Major Malfunction. Garden City, NY: Doubleday, 1987. A. J. McDonald, Commission Interview Transcript, March 19, 1986, National Archives, Washington, DC, 1986. H. McIntosh, Commission Interview Transcript, April 2, 1986, National Archives, Washington, DC, 1986. A. Penzias, “Teaching statistics to engineers,” Science, vol. 244, p. 1025, June 2, 1989. C. Perrow, Normal Accidents: Living With High-Risk Technologies. New York: Basic Brooks, 1984. Report of the Presidential Commission on the Space Shuttle Challenger Accident, June 6 , 1986. A. Thompson, Commission Interview Transcript, April 4 , 1986, National Archives, Washington, DC, 1986. Frederick F. Lighthall was born in Stamford, CT on November 6 , 1930. He received the A.B. degree in history from Oberlin College, Oberlin, OH, and the M.A. and Ph.D. degrees ACKNOWLEDGMENT in educational psychology from Yale University, New Haven, CT. I am deeply grateful to Don R. Swanson, Larry V. Hedges, Early professional positions include Internal Roger M. Boisjoly, and Allan J. McDonald for their thoughtful Auditor in a manufacturing concern and Elecomments on earlier versions of this paper and to William M. mentary School Teacher. He is a member of the faculty of the University of Chicago in the Goldstein, Thomas R. Trabasso, and Robin M. Hogarth for Departments of Education and Psychology, and insightful discussions during its preparation. he chaired the Educational Psychology Faculty from 1973 and 1979. At present, he is studying the organizational processes leading up to the REFERENCES decision to launch the Challenger, drawing on published and unpubJ. Bum, Commission Interview Transcript, March 25, 1986, lished testimony in the National Archives, and upon recent interviews and correspondence. His research interests include organizational probNational Archives, Washington, DC, 1986. R. P. Boisjoly, E. F. Curtis, and E. Mellican, “Roger Boisjoly lem solving and conflict management, processes of effective organizaand the Challenger disaster: The ethical dimensions,” Journal of tional change, and discussion processes in the classroom. His most recent related publications include Local Realities, Local Business Ethics, vol. 8, pp. 217-230, 1989. Committee on Shuttle Criticality Review and Hazard Analysis Adaptations: Problem, Process, and Person in a School’s GoverAudit, Post-Challenger Evaluation of Space Shuttle Risk As- nance (Falmer Press, 1989), and “Making and Transcending Local sessment and Management. Washington, DC: National Academy Adaptations: A Pragmatic Constructivist Perspective on Participation, ‘ ‘ in Research in the Sociology of Organizations, (vol. 7, JAI Press, of Sciences Press, 1988. S . R. Dalal, E. B. Fowlkes, and B. Hoadley, “Risk analysis of 1988, S . B. Bacharach and R. Magjuka, Eds.). Mr. Lighthall is a Life Member of the American Association for the the space shuttle: Pre-Challenger prediction of failure,” Journal of the American Statistical Association, vol. 84, pp. 945-957, Advancement of Science and a member of thc American Psychological Association and the American Sociological Association. 1989. ...
View Full Document

This note was uploaded on 10/09/2009 for the course ENSC 1146 taught by Professor Michaelsjoerdsma during the Spring '07 term at Simon Fraser.

Page1 / 12

LaunchingTheSpaceShuttleChallenger - IEEE TRANSACTIONS ON...

This preview shows document pages 1 - 2. Sign up to view the full document.

View Full Document Right Arrow Icon
Ask a homework question - tutors are online