It is true that if the probability of failure was as low as 1 in 100,000 it would take an inordinate number of tests to determine it ( you would get nothing but a string of perfect flights from which no precise figure, other than that the probability is likely less than the number of such flights in the string so far). But, if the real probability is not so small, flights would show troubles, near failures, and possible actual failures with a reasonable number of trials. and standard statistical methods could give a reasonable estimate. In fact, previous NASA experience had shown, on occasion, just such difficulties, near accidents, and accidents, all giving warning that the probability of flight failure was not so very small. The inconsistency of the argument not to determine reliability through historical experience, as the range safety officer did, is that NASA also appeals to history, beginning "Historically this high degree of mission success..."Distressingly, this appears to be exactly what happened with the Columbia. Foam had come off the shuttle before, but never with disastrous results; NASA accordingly seems to have decided that it must therefore be safe to have the insulation break free. This heuristic was probably the best we could do as East African Plains Apes. In the modern world, however, we have better substitutes, like reason, if we'll only use them.
Finally, if we are to replace standard numerical probability usage with engineering judgment, why do we find such an enormous disparity between the management estimate and the judgment of the engineers? It would appear that, for whatever purpose, be it for internal or external consumption, the management of NASA exaggerates the reliability of its product, to the point of fantasy.
The history of the certification and Flight Readiness Reviews will not be repeated here. (See other part of Commission reports.) The phenomenon of accepting for flight, seals that had shown erosion and blow-by in previous flights, is very clear. The Challenger flight is
an excellent example. There are several references to flights that had gone before. The acceptance and success of these flights is taken as evidence of safety. But erosion and blow-by are not what the design expected. They are warnings that something is wrong. The equipment is not operating as expected, and therefore there is a danger that it can
operate with even wider deviations in this unexpected and not thoroughly understood way. The fact that this danger did not lead to a catastrophe before is no guarantee that it will not the next time, unless it is completely understood. When playing Russian roulette the
fact that the first shot got off safely is little comfort for the next. The origin and consequences of the erosion and blow-by were not understood. They did not occur equally on all flights and all joints; sometimes more, and sometimes less. Why not sometime, when whatever conditions determined it were right, still more leading to catastrophe?
In spite of these variations from case to case, officials behaved as if they understood it, giving apparently logical arguments to each other often depending on the "success" of previous flights. For example. in determining if flight 51-L was safe to fly in the face of
ring erosion in flight 51-C, it was noted that the erosion depth was only one-third of the radius. It had been noted in an experiment cutting the ring that cutting it as deep as one radius was necessary before the ring failed. Instead of being very concerned that
variations of poorly understood conditions might reasonably create a deeper erosion this time, it was asserted, there was "a safety factor of three." This is a strange use of the engineer's term ,"safety factor." If a bridge is built to withstand a certain load without the
beams permanently deforming, cracking, or breaking, it may be designed for the materials used to actually stand up under three times the load. This "safety factor" is to allow for uncertain excesses of load, or unknown extra loads, or weaknesses in the material that might have unexpected flaws, etc. If now the expected load comes on to the new bridge and a crack appears in a beam, this is a failure of the design. There was no safety factor at all; even though the bridge did not actually collapse because the crack went only one-third of the way through the beam. The O-rings of the Solid Rocket Boosters were not designed to erode. Erosion was a clue that something was wrong. Erosion was not something from which safety can be inferred.
There was no way, without full understanding, that one could have confidence that conditions the next time might not produce erosion three times more severe than the time before. Nevertheless, officials fooled themselves into thinking they had such understanding and confidence, in spite of the peculiar variations from case to case. A mathematical model was made to calculate erosion. This was a model based not on physical understanding but on empirical curve fitting. To be more detailed, it was supposed a stream of hot gas impinged on the O-ring material, and the heat was determined at the point of stagnation (so far, with reasonable physical, thermodynamic laws). But to determine how much rubber eroded it was assumed this depended only on this heat by a formula suggested by data on a similar material. A logarithmic plot suggested a straight line, so it was supposed that
the erosion varied as the .58 power of the heat, the .58 being determined by a nearest fit. At any rate, adjusting some other numbers, it was determined that the model agreed with the erosion (to depth of one-third the radius of the ring). There is nothing much so wrong with this as believing the answer! Uncertainties appear everywhere. How strong the gas stream might be was unpredictable, it depended on holes formed in the putty. Blow-by showed that the ring might fail even though not, or only partially eroded through. The
empirical formula was known to be uncertain, for it did not go directly through the very data points by which it was determined. There were a cloud of points some twice above, and some twice below the fitted curve, so erosions twice predicted were reasonable from that cause alone. Similar uncertainties surrounded the other constants in the formula, etc., etc. When using a mathematical model careful attention must be given to uncertainties in the model.
Of course, engineering a space shuttle, like the financial markets, is so complicated that we may never gain full understanding. The most dangerous thing is that we are so confident in our assessments of the uncertainties.
Another example is the Concorde, which was statistically the safest airplane that ever flew until one crashed, when it became statistically the most dangerous (and ceased to fly).
Me likey the recent merger with OB.
It would appear that, for whatever purpose, be it for internal or external consumption, the management of NASA exaggerates the reliability of its product, to the point of fantasy.
This is an interesting analogy, but it probably applies just as strongly to the architects of the bailout as it does to those who created the crisis.
The Concorde may have been "statistically, the safest" airplane, but economically, it was a frickin' black hole.......all it took was one crash to nail it's coffin shut.
Ed Tufte in one of this books, The Visual Display of Quantitative Information (I believe), explored the visual presentation of data that were used to make the decision to launch the Challenger. And critiqued it. He presented alternate visualizations that made the problem and risk, much clearer.
Your example here makes me wonder if there's any way that Tufte would be willing or able to tackle the same problem for the kinds of visual data of the market, or Credit default swaps, or some other relevant aspect of the (faulty) conditions that led to this mess.
The space shuttle is easier than the credit markets. It's just an engineering problem. A really big engineering problem, but still in the category of something we understand and break down into smaller manageable chunks and solve all of them.
We don't understand how credit markets work. We simply don't. Anyone who says otherwise is either a fool or fooling you.
This is a common problem in many arenas. In mountaineering it has the name "non-event feedback loop". For every non-event that happens during the iteration of some risky activity the perceived risk of that activity goes down, regardless of the inherent risk. An activity could be insanely risky, say a 1 in 10 chance of death, but once you've done it without hurting yourself even just a few times you'll tend to start to think it's pretty safe right up until the inherent risk catches up to you and bad things happen.
An excellent and very commonplace example is automobile accidents. We tend to have the impression that driving is pretty safe, even compared to things like air travel, but it is one of the most dangerous activities people engage in on a regular basis.
If credit markets are like the Space Shuttle, we should severely restrict their role in the economy and set hard leveraging limits to ensure they don't get very big. If something is crash-prone, you don't rely on it.
If the market problem we are seeing is basically a result of human psychology, then we should severely restrict any innovation in financial markets. Innovating with technology is one thing; innovating with human psychology is obviously pretty different.
I think you want to be careful about what argument you're making here.
This elaborates something noted earlier this month here: http://www.economy.com/dismal/blog/blog.asp?cid=108844 . The definitive book on the subject is Charles Perrow's 1983 "Normal Accidents."
Check it out...
I am reminded of Einstein's quip:
The same might be said of the financial engineers who came up with the CDOs and other complicated instruments without realizing their implications: finance is too hard for them.Quoting Feynman can never be wrong...
The other thing is that rocket flames weren't being paid bucketloads of money to eat through the O-rings.
There's something to be said in the NASA cases for the decision and command structures. Prior to both the Challenger and Columbia disasters, there were knowledgeable people within NASA who thought there was a good chance there would be a catastrophe. However, these people's opinions were always drowned out by others who had more practical concerns (schedule constraints, cost considerations, even public affairs worries).
At least within the space shuttle program today, this has been addressed through the use of "Time Out" cards, little plastic cards given to each employee who can effectively stop the operations if they feel they have a valid concern regarding the safety of the space shuttle crew. However, I'm not sure anyone thinks this will prevent another accident from occurring.
As we know,
There are known knowns.
There are things we know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don't know
We don't know.
Google "black swan" for someone who studies exactly these types of issues. If we don't understand something very well, we can't afford to pretend to.
Small threadjack-
As I read the part about "predicting" erosion, I wondered if you realize, Megan, that the argument that Feynman advances against the use of mathematical models created through empiric curve fitting applies quite well to good ol' Global War... excuse me, Climate Change.
Oddly this reminds me of my advanced econometrics class, I recall being admonished not to do my stats like a little monkey. I replied that I am a little naked monkey.
bud,
Taking Feynman's words and applying them so broadly shows amazing stupidity. Well done!
That you were able to gleen some information from his lesson, yet apply them so poorly makes me wonder if you have A.) a terrifyingly short attention span or B.) read only so far until something matches with your preconceived notions.
The fact is, the only way this could apply to the uncertainties in climate change is if people were creating climate models based on limited variables with wildly uncertain variations. Taken as a whole, one would be naive to say that the climate models guarantee an X degree increase in global temperatures within a given year, but they can almost guarantee that the temperatures are increasing and they're increasing corresponds eerily well with an increase in global carbon output.
the only way this could apply to the uncertainties in climate change is if people were creating climate models based on limited variables with wildly uncertain variations.
They're not?
The only way curve fitting works is if you have a good model. I remember TAing a class involving a magnetically damped air track slider. Thanks to the magic of rapid curve-fitting software, one of the students fitted a parabola to a graph of the acceleration vs. time. the R^2 value was very high; .98 or so, so he though he must have done it right. I asked him if he expected the acceleration to launch itself off to infinity after a few more seconds. Hmmm...maybe an exponential decay curve would be a better choice.
Extrapolation is extraordinarily dangerous, as any statistician will tell you, and it's only possible if you have a truly thorough theoretical understanding of the system such that your early data points are just used to fix a few parameters. If your early data points are used to determine the model itself, then extrapolation is completely and totally impossible.
Jeremy -
Not to threadjack, but if you think people actually _understand_ the Global Circulation Models used for climate "prediction", you are seriously deluded.
Global carbon output continues to grow, yet global temperatures have been on a downward trend since 1996.
I hate to argue from authority, but Freeman Dyson (among others) have argued that _all_ climate prediction models are inherently flawed, as the models are riddled with "fudge factors" needed to coerce the model into producing results that correspond to present-day conditions.
The problem is that there is no particular reason to assume that those "fudge factors" are constant as the model evolves into the future.
And, of course, there is no theoretical basis for those "fudge factors" - if there was a basis, they wouldn't be "fudge factors".
As I read the part about "predicting" erosion, I wondered if you realize, Megan, that the argument that Feynman advances against the use of mathematical models created through empiric curve fitting applies quite well to good ol' Global War... excuse me, Climate Change.
Quite true. If the models predict a five-degree temperature increase by a given date, that is no proof that there will not be a ten-degree increase.
I think a comparison to preventing 9/11 is more apt. It is easy to 'armchair quarterback' with 20/20 historical vision. I completely agree with Feynman's main point, which is that the reliability and safety were way overstated. Any attempt to read his statement as though the potential reliability could have easily been increased, or that the disaster should have been prevented may be well wide of the mark. An illuminating question to him would be how many similar concerns were raised before the fact.
This is where I see the tie in with 9/11. In that case, we had actionable data, but it was buried in a sea of seemingly similar, yet worthless data with no way to filter for the right data a priori.
I would also offer my agreement that Feynman's complaint about the empirical model for O-ring penetration is applicable to GW models. My only caveat would be that I think the GW models have larger problems. They are much less specific with more areas that are not fully known.
That being said, the sharp rise in CO2 which corresponds to the recent temperature rise is a valid area of significant concern. I just think many people are overstating our understanding of what is going on.