Thursday, April 09, 2009

Statistics, Damn Statistics, and Lies

After all, facts are facts, and although we may quote one to another with a chuckle the words of the Wise Statesman, “Lies - damn lies - and statistics,” still there are some easy figures the simplest must understand, and the astutest cannot wriggle out of. ~ Leonard Henry Courtney

Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: "There are three kinds of lies: lies, damned lies, and statistics." ~Mark Twain

Mark Twain is said to have heard the quote and created his own version later, embellishing it with the attribution to Disraeli, whom he evidently thought was the "Wise Statesman." I find it ironic that a quote about the science of Statistics, maligned for being used to manipulate facts, has in fact been manipulated itself.

Most people will admit that statistics are useful, at least until the numbers don't go their way. Then statistics are an elegant way of telling half-truths (as someone else once said). As usual, it depends on whose ox is being gored.

To my mind there are two very distinct uses of statistics. One is the evaluation of actual physical data, that is, physical measurements of a sample of things are taken, and statistics are used to determine what this sample tells us about the population as a whole. More often, statistics are used in this context to compare to populations of things without having to measure all of them. Why not just measure all of them, you say? Well, there may be too many things in the populations, or the measurement could be destructive. If you were planning to sell the things, destroying all of them in the measurement process would not be a profit-making methodology.

The other use is polling. Polling is an entirely different animal because it is highly dependent on technique. For example, there were stories recently about why polls in New Hampshire picked Obama to beat Clinton (who actually won). The bottom line is that polling technique was not good.

Opinion polling depends on getting a representative sample, asking the right questions or at least asking the same questions consistently, and hoping the respondents don't lie. When any of those things fail, the poll can look pretty stupid (as in "Dewey Defeats Truman").

Personally, I don't much care about opinion polls. I make my decisions on whatever facts I can obtain, whether it's about a political candidate, brands of cars, or what to watch on TV. What the crowd thinks isn't my only measure of what's good or right. It's merely another factor, and usually a small factor at that.

However, measurement statistics can be rendered unreliable as well. I was in quality control for a long time, and statistics is a big part of that game. I saw stats misused often, sometimes on purpose, more often because of errors in method or interpretation. Let me give you a classic example.

I was working at a factory that made blades for a well-known label maker. The tolerances on the height of the blade were tight because the blade had to just cut through the plastic label, but not through the paper backing. That way you could grab the little piece of plastic and pull the paper off. Those of you who got a label maker with a blade that wasn't right know how much fun that could be.

Well, we seemed to have a lot of trouble making those blades, which was odd because we held equally tight tolerances elsewhere. But there we were, throwing out a good chunk of product. Worse, the customer was sending back another good chunk of product because we weren't catching all of it.

Our industrial products quality engineer went out and ran a process control study, which consisted of measuring the height of five blades every 10 minutes and recording them over a number of hours. She came back and announced that the process was out of statistical control and that the only solution was to measure more parts more often. Remember that business about destructive measurements? Measuring those little blades wrecked their cutting edge, so measuring more of them meant more going into the trash. Moreover, she couldn't guarantee that would catch all the bad product.

My boss didn't think management was going to buy that, so he came to me, the consumer products quality engineer and said, "HELP!"

In reviewing the data, I found a few things. First, there were a couple of arithmetic errors that made the data look stranger than it was. Second, the process was out of statistical control, but that wasn't bad. "Statistical control" means that data points are randomly scattered over the time frame taken. Few processes are actually in statistical control because tools wear, which causes a dimension to change over time. This is a predictable condition and can be dealt with. In fact, the process would produce parts that would be well within tolerance for a period of about eight hours before the grinding wheels had to be adjusted.

The third thing was the killer, though. At the time that the adjustment had to be made, the process would go nuts for about an hour, then level out and run smoothly again. So, something was going weird when the operator was making adjustments. I went to the manufacturing engineer (who was my legendary fishing buddy, Moon) and asked what could be going on. He said we should just go out and see what the operator was doing.

It turned out to be an issue of timing. The operator made adjustments but measured the product before the adjustment had taken effect, so he was always over- or under-adjusting until he got lucky. Once he was instructed to wait 10 seconds before measuring a part after an adjustment, the problem went away.

So the statistics didn't lie; they were screaming at us what the problem was, and no one took the time to find out what that was. In other words, the statistics were being used in a vacuum. That was the mistake. The statistics were about the world, but no one went out into the world to see what was really happening.

This happens in science at times. People get enamored with a set of measurements and overlook the fact that they just might be in error or that the wrong method of interpretation has been applied. It's not the statistics that are bad, it's either the data or our methods. Either way, we should approach statistical summaries with caution. Don't be afraid to ask to look at the raw data because there might be some surprises there.

By the way, as a result of our little investigation, it was determined that we could significantly reduce inspection. Things worked great for two weeks when the quality engineer came to me and said, "Your reduced inspection doesn't work. Last night's production was rejected this morning." I asked her what the inspection data during the shift showed. When she got it, it turned out it showed that the entire run was bad. The machine had a mechanical problem, but the shift supervisor told the operator to keep running anyway. The engineer would have known this if she had simply looked at the statistical charts.

Numbers don't lie; people, on the other hand ...

No comments: