Debunking Dubious Statistics from a Six Sigma Critic

	Bookmark This Page Email This Page Format for Printing Cite This Article Submit an Article Read More Articles
	Discussion Forum "There is now actual evidence that many of the companies that have implemented Six Sigma have performed below the Standard & Poor's Index. So is it really good for a business?" Contribute to this Discussion

Download Products Six Sigma Green Belt Training Slides - 2009 Version Six Sigma DMAIC Training Slides - 2009 Version CSSBB Preparation Pack - 2009 Version

By Forrest W. Breyfogle III
 
"There are lies, damn lies and statistics," we have all been warned since the 19th century. The warning is apt these days because of questions about the use of statistics that have been on exhibit in recent months. The principal and founder of a performance improvement consulting firm professes that businesses using Six Sigma are underperforming their peers in terms of stock price.

Response and Comments This article is in response to material published in a number of business media earlier this year, including The Wall Street Journal in January. Related commentary can be found in a blog by Michael Marx of the iSixSigma Blogosphere and several threads in the iSixSigma discussion forum, one of which is hightlighted in the discussion forum box to the right.

Charles Holland of QualPro claims that among 58 companies that have announced broad use of Six Sigma, the stock performance of 91 percent trailed the S&P 500 index since the launch date of their implementations. Regrettably, some otherwise savvy business publications have been seduced by this data and quoted the conclusions.

QualPro offers businesses a design of experiments system that the firm's website claims is superior to Six Sigma. The system, called multivariable testing, "uses more complex mathematics than is used in a Polaris Missile," according to the website. Nowhere on the site is the obvious stated: That design of experiments is one of the many tools of Six Sigma.

Six Sigma achieves only 10 percent of potential process improvement, the site asserts, while multivariable testing is credited for capturing a dazzling "90 percent of process improvement missed by most Six Sigma efforts." How can missed potential be measured? The site is silent on this question.

Scholarship Versus Magical Thinking

Let us examine the logic QualPro uses in its show trial of Six Sigma.

The company uses a research design that equates correlation with its distant cousin – causation. It examines two variables – a company's use of Six Sigma and its stock performance – and concludes there is causation when it has shown only correlation. Nowhere to be seen are the many different factors that affect stock price movement. Where was multivariable testing in this experiment?

Questions abound:

Are the 58 companies looked at in this exercise representative of the Fortune 500 or were they chosen selectively?

Is stock price the only reliable measure of a company's overall performance? The investors who bought into the dot.com boom clearly know the answer.

What constitutes the "launch date" of a company's Six Sigma implementation? Is there a definable period or did the exercise include a date to help support a premise?

Indeed, does QualPro even understand Six Sigma? A review of its website provides little assurance. It defines the methodology as a four-step process, when, in fact, it is a five-step process. Six Sigma, it says, "lays out a very loosely structured path" while, in truth, the process is rigorous and the roadmap it lays out is as straight as an arrow.

Holland demonstrates a woeful misunderstanding of the process in an article in the January-February 2007 issue of Chief Executive magazine. "Six Sigma is merely an overpriced set of low-level statistics courses," he says. "While Six Sigma may detect trouble, it does not lead to a redesigned, more effective process. The best that can be expected is slow, incremental improvement – and most companies do not reach this stage."

A Glimmer of Hope

QualPro's website is correct about Six Sigma in one of its statements. "With the organizational pressure on managers to do Six Sigma comes a tendency to do something and activity often is expended on projects with no potential," it says. This cannot be denied.

Six Sigma sometimes is deployed for projects that provide little payoff. Moreover, standing alone the methodology cannot provide the level of control an organization's leaders need to ensure that everyone in the business is doing the right things right and at the right time – what we call the "3Rs of Business." Even when combined with Lean, Six Sigma does not provide the integrated excellence at all operational and corporate levels that should be required and can be measured with bottom-line predictability.

But the contributions Six Sigma is making to businesses cannot be dismissed so cavalierly. The methodology has saved Fortune 500 companies more than $400 billion since 1987, according to research by Michael Marx of iSixSigma Research. DuPont recently credited Six Sigma for $3 billion in savings. On QualPro's website, in stark comparison, DuPont credits multivariable testing for improvements worth $23 million.

Same Exercise, an Opposite Conclusion

The correlation-equals-causation hypothesis Holland has relied on can be used to reach any wished-for conclusion. Ron Marchelletta, an advisory engineer who is a Lean Six Sigma Black Belt, demonstrates this with an exercise that uses QualPro's own data.

Marchelletta examined the stock performance of 12 companies identified as QualPro clients on the company's website or in and article in The Wall Street Journal that quoted Holland. He then looked at the stock performance of 12 companies listed as Six Sigma users from the same sources. He improved on Holland's research design, which fails to account for market swings, by establishing a common date as a baseline to compare the stock performance of these companies. Making a solid choice, he selected Sept. 17, 2001, the lowest point of the stock market.

The result? The mean increase in stock price for the Six Sigma users was just under 95 percent, while it was only 84 percent for the multivariable testing companies. Does this demonstrate that Six Sigma is superior to multivariable testing? No, because correlation does not prove causation. Though Holland may believe it does, we doubt that he will cite this exercise in his next polemic about Six Sigma.

"The Bottom Line" Links

BEST SELLING PRODUCTS (iSixSigma Publications) Six Sigma Black Belt (DMAIC) Training Slides - 2009 Version! The 2009 Six Sigma Black Belt course includes over 40 more slides than the 2008 version. Contents include: 1,220 PowerPo... Certified Lean Six Sigma Black Belt Assessment Exam Interested in assessing your knowledge of Lean Six Sigma? Preparing for certifications? Testing your students and traine... Certified Lean Six Sigma Green Belt Assessment Exam This assessment exam is useful for students interested in assessing their knowledge of Lean Six Sigma on the Green Belt ... Kaizen Workshop E-book This 150+ page ebook teaches key tools and techniques of Kaizen, as well as real application to enhance learning. Kaizen... Certified Lean Six Sigma Black Belt E-book In 670 pages learn everything within the Lean Six Sigma DMAIC body of knowledge to successfully achieve Black Belt certi... Process Management Training Slides The 2008 Process Management course is designed in two phases comprised of:352 Powerpoint slidesInstructor notesSlide exp... Design For Six Sigma (DFSS) E-Book or Print Need an "encyclopedia" consisting of many of the tools you’ll study? Need a helpful refresher to apply the DFSS process?... Sell Your Products