Debunking Dubious Statistics from a Six Sigma Critic

	Bookmark This Page Email This Page Format for Printing Cite This Article Submit an Article Read More Articles
	Discussion Forum "There is now actual evidence that many of the companies that have implemented Six Sigma have performed below the Standard & Poor's Index. So is it really good for a business?" Contribute to this Discussion

B

New from iSixSigma Capability Analysis Course Stat Free Six Sigma ebook Basic Statistics Reference Guide

y Forrest W. Breyfogle III
 
"There are lies, damn lies and statistics," we have all been warned since the 19th century. The warning is apt these days because of questions about the use of statistics that have been on exhibit in recent months. The principal and founder of a performance improvement consulting firm professes that businesses using Six Sigma are underperforming their peers in terms of stock price.

Response and Comments This article is in response to material published in a number of business media earlier this year, including The Wall Street Journal in January. Related commentary can be found in a blog by Michael Marx of the iSixSigma Blogosphere and several threads in the iSixSigma discussion forum, one of which is hightlighted in the discussion forum box to the right.

Charles Holland of QualPro claims that among 58 companies that have announced broad use of Six Sigma, the stock performance of 91 percent trailed the S&P 500 index since the launch date of their implementations. Regrettably, some otherwise savvy business publications have been seduced by this data and quoted the conclusions.

QualPro offers businesses a design of experiments system that the firm's website claims is superior to Six Sigma. The system, called multivariable testing, "uses more complex mathematics than is used in a Polaris Missile," according to the website. Nowhere on the site is the obvious stated: That design of experiments is one of the many tools of Six Sigma.

Six Sigma achieves only 10 percent of potential process improvement, the site asserts, while multivariable testing is credited for capturing a dazzling "90 percent of process improvement missed by most Six Sigma efforts." How can missed potential be measured? The site is silent on this question.

Scholarship Versus Magical Thinking

Let us examine the logic QualPro uses in its show trial of Six Sigma.

The company uses a research design that equates correlation with its distant cousin – causation. It examines two variables – a company's use of Six Sigma and its stock performance – and concludes there is causation when it has shown only correlation. Nowhere to be seen are the many different factors that affect stock price movement. Where was multivariable testing in this experiment?

Questions abound:

Are the 58 companies looked at in this exercise representative of the Fortune 500 or were they chosen selectively?

Is stock price the only reliable measure of a company's overall performance? The investors who bought into the dot.com boom clearly know the answer.

What constitutes the "launch date" of a company's Six Sigma implementation? Is there a definable period or did the exercise include a date to help support a premise?

Indeed, does QualPro even understand Six Sigma? A review of its website provides little assurance. It defines the methodology as a four-step process, when, in fact, it is a five-step process. Six Sigma, it says, "lays out a very loosely structured path" while, in truth, the process is rigorous and the roadmap it lays out is as straight as an arrow.

Holland demonstrates a woeful misunderstanding of the process in an article in the January-February 2007 issue of Chief Executive magazine. "Six Sigma is merely an overpriced set of low-level statistics courses," he says. "While Six Sigma may detect trouble, it does not lead to a redesigned, more effective process. The best that can be expected is slow, incremental improvement – and most companies do not reach this stage."

A Glimmer of Hope

QualPro's website is correct about Six Sigma in one of its statements. "With the organizational pressure on managers to do Six Sigma comes a tendency to do something and activity often is expended on projects with no potential," it says. This cannot be denied.

Six Sigma sometimes is deployed for projects that provide little payoff. Moreover, standing alone the methodology cannot provide the level of control an organization's leaders need to ensure that everyone in the business is doing the right things right and at the right time – what we call the "3Rs of Business." Even when combined with Lean, Six Sigma does not provide the integrated excellence at all operational and corporate levels that should be required and can be measured with bottom-line predictability.

But the contributions Six Sigma is making to businesses cannot be dismissed so cavalierly. The methodology has saved Fortune 500 companies more than $400 billion since 1987, according to research by Michael Marx of iSixSigma Research. DuPont recently credited Six Sigma for $3 billion in savings. On QualPro's website, in stark comparison, DuPont credits multivariable testing for improvements worth $23 million.

Same Exercise, an Opposite Conclusion

The correlation-equals-causation hypothesis Holland has relied on can be used to reach any wished-for conclusion. Ron Marchelletta, an advisory engineer who is a Lean Six Sigma Black Belt, demonstrates this with an exercise that uses QualPro's own data.

Marchelletta examined the stock performance of 12 companies identified as QualPro clients on the company's website or in and article in The Wall Street Journal that quoted Holland. He then looked at the stock performance of 12 companies listed as Six Sigma users from the same sources. He improved on Holland's research design, which fails to account for market swings, by establishing a common date as a baseline to compare the stock performance of these companies. Making a solid choice, he selected Sept. 17, 2001, the lowest point of the stock market.

The result? The mean increase in stock price for the Six Sigma users was just under 95 percent, while it was only 84 percent for the multivariable testing companies. Does this demonstrate that Six Sigma is superior to multivariable testing? No, because correlation does not prove causation. Though Holland may believe it does, we doubt that he will cite this exercise in his next polemic about Six Sigma.

"The Bottom Line" Links

BMG Upgrade Your Skills Master Black Belt Certification Get the industry's most Flexible, Robust Program for Professionals >> Find out more

VILLANOVA UNIVERSITY   Earn Your Lean, Green or Black Belt Six Sigma Master Certificate Online BOOST YOUR SALARY! *$38k more than uncertified counterparts. Learn From Industry Leaders!   START NOW

SIGMAPRO MBB, Lean Sigma, & DFSS   when experience matters most...

M O T O R O L A U N I V E R S I T Y Learn from the most experienced practitioners of Six Sigma in the world Public Training & Certification Click here to take a free Six Sigma Lesson

J URΛN   Lean Six Sigma Public Workshops Atlanta, GA - Sept 2008 Become one of your organization's 'vital few'. Get Juran Certified.

. ISSSP Follow the Leaders to Arizona! 9th Annual Six Sigma Leadership Conference May 19th-22nd, 2008 • Scottsdale, AZ Click Here: More information & REGISTRATION .

HOWARD UNIVERSITY 4 weeks + 1 cost saving project + 1 mentor + 1 caring instructor = 1 Marketable Certified Lean Six Sigma Black Belt

WINSTON SALEM STATE UNIVERSITY 4 weeks + 1 cost saving project + 1 mentor + 1 caring instructor = 1 Marketable Certified Lean Six Sigma Black Belt

THE UNIVERSITY OF TEXAS AT AUSTIN 2 weeks + 1 project = Black Belt Certification

. Find us on LinkedIn Join the iSixSigma Network and Connect with Other Six Sigma Pros .

Download the iSixSigma Toolbar for 1-Click access. Search Your Way. Everyday. Without Delay.

BEST SELLING PRODUCTS (iSixSigma Publications) Organizational Culture Change Research Report Every company has a prevailing culture – the characteristic behaviors and norms that define what it is like to work in ... 2008 VERSION! Six Sigma DMAIC Training Slides The OSSS Six Sigma DMAIC course is comprised of:1,176 slides, Instructor notes, Slide explanations, 37 data sets, 20 sup... Certified Lean Six Sigma Black Belt Assessment Exam This test is useful for students interested in assessing their knowledge of Lean Six Sigma on the Black Belt level. It c... Gage R&R Excel Template Gage Repeatability and Reproducibility (R&R) studies measure the amount of measurement variation that is attributabl... 5S Assessment Tool Download this Excel template to assess any work area on their 5S activity. Breaks down assessment into all 5 groups: So... Cost of Poor Quality Reference Guide This 13-page reference guide focuses on understanding the concept of Cost of Poor Quality and the various approaches to ... NEW VERSION! Process Management Training Slides The OSSS Process Management course is designed in two phases comprised of:352 Powerpoint slidesInstructor notesSlide exp... Sell Your Products