Margin of Error and Confidence Levels Made Simple

	Bookmark This Page Email This Page Format for Printing Cite This Article Submit an Article Read More Articles
	Related Tools & Articles Sample Size Calculator Is There Bias in Your Random Sample? How to Determine Sample Size
	Discussion Forum "I was asked today to determine a sample size required for different confidence intervals (80, 90, and 95 percent respectively) I have not found a way to accurately answer this question." Contribute to this Discussion

B

New from iSixSigma Electric Load Forecasting Project Example Knowing the Customer Research Report Account Withdrawal Accuracy Project Example

y Pam Hunter

A survey is a valuable assessment tool in which a sample is selected and information from the sample can then be generalized to a larger population. Surveying has been likened to taste-testing soup – a few spoonfuls tell what the whole pot tastes like.

The key to the validity of any survey is randomness. Just as the soup must be stirred in order for the few spoonfuls to represent the whole pot, when sampling a population, the group must be stirred before respondents are selected. It is critical that respondents be chosen randomly so that the survey results can be generalized to the whole population.

How well the sample represents the population is gauged by two important statistics – the survey's margin of error and confidence level. They tell us how well the spoonfuls represent the entire pot. For example, a survey may have a margin of error of plus or minus 3 percent at a 95 percent level of confidence. These terms simply mean that if the survey were conducted 100 times, the data would be within a certain number of percentage points above or below the percentage reported in 95 of the 100 surveys.

In other words, Company X surveys customers and finds that 50 percent of the respondents say its customer service is "very good." The confidence level is cited as 95 percent plus or minus 3 percent. This information means that if the survey were conducted 100 times, the percentage who say service is "very good" will range between 47 and 53 percent most (95 percent) of the time.

Survey Sample Size	Margin of Error Percent*
2,000	2
1,500	3
1,000	3
900	3
800	3
700	4
600	4
500	4
400	5
300	6
200	7
100	10
50	14
*Assumes  a 95% level  of confidence

Sample Size and the Margin of Error

Margin of error – the plus or minus 3 percentage points in the above example – decreases as the sample size increases, but only to a point. A very small sample, such as 50 respondents, has about a 14 percent margin of error while a sample of 1,000 has a margin of error of 3 percent. The size of the population (the group being surveyed) does not matter. (This statement assumes that the population is larger than the sample.) There are, however, diminishing returns. By doubling the sample to 2,000, the margin of error only decreases from plus or minus 3 percent to plus or minus 2 percent. Although a 95 percent level of confidence is an industry standard, a 90 percent level may suffice in some instances. A 90 percent level can be obtained with a smaller sample, which usually translates into a less expensive survey. To obtain a 3 percent margin of error at a 90 percent level of confidence requires a sample size of about 750. For a 95 percent level of confidence, the sample size would be about 1,000.

Determining the margin of error at various levels of confidence is easy. Although the statistical calculation is relatively simple – the most advanced math involved is square root – margin of error can most easily be determined using the chart below. A few web sites also calculate the sample size needed to obtain a specific margin of error. Thus, if the researcher can only tolerate a margin of error of 3 percent, the calculator will say what the sample size should be. One such free site is http://www.azplansite.com/samplesize.htm. Among the best sites is http://www.raosoft.com/samplesize.html which allows researchers to compare the sample sizes needed with various levels of confidence similar to comparison shopping.

Calculating Margin of Error for Individual Questions

Margins of error typically are calculated for surveys overall but also should be calculated again when a subgroup of the sample is considered. Some surveys do not require every respondent to receive every question, and sometimes only certain demographic groups are analyzed. If only those who say customer service is "bad" or "very bad" are asked a follow-up question as to why, the margin of error for that follow-up question will increase because the number of respondents is smaller than the overall survey sample. Similarly, if results from only female respondents are analyzed, the margin of error will be higher, assuming females are a subgroup of the population.

Survey Data Is Imprecise

Margin of error reveals the imprecision inherent in survey data. Survey data provide a range, not a specific number. A researcher surveying customers every six months to understand whether customer service is improving may see the percentage of respondents who say it is "very good" go from 50 percent in one period to 47 percent in the next six-month period. Both are accurate because they fall within the margin of error. The decrease is not statistically significant. On the other hand, if those percentages go from 50 percent to 54 percent, the conclusion is that there is an increase in those who say service is "very good" albeit a small one.

The Dark Side of Confidence Levels

A 95 percent level of confidence means that 5 percent of the surveys will be off the wall with numbers that don't make much sense. Therefore, if 100 surveys are conducted using the same customer service question, five of them will provide results that are somewhat wacky. Normally researchers do not worry about this 5 percent because they are not repeating the same question over and over so the odds are that they will obtain results among the 95 percent. However, if the same question is asked repeatedly such as a tracking study, then researchers should beware that unexpected numbers that seem way out of line may come up. For example, customers are asked the same question about customer service every week over a period of months, and "very good" is selected each time by 50 percent, then 54 percent, 52 percent, 49 percent, 50 percent, and so on. If 20 percent surfaces in another period and a 48 percent follows in the next period, it is probably safe to assume the 20 percent is part of the "wacky" 5 percent, assuming proper methodology is followed.

About the Author

Pamela Hunter is director of the Center for Survey Research and Analysis at the University of Connecticut's Stamford campus. She has worked for the Pew Research Center for the People and the Press and an international polling firm in Washington, D.C. Dr. Hunter has worked on projects ranging from a presidential campaign to international studies on workplace issues to investigation of the market for genetic products. She has fielded surveys in many countries from India to the United Kingdom. She received her Ph.D. from the University of Connecticut specializing in survey research. As a graduate student, she worked for the Roper Center for Public Opinion Analysis, the world's largest archive of public opinion data. The Roper Center is located at the University of Connecticut. Dr. Hunter can be reached at drhunter@isixsigma.com.

"The Bottom Line" Links

BMG CERTIFICATION Enhance Your Career With Lean Six Sigma Skills Green Belt or Black Belt - May 12 :: Atlanta, GA Green Belt to Black Belt Upgrade - July 7 :: Atlanta, GA

VILLANOVA UNIVERSITY   Earn Your Lean, Green or Black Belt Six Sigma Master Certificate Online BOOST YOUR SALARY! *$38k more than uncertified counterparts. Learn From Industry Leaders!   START NOW

SIGMAPRO MBB, Lean Sigma, & DFSS   when experience matters most...

M O T O R O L A U N I V E R S I T Y Learn from the most experienced practitioners of Six Sigma in the world Public Training & Certification Click here to take a free Six Sigma Lesson

J URΛN   Lean Six Sigma Public Workshop Upgrade to Black Belt- May 2008 Become one of your organization's 'vital few'. Get Juran Certified.

. ISSSP Follow the Leaders to Arizona! 9th Annual Six Sigma Leadership Conference May 19th-22nd, 2008 • Scottsdale, AZ Click Here: More information & REGISTRATION .

HOWARD UNIVERSITY 4 weeks + 1 cost saving project + 1 mentor + 1 caring instructor = 1 Marketable Certified Lean Six Sigma Black Belt

WINSTON SALEM STATE UNIVERSITY 4 weeks + 1 cost saving project + 1 mentor + 1 caring instructor = 1 Marketable Certified Lean Six Sigma Black Belt

THE UNIVERSITY OF TEXAS AT AUSTIN 2 weeks + 1 project = Black Belt Certification

. Find us on LinkedIn Join the iSixSigma Network and Connect with Other Six Sigma Pros .

Download the iSixSigma Toolbar for 1-Click access. Search Your Way. Everyday. Without Delay.

BEST SELLING PRODUCTS (iSixSigma Publications) 2008 VERSION! Six Sigma DMAIC Training Slides The OSSS Six Sigma DMAIC course is comprised of:1,176 slides, Instructor notes, Slide explanations, 37 data sets, 20 sup... Gage R&R Excel Template Gage Repeatability and Reproducibility (R&R) studies measure the amount of measurement variation that is attributabl... Certified Lean Six Sigma Black Belt Assessment Exam This test is useful for students interested in assessing their knowledge of Lean Six Sigma on the Black Belt level. It c... Certified Lean Six Sigma Green Belt Assessment Exam This assessment exam is useful for students interested in assessing their knowledge of Lean Six Sigma on the Green Belt ... NEW VERSION! Process Management Training Slides The OSSS Process Management course is designed in two phases comprised of:352 Powerpoint slidesInstructor notesSlide exp... NEW VERSION! Six Sigma Green Belt Training Slides The OSSS Six Sigma Green Belt course is comprised of: 1047 slidesInstructor notesSlide explanations35 data sets20 suppo... 5th Annual iSixSigma Global Salary Survey Research Report The 5th Annual iSixSigma Global Salary Survey report is based on the responses of 2,142 Six Sigma professionals currentl... Sell Your Products