Sample Size

Thanks Katie! sorry, but I am little weak in Maths, so the movement i see formula(s)… topic starts bouncing over my head:(

I am confused after googling sample size calc, as it gives the output but not sure how it concludes on sample number..hence, wanted a detailed explanation about various factors that are considered to determine the sample size. wanted a detailed explanation on what confidence interval, confidence level, Z value etc are while calculating sample size

Sample Size is dependent on what statistical question you are trying to answer, the degree of confidence you want to determine a difference exists and a degree of confidence the difference does not exist (alpha and beta risk), standard deviation, and what practical difference you are attempting to see.

@rbutler is a great resource. You don’t have to understand maths, statistical programs like Minitab can help spit out the numbers out of their black box that does the calculations for you but you really need to understand the ramifications of the inputs you are giving to the packages. Read up on some of the sources he mentioned or use google.

Thanks all for your responses & the magic table!!

I am continuing with my research on sample size.. in the meantime, can anyone list down all the factors considered to determine sample size…

eg. (1)confidence level (if its a factor considered)
(2) confidence interval etc

There are two steps to calculating sample size:
1. Pick the formula to use
2. Pick 4 out of 5 factors

For step 1. The best way to determine which equation to use is to look at which hypothesis test you’ll want to run later. Will you want to compare two proportions? 2 means? 2 variances. A good hypothesis testing road map will guide you to the right test, then if you use a program like Minitab, you’ll see its sample size options align with each of the hypothesis tests (except the non-paramentrics, for non-parametric tests add 15%, reply to the post for more info).

Step 2: I’m going to assume you are using continuous data (reply to the post for the proportional data portion). For continuous data there are 5 factors. Decide which you want to solve for an plug in the other 4. Below are the factors and recommended assumptions.

- Alpha risk – this is the risk of committing a Type 1 error (i.e., innocent man sent to jail). The default in the stats books is 0.05, but you can vary it based on the risk you want to assume. You can go as high as 0.1 without making the stats experts freak out. Higher the alpha, fewer the samples.

- Power – this is one minus the beta risk (the inverse of the beta risk). Beta risk is the probability of committing a Type 2 error (i.e., guilty man goes free). The default for the power in the stats books is 0.90 (10% beta risk) and you can go as high as 0.80 before the stats experts freak out. Lower the power, fewer the samples.

- Standard deviation: what is the standard deviation in your data? Most people struggle here because they want to know how many samples to take and therefore don’t have any data. If you do have some representative sample data you can substitute that value. If you have no data, none, there is a trick I’ll talk about in the next bullet. Smaller the standard deviation, fewer the samples.

- Difference: how much of a difference in the means do you want to see before your test will identify it as being statistically significant. For example, you have two piles of boards. One pile was cut by operator A and the other by operator B, you think they are different. How big of a difference do you want to see before you claim “statistically significant”? Is a quarter inch enough? 1/8 of an inch? The smaller the difference you want to get excited about, the more samples you’ll need. Common mistake: Do not use the difference between the means as the difference in this calculation. You won’t pick up the difference. Rather, how fine of a measuring stick do you want to use?

Okay, here is the forementioned tip: If you don’t know the standard deviation, that’s okay. In the equations it is a ratio of the difference to the standard deviation. So if you don’t know the standard deviation: set the standard deviation to 1 sigma and set the difference to how many standard deviations appart should the means be before you get excited. Do you want to know a 1/2 standard deviation difference, a 1/4 standard deviation difference? Larger the difference, fewer samples needed (but that also means you’ll miss detecting small shifts).

- Sample size – how many samples do you need to collect? On first pass, this is the one you are usually solving for. However, after you see your initial sample size calc, you might freak and say “well, I can’t collect 1250 samples!!” So now you play your numbers games. One option is to set this number to “how many samples do you want to collect? Or how many can you afford?” Then you can solve for one of the other values like: If I take 30 samples, then I’ll only be able to detect differences greater than 1″.

I found this approach simplifies all the equations and calculations and gives Belts more control. So “what hypo test will you eventually want to run?” and “pick which factor to solve for.”