iSixSigma

Test vs. Control Group

Six Sigma – iSixSigma Forums Forum Basics Welcome Test vs. Control Group

Viewing 11 posts - 1 through 11 (of 11 total)
  • Author
    Posts
  • #53480

    d
    Participant

    Hi!

    We are doing a test with 30 samples in our test group and 30 in control group. I understand that 30 is a statistically significant sample size but still wondering if this is enough to come to any conclusions.

    Thanks,
    Pam

    0
    #190297

    Robert Butler
    Participant

    A sample of 30, or any other number, is just that – a sample of a given size – there isn’t anything statistically significant about it.

    As to your question – the only answer that can be offered is an extremely generic one – if you have two groups of something and if you have treated them differently and taken some measurement on each member of the two groups there are statistical tests that could be used to assess the possibility of differences between a measurement (or measurements) made on the two groups.

    For anything more specific you will need to provide more information before anyone can offer any meaningful suggestions/answers.

    Questions:

    1. you say you have a “test group” and a “control group” – are these groups samples of items from some kind of production run or are the biological in nature (people, for example) or are they something else?

    2. If they are objects of some sort – what kind of a sample – random across some period of time, a simple grab sample from a production stream, or something else?

    3. If they are biological – is this a case/contol study, a simple random draw of patients from a population. are the test and control group matched in any way (gender and age, for example) or have they been selected in some other manner?

    4. What kind of a measurement are you taking on your control and test groups – some kind of continuous measure, binary (yes-no), or something else?

    0
    #190304

    d
    Participant

    Thanks Robert. The test and control groups are for agents and we’ll be measuring their metrics – service levels, resolution rate etc. So, we are taking 30 agents in each group and will be comparing the results of the 2 groups.

    0
    #190311

    Robert Butler
    Participant

    If you are going to compare group of people and look for differences in various performance measures as a result of some kind of additional training/reward/punishment applied to one group and not to the other then you have a lot of work to do.

    Some of the things you will need to consider:

    1. Are the groups equivalent before the treatment? That is equivalent with respect to such things as age, gender, years of experience on the job, position level, work environment, etc.

    2. Are the groups equivalent at baseline? You will have to have measures of their baseline measures and you will have to run comparisons to make sure they are not significantly different.

    3. What kinds of differences in metrics do you think you could reasonably expect to see as a result of before and after treatment? You will want to run power calculations to get some idea of the degree of power associated with the differences you have in mind. Low power doesn’t mean you shouldn’t run the test but it will mean that the odds of repeating the significance of any result you do find will be low – this, in turn will mean that if you do get a significant result (P< .05 or whatever P-value you are using) you will have to repeat the test with an adequately powered study to confirm the significant finding before you press on with trying to apply the changes your initial study suggested might be beneficial.

    4. You might want to think about running paired tests. You could look at the differences in before and after treatment for both control and treated groups (you will want to measure the control group at the beginning of the experiment as well as at the end even though you didn’t “do” anything to the control group whether you are running paired tests or not). You could then examine the change in metrics for both to see if the differences were significantly different from 0. The issue here is that you are no longer looking at actual metric values but at their differences.

    5. What are the metrics – one would assume they are some kind of ordered variable – continuous, discrete, ordinal. Regardless you will have to think about the kinds of statistical tests you will need to use.

    There are other things to consider as well but the above are some of the important issues you will need to consider before running your test.

    0
    #190312

    d
    Participant

    Thanks Robert. This really helps

    0
    #190321

    will
    Participant

    Pam,

    Going back to your original question regarding whether sample size is adequate to determine significance between the control sample and the other, there is a statistical method to determine sample size needed.

    You have to first select the ? and ? (Type I and Type II risks) you want to work with. Secondly you will have to determine how much minimum difference in the measured characteristic you want to detect. Finally you must have historical variation data for the characteristic you are measuring (i.e. s or ?).

    Then you can use the simple formula:

    n = Z2 ?2 / ?2
    where
    Z= 1.96 at the 95% confidence level (? = 0.05)
    ? = Standard deviation
    ? = Difference in the mean to be detected

    OR
    Minitab also can do this caluclation. It also uses different ? risk levels or (1-?) power levels in the calculation.

    Hope this helps.

    0
    #190322

    d
    Participant

    Thanks Cyrus.

    0
    #190324

    Robert Butler
    Participant

    Power and sample size calculations are fine if you have something other than guesses on which to base them. If all you have is a belief that such-and-such a change may prove beneficial, sample size estimates based on guesses of assumed means and assumed standard deviations (or, if these are not available, assumed percentage changes) are a great way to waste time and money.

    When my investigators come to me with problems for which no prior art is available to provide estimates of the above the sample size will be based on what they can afford in terms of time and money. The study will be treated as a pilot. We will analyze the data and the alpha levels will be whatever they will be. Should we find something that is statistically and physically significant we will run a post-hoc power analysis. Typically, the post-hoc analysis will show that while the alpha was significant the study was very underpowered.

    If the results are deemed to be worthy of further investigation we will use the results of the pilot to provide meaningful estimates of means/standard deviations/percentage changes which, in turn will allow us to identify a realistic sample size for a given power and our alpha value of choice. It is this sample size we will use for follow up/confirmation work.

    0
    #190325

    bbusa
    Participant

    Apart from all the good advice you have , I think you should first check if your approach is right (i.e Test Vs Control Group) for the metrics mentioned .

    In an organizational context , I feel it will be very difficult to create two independent groups such as these while keeping many other factors constant .

    Just my opinion

    0
    #190338

    d
    Participant

    Good pt, Robert. Its gving me some more things to think about.

    0
    #190339

    d
    Participant

    It is difficult to islolate all factors, I understand but at this point, test vs control is our only option. How else do we find out if our new methodolgy is working or not. Thanks for your opinion though.

    0
Viewing 11 posts - 1 through 11 (of 11 total)

You must be logged in to reply to this topic.