iSixSigma

multi analysis comparison

Six Sigma – iSixSigma Forums Old Forums General multi analysis comparison

Viewing 10 posts - 1 through 10 (of 10 total)
  • Author
    Posts
  • #48364

    Andrew Kennett
    Participant

    G’day,
    We are looking at a supplier change in one of our raw materials.  The new supplier provided a material which was all in specification and quite close to the previous material.   We then made a short production run with the new material and so I now have 2 sets of samples, I’ve measured a number (12) paramters on the sample sets.  Now clearly I can t-test each parameter and I can visualise all the data on a siper/radar chart but can I do some sort of overall comparison?   I’ve thought of doing a paired t-test on the averages of the each parameter but I’m not sure this is wise (or valid).   Any ideas?
    Cheers,
    Andrew

    0
    #162834

    Erik L
    Participant

    Andrew,
     
    Based off of the information that you’ve provided, it would appear that conducting t-tests on the characteristics is your best route forward.  I would not recommend the paired t-test.  If the same experimental units/products generated the responses with your historical provider and the new supplier, then you would have a reason to conduct this analysis.  It appears that you have independent data. 
     
    As with any hypothesis scenario, you need to know what is considered a ‘practically significant’ difference, which would be considered undesirable, from your historical performance.  You would like to have a certain level of confidence in saying that there has been a statistically significant change and you need to know, for the alternative, what is the directionality of the change.  Is any difference of interest?  Do you want to ensure that the new supplier’s performance is not greater than the old, but less is fine?  Or vice versa. 
     
    The key question will be whether there is enough data, to make conclusions of statistical and practical significance, with a high enough actual power.
     
    Regards,
    Erik

    0
    #162962

    Andrew Kennett
    Participant

    Erik,
    Well here is my data set:

     
    Std
    Trial
    T-Test p-val

    Break Force
    4385.9
    4007.4
    0.260

    Bend Dist
    1.3
    1.2
    0.537

    Packet Length (per 5)
    34.7
    35.0
    0.105

    Packet Weight (per 5)
    40.0
    40.2
    0.364

    Across (per 1)
    64.3
    64.9
    0.020

    Along (per 1)
    65.2
    65.6
    0.025

    Top Minolta L*
    59.2
    57.4
    0.183

    Top Minolta a*
    10.3
    9.8
    0.022

    Top Minolta b*
    30.4
    31.7
    0.101

    Bot Minolta L*
    59.3
    58.5
    0.224

    Bot Minolta a*
    10.1
    9.7
    0.055

    Bot Minolta b*
    31.5
    30.9
    0.015

    Crmb Minolta L*
    76.5
    77.0
    0.126

    Crmb Minolta a*
    2.1
    1.1
    0.000

    Crmb Minolta b*
    24.1
    23.8
    0.342
     
    As you can see 15 parameters, 5 had T-Test p-vaules less than 0.05 so I can probably say the 2 samples are different but can I summarise this by some overall test.  My first thought is a paired t-test (p-val = 0.33) using the above data pairs but the stats wizs will probably have a heart attack so is there another way?
     
    Andrew

    0
    #162964

    Szentannai
    Member

    Hi Andrew,
    maybe you could try to calculate a sigma values for each parameter – as you probably have the spec limits and also a summarized sigma value for each supplier.
    The advantage of this would be that you take the spec limits into account : i.e. you dont only show that there is a difference in the parameters between the two suppliers but also whether this difference is important (or not) from practical POV.
    The disadvantage will be that the summarized sigmas are do not have a lot of information – but if you need a single number they might do the trick.
    Regards
    Sador
     

    0
    #162967

    Robert Butler
    Participant

      A couple of thoughts/questions:
      I don’t know anything about the kinds of measurements you made nor about the measurement methods but looking over your list I’m struck by the fact that you have numerically very small differences that are testing out as significant.  
    Two questions: Were the samples really independent?  If they were, do differences of this kind matter with respect to your process/final product? 
      If they were independent and if differences of this size do matter I would ask the following:
     1. Does the list have a hierarchy of importance?  That is, can you rank the properties from most important to least important?
      If you can then the first question is this: Where do those properties that tested out as exhibiting a significant difference fall.  If all 5 are on the bottom end then, while there may be a difference, the question that needs to be addressed is does this kind of a difference matter physically and financially?
     2.  Given that you have the sample size and the mean and standard deviations for the two samples what is the resultant power of the significant differences you have identified?  If the differences are underpowered (as they probably are) you may want to take an adequate sample and test the two materials again on those properties that did exhibit a significant difference just to make sure you are confident that the difference really exists.  This would be particularly worthwhile if a change of suppliers is going to cost in terms of time, money, and effort.

    0
    #162991

    Jerry C
    Participant

    Andrew-
    Did you consider using MANOVA (multivariate analysis of variance) to test the suppliers simultaneously across all 15 response variables? It would be superior to attempting 15 individual t-tests on the 15 different characteristics, for a couple of reasons. First, you would obtain a single integrated test of product quality differences (between the two suppliers). Second, you would not struggle with how to reach a single conclusion (different or not different) by interpreting 15 different test results (some of which pointed to differences, most of which did not).I assume that your 15 parameters are all product characteristics that were measured in product samples made from the two different suppliers’ materials. I also assume that you showed us your parameters’ average values, along with p-values from their associated two-sample (2-sided?) t-tests. If so, then you had to have multiple measurements of each parameter, for each of the two production samples, to perform the two-sample t-tests. Thus, you have enough information to do either a general (or possibly a balanced) MANOVA…I don’t know anything about your parameters, but some of them sound very similar to one another. You do not want to have multiple parameters logically representing the same characteristic or property of the output, in the analysis – especially if they have strong statistical correlation, as well. In this case, select a single variable to represent each critical dimension/characteristic of the product. If two measurements on a unit of product (say value of response measured on the top of the unit, and value of response measured on the bottom of unit) are really the same property of the unit, just different locations, you could combine them into a single variable – just include both measurements as replicates.Good luck.Jerry

    0
    #163008

    Andrew Kennett
    Participant

    I’d like to thank everybody for their replies and I’ll go ahead and do MANOVA and determine the power of my tests.  I should also look at some sort of weighting.
    For your information I can say the two samples are wholemeal cookies baked on an industrial plant within a hour of each other, the wholemeal flour (about 25% of the recipe) was changed from one supplier to another.  Some of the test parameters (like packet length) are physical parameters used for process control and for which we have well established UCL and LCL.  The colour parameters are more of an isseue as we have no data history or specs and some people thought there was a colour difference (proper statisical sensory testing will be done this afternoon).  Of course baking effects the colour and hence the 3 colour measurements (top, bottom and crumb) and I used the Minolta colour measurment system (gives L*, a* and b*).
    Thanks again for your help.
     
    Andrew

    0
    #163023

    Robert Butler
    Participant

      Multiple samples from a single batch of cookies made with one kind of flour are not independent.  Your sampling method is repeated measures which probably explains the small differences and the fact that some of the small differences tested out as statistically significant. 
       The method you have used does not capture the real variation present in your system.  Most likely, you have an extreme under estimation of the variation.  This, in turn, probably means that the differences you detected will not translate into anything of physical significance. There are methods available for the analysis of repeated measures but they are not trivial and will require some time to understand and apply.

    0
    #163049

    Tim
    Member

    Robert – the implication being that a number of independent batches/production runs be made to generate a sufficiently large number of samples?  does this conclusion change if the process is continuous rather than a batch method?  also, thank you for your continuing practice of providing sound advice.  I always make a point of reading threads on which your name appears. 

    0
    #163337

    Perryman
    Participant

    Andrew,
    My question to you is why would you want to compare the two materials in the first place?  Shouldn’t you just be testing to make sure that the new material produces a final product that is of equal or better quality.  I am not sure why you are considering re-sourcing but assuming it is for cost reasons, if using the new material produces a product that falls within acceptable limits – and is cheaper – than there is no need to compare the two raw materials.
    My 2 cents,
    Patch

    0
Viewing 10 posts - 1 through 10 (of 10 total)

The forum ‘General’ is closed to new topics and replies.