multi analysis comparison
Six Sigma – iSixSigma › Forums › Old Forums › General › multi analysis comparison
 This topic has 9 replies, 7 voices, and was last updated 14 years, 10 months ago by Perryman.

AuthorPosts

October 9, 2007 at 12:53 am #48364
Andrew KennettParticipant@AndrewKennett Include @AndrewKennett in your post and this person will
be notified via email.G’day,
We are looking at a supplier change in one of our raw materials. The new supplier provided a material which was all in specification and quite close to the previous material. We then made a short production run with the new material and so I now have 2 sets of samples, I’ve measured a number (12) paramters on the sample sets. Now clearly I can ttest each parameter and I can visualise all the data on a siper/radar chart but can I do some sort of overall comparison? I’ve thought of doing a paired ttest on the averages of the each parameter but I’m not sure this is wise (or valid). Any ideas?
Cheers,
Andrew0October 9, 2007 at 5:13 pm #162834Andrew,
Based off of the information that youve provided, it would appear that conducting ttests on the characteristics is your best route forward. I would not recommend the paired ttest. If the same experimental units/products generated the responses with your historical provider and the new supplier, then you would have a reason to conduct this analysis. It appears that you have independent data.
As with any hypothesis scenario, you need to know what is considered a practically significant difference, which would be considered undesirable, from your historical performance. You would like to have a certain level of confidence in saying that there has been a statistically significant change and you need to know, for the alternative, what is the directionality of the change. Is any difference of interest? Do you want to ensure that the new suppliers performance is not greater than the old, but less is fine? Or vice versa.
The key question will be whether there is enough data, to make conclusions of statistical and practical significance, with a high enough actual power.
Regards,
Erik0October 11, 2007 at 7:09 am #162962
Andrew KennettParticipant@AndrewKennett Include @AndrewKennett in your post and this person will
be notified via email.Erik,
Well here is my data set:
Std
Trial
TTest pvalBreak Force
4385.9
4007.4
0.260Bend Dist
1.3
1.2
0.537Packet Length (per 5)
34.7
35.0
0.105Packet Weight (per 5)
40.0
40.2
0.364Across (per 1)
64.3
64.9
0.020Along (per 1)
65.2
65.6
0.025Top Minolta L*
59.2
57.4
0.183Top Minolta a*
10.3
9.8
0.022Top Minolta b*
30.4
31.7
0.101Bot Minolta L*
59.3
58.5
0.224Bot Minolta a*
10.1
9.7
0.055Bot Minolta b*
31.5
30.9
0.015Crmb Minolta L*
76.5
77.0
0.126Crmb Minolta a*
2.1
1.1
0.000Crmb Minolta b*
24.1
23.8
0.342
As you can see 15 parameters, 5 had TTest pvaules less than 0.05 so I can probably say the 2 samples are different but can I summarise this by some overall test. My first thought is a paired ttest (pval = 0.33) using the above data pairs but the stats wizs will probably have a heart attack so is there another way?
Andrew0October 11, 2007 at 9:30 am #162964Hi Andrew,
maybe you could try to calculate a sigma values for each parameter – as you probably have the spec limits and also a summarized sigma value for each supplier.
The advantage of this would be that you take the spec limits into account : i.e. you dont only show that there is a difference in the parameters between the two suppliers but also whether this difference is important (or not) from practical POV.
The disadvantage will be that the summarized sigmas are do not have a lot of information – but if you need a single number they might do the trick.
Regards
Sador
0October 11, 2007 at 12:11 pm #162967
Robert ButlerParticipant@rbutler Include @rbutler in your post and this person will
be notified via email.A couple of thoughts/questions:
I don’t know anything about the kinds of measurements you made nor about the measurement methods but looking over your list I’m struck by the fact that you have numerically very small differences that are testing out as significant.
Two questions: Were the samples really independent? If they were, do differences of this kind matter with respect to your process/final product?
If they were independent and if differences of this size do matter I would ask the following:
1. Does the list have a hierarchy of importance? That is, can you rank the properties from most important to least important?
If you can then the first question is this: Where do those properties that tested out as exhibiting a significant difference fall. If all 5 are on the bottom end then, while there may be a difference, the question that needs to be addressed is does this kind of a difference matter physically and financially?
2. Given that you have the sample size and the mean and standard deviations for the two samples what is the resultant power of the significant differences you have identified? If the differences are underpowered (as they probably are) you may want to take an adequate sample and test the two materials again on those properties that did exhibit a significant difference just to make sure you are confident that the difference really exists. This would be particularly worthwhile if a change of suppliers is going to cost in terms of time, money, and effort.0October 11, 2007 at 5:00 pm #162991
Jerry CParticipant@JerryC Include @JerryC in your post and this person will
be notified via email.Andrew
Did you consider using MANOVA (multivariate analysis of variance) to test the suppliers simultaneously across all 15 response variables? It would be superior to attempting 15 individual ttests on the 15 different characteristics, for a couple of reasons. First, you would obtain a single integrated test of product quality differences (between the two suppliers). Second, you would not struggle with how to reach a single conclusion (different or not different) by interpreting 15 different test results (some of which pointed to differences, most of which did not).I assume that your 15 parameters are all product characteristics that were measured in product samples made from the two different suppliers’ materials. I also assume that you showed us your parameters’ average values, along with pvalues from their associated twosample (2sided?) ttests. If so, then you had to have multiple measurements of each parameter, for each of the two production samples, to perform the twosample ttests. Thus, you have enough information to do either a general (or possibly a balanced) MANOVA…I don’t know anything about your parameters, but some of them sound very similar to one another. You do not want to have multiple parameters logically representing the same characteristic or property of the output, in the analysis – especially if they have strong statistical correlation, as well. In this case, select a single variable to represent each critical dimension/characteristic of the product. If two measurements on a unit of product (say value of response measured on the top of the unit, and value of response measured on the bottom of unit) are really the same property of the unit, just different locations, you could combine them into a single variable just include both measurements as replicates.Good luck.Jerry0October 11, 2007 at 9:46 pm #163008
Andrew KennettParticipant@AndrewKennett Include @AndrewKennett in your post and this person will
be notified via email.I’d like to thank everybody for their replies and I’ll go ahead and do MANOVA and determine the power of my tests. I should also look at some sort of weighting.
For your information I can say the two samples are wholemeal cookies baked on an industrial plant within a hour of each other, the wholemeal flour (about 25% of the recipe) was changed from one supplier to another. Some of the test parameters (like packet length) are physical parameters used for process control and for which we have well established UCL and LCL. The colour parameters are more of an isseue as we have no data history or specs and some people thought there was a colour difference (proper statisical sensory testing will be done this afternoon). Of course baking effects the colour and hence the 3 colour measurements (top, bottom and crumb) and I used the Minolta colour measurment system (gives L*, a* and b*).
Thanks again for your help.
Andrew0October 12, 2007 at 11:55 am #163023
Robert ButlerParticipant@rbutler Include @rbutler in your post and this person will
be notified via email.Multiple samples from a single batch of cookies made with one kind of flour are not independent. Your sampling method is repeated measures which probably explains the small differences and the fact that some of the small differences tested out as statistically significant.
The method you have used does not capture the real variation present in your system. Most likely, you have an extreme under estimation of the variation. This, in turn, probably means that the differences you detected will not translate into anything of physical significance. There are methods available for the analysis of repeated measures but they are not trivial and will require some time to understand and apply.0October 12, 2007 at 9:46 pm #163049Robert – the implication being that a number of independent batches/production runs be made to generate a sufficiently large number of samples? does this conclusion change if the process is continuous rather than a batch method? also, thank you for your continuing practice of providing sound advice. I always make a point of reading threads on which your name appears.
0October 18, 2007 at 1:19 pm #163337Andrew,
My question to you is why would you want to compare the two materials in the first place? Shouldn’t you just be testing to make sure that the new material produces a final product that is of equal or better quality. I am not sure why you are considering resourcing but assuming it is for cost reasons, if using the new material produces a product that falls within acceptable limits – and is cheaper – than there is no need to compare the two raw materials.
My 2 cents,
Patch0 
AuthorPosts
The forum ‘General’ is closed to new topics and replies.