Sig Figs versus Statistical Significance

Six Sigma – iSixSigma Forums General Forums Methodology Sig Figs versus Statistical Significance

Viewing 1 post (of 1 total)
  • Author
  • #53733


    So I’ve run up against this one a few times and I’ve flip flopped on my position enough times that I felt it was best to open the discussion up to those who are more knowledgeable than I am…

    I have a product whose tolerance is ± .002″. It is being measured with a laser mic which has a digital readout to 5 decimal places. The laser mic frequently alarms because it finds sections where the product dimension drops below the nominal by .0024″. The engineering department argues that they are asking for only three decimal places of accuracy, therefore the lower spec limit on the mic should not be set at -.002″, but rather at -.0025″ since anything 5 tenths or greater would round up and be within specification. From the specification standpoint I can agree with their logic, however if I run a process capability analysis on the product using the rounded measurements I fail because discrete jumps of .001″ inflate the standard deviation and fail to satisfy the assumption of normality implicit in such a study.

    By the same token, if I use all the resolution on the gage I have units which are outside of specification limits and therefore fail. I can move to nonparametric methods for assessing capability and probably pass, but that’s going to drive up the sampling costs and in a way I feel that it’s cheating.

    What is the resolution when what is being asked for technically flies in the face of what is statistically appropriate?

    … or am I just stupid as Stan would say?

Viewing 1 post (of 1 total)

You must be logged in to reply to this topic.