Dan:
If you are trying to reduce in-process variation (% disagreement) within this group of 23, I’d recommend you use an “Expert Gage R&R”, a slightly different form of “Attribute Gage R&R” suggested above.
Since you’re dealing with attribute (discrete) data, bear in mind that a true Gage R&R is not possible (as that needs data to be variable or continuous). What you can do, is measure “percentage agreement” amongst the group of 23 operators, either amongst themselves (as suggested by first response on this string), or, with respect to an “expert” (the “should be”), which would be “expert gage R&R”.
You might want to include two separate measures in your MSA – first, the operator selects whether there is a defect or not (i.e. sample pick list item is a “defective”), and second, what type of defect it is (amongst the list of possible defects you mention).
Attribute MSA’s give you Alpha and Beta errors – the percentage chances that the operator incorrectly accepts a rejectable part, as well as the percentage chances the operator rejects an acceptable part.
So while you won’t get true Gage R&R measures (%Tolerance etc.), you will get a first pass rolled throughput yield (RTY), using which you can compute sigma capability of your measurement system.
An expert gage R&R would allow you the option (which you might NEED, based on your findings!) to benchmark the operator-to-operator variation as part of your study.
Tabulating both – Defect/No Defect, AND What type of defect – will help you analyze MSA results using a P-chart and U-chart, on whether the MSA is STABLE or not.
If the MSA results show, for instance, that the LOWEST operator-to-expert agreement is, say, 95%, then your measurement system is 3 sigma capable with a 1.5 sigma shift.
Hope this helps…..
.
