Resolution of Ordinal Rating Scale
- April 7, 2011 at 4:16 pm #53781
BradshawParticipant@mbsh Include @mbsh in your post and this person will
be notified via email.
I have a situation where an ordinal rating system is utilized to make decisions off of visually assessed parts. The ratings are ordered by severity of appearance. I have performed an Attribute Agreement Analysis to determine the within appraiser and between appraiser agreeements. Where I am stuck is determining a resolution of the rating system. Since it is ordinal, I would expect to have some sort of estimation of resolution (i.e. ± 1 rating unit), like that which is easily obtained with a continuous measurement scale.
Are there any tools/procedures out there to assist one in determining a “practical” resolution for an ordinal rating system gage? Thanks in advance!0April 11, 2011 at 3:29 pm #191438
BreytenbachParticipant@wynandpb Include @wynandpb in your post and this person will
be notified via email.
I assume that you refer to Resolution as the smallest change a measurement system can detect in the quantity that it is measuring. For instance, a rulers resolution is 1 mm, a speedometers resolution is 1 km, and the resolution of digital display showing 2 decimal numbers is 0.01 xx (where xx is the applicable SI unit).
I had a case where the SOP determined that a Lab had to split a composite sample into three and, after analysing them, had to report the average as an individual reading. In this case, the resolution is determined by number of repeated measurements involved: i.e., dividing by 3 will give a resolution of 0,333; dividing by 4 a resolution of 0,25.
The common criterion for variable data is that Resolution should be at least 1/10th of the tolerance (UspecL LspecL). No criterion exists for discrete data.
Since ordinal measurements describe order, but not the relative size or degree of difference between the items measured, its resolution is a function of the ordinals definition. For instance:
For ordinal numbers where the ordinal scale has a constant length, such as: “very bad = 0 20%; bad = 21 40%…; very good = 80 100%”, the resolution would be 20%. A 5-point scale like this cannot be used with a criterion that demands Acceptable performance to be 4 or better, since the Resolution is equal to the interval of acceptable performance.
If the ordinal scale rank stuff in classes, such as Class A, B, C … and the scale of the classes are different, or who came first, second, or third (i.e., the results of a horse race which says who came first, second, or third but include no information about race times), the Resolution cannot be determined.
The central tendency of ordinal numbers can only be represented by its mode (the class or rank that occurred the most times) or its median, but not its arithmetic mean. With both the Mode and the Median, the results must be interpreted by taking the scaling into consideration. See Bashkansky & Gadrich for more elaborated analysis.
The dispersion can be expressed as a Range = the number of ordinals/classes covered by the data.
Wikipedia http://en.wikipedia.org/wiki/Resolution, under the tag Measurement Resolution for other forms of resolution
Bashkansky, E. & Gadrich, T. Some metrological aspects of ordinal measurements, http://www.springerlink.com/index/e8x73vn3m3534642.pdf presents a way to evaluate classical metrological characteristics, such as error, uncertainty and precision of single and repeated measurements based on the legitimate basic operations for ordinal data.
wynandpb0April 14, 2011 at 5:34 pm #191444
PaulonisParticipant@paulonis Include @paulonis in your post and this person will
be notified via email.
I wonder based on your description if you are truly asking for resolution as defined by wynandpb (the typical definition) or if you are looking for something more like a confidence interval of the assessment (e.g., given an appearance rating of 6, there is a 90% confidence that the true appearance is between 5 and 7).
If it’s more of the latter that you are looking for, you may want to look at intraclass correlations. A nice introduction is provided in When Quality is a Matter of Taste by Futrell.
If this is the type of thing you’re after, it’s possible that the Bashkansky & Gadrich reference provided by wynandpb could have much greater detail (I’m not familar with it). You just wouldn’t find what you’re looking for under “resolution”.0
You must be logged in to reply to this topic.