Attribute Agreement – Multiple Cases, Multiple Scorers
April 26, 2018 at 2:07 am #55987
Any tips as to how I can use an attribute agreement study for my data?
I have 500 cases.
Each case has 10 questions scored on a 1-10 ranking (so I’m assuming it’s ordinal data).
Each case has 5 people scoring it.
The 5 scorers may be completely different people to one another. At this point I’m looking for overall attributes across the sample, rather than how individuals are performing.
I can’t seem to lay it out such a way that gets me anything other than error messages.0April 28, 2018 at 7:34 am #202514
Based on your description it doesn’t sound like you have data that could be used in an agreement analysis (I’m assuming when you say “agreement” you want to look at something like the Kappa statistic).
For an agreement study you have to have some kind of contrast on the same things you are measuring. For example, if you want to know how well an individual rater agrees with themselves the usual procedure is to provide a series of things (let’s use x-ray images of joint injury as an example) and have the individual rate each of the images on whatever scale is being used to assess the images. The images are given to the rater in a random order and then, at some later date they are re-randomized and given to the same rater. The rating scores are matched by images and the Kappa statistic is calculated on the before and after ratings.
On the other hand, if by “agreement” you mean equivalence then you would use the t-test and run the Schuirmann’s TOST equivalence test. However, the problem with this approach is the same as the one for the Kappa – you need some kind of pairing of measured responses.
Rather than continue with a series of “if” statements I think you will need to provide more detail with respect to what it is that you are trying to do.
Some issues to address:
500 cases – what are these – all the same kind of thing, groups of similar things but not all the same, etc.?
10 questions on an ordinal scale – same questions for all – all questions different or do they group into question types (i.e. questions 1-3 focus on patient demographics, questions 4 and 5 are focused on economic status, etc.)
Why 5 people per case?
Are the cases randomly assigned?
What do you mean by overall attributes across the sample?0
You must be logged in to reply to this topic.