Measurement system analysis – essay grading

Hi Pal,
I would work out some grading guidelines – unless of course you already have one. I’m thinking of something like listing what are the “requirements” you have towards the recruitees – 1. Basic knowledege of statistics :)
2. English proficiency
3. Creativity
4. etc.Then go one level below and think what are the criteria for demonstrating a basic knowledge of statistics – depending on the subject of the essay of course.
E.g. :1.1 Correctly mentions two-sample t-tests in the context of the essay
1.2 Is aware of sampling strategy
1.3 etc.If you have such grading guidelines then you can figure out where you have the biggest variability and work from there.
I hope this helps a bit.
Regards
Sandor

Welcome to the mad world of six sigma in bpo recruitment and good luck getting some method to the madness.
 
You want to set up a process for essay grading. I see that part of your question also involves uniformity in grading. I would sufggest the following steps:
 
1. Identify exactly what parameters you would be looking for, just the same way as you would for call quality. Define and list what is it that your grading would depend on – spelling mistakes/ punctuations missed or misused/ incorrect sentence structures/ worong or missing prepositions/ conjunctions/ lack of clarity etc.
2. Get some members (the ones with real good language/grammar only) to list down the grading system as per which parameters are critical. Arrive at a points system or a binomial grading system (follow the pattern of a call quality grading system, should be easy)
3. Find and fix a team of people who would take up all the essay assessments. It’s pointless for R&R if every tom, dick and harry is involved in this process.
4. Get the team to understand the whole idea, the system and the objectives. Have them go through a training session if needed. Take in suggestions whenever suitable.
5. Start with a bunch of old essays. Let everyone in the team grade them for practice. Perform an R&R (callibration as in common bpo speak for R&R)
6. Once everyone is set within control limits, let them loose on the job.
7. Perform periodical R&R’s and a proper analysis and follow through for each r&r.
 
The principle things are
a) Having a limited no. of people grading essays
b) Identifying the parameters to look for in the essays (so that English proficiency levels are limited to established rules of grammar rather than subjectivity). Go by the route of looking for defects in composition and avoid looking at content as much as possible.
 
My two penny’s worth. Get back if you have any doubts/questions.

Great suggestion. I was getting ready to direct to the same example. Nice demonstration of Attribute Agreement Analysis.