iSixSigma

Formula for Overall Quality Score?

Six Sigma – iSixSigma Forums General Forums Implementation Formula for Overall Quality Score?

This topic contains 9 replies, has 6 voices, and was last updated by  MBBinWI 3 weeks, 2 days ago.

Viewing 10 posts - 1 through 10 (of 10 total)
  • Author
    Posts
  • #209834

    Anonymous

    In the BPO industry, it has been common practice to use average of average QA scores as the final score. In this scenario, the Quality form being used has 41 line items and not all the time that the 41 items are applicable to be scored. In this case, do we proceed with averaging the average scores or should it be weighted?

    0
    #209840

    Chris Seider
    Participant

    BPO?

    happy holidays–and no i’m not a bot LOL

    0
    #209851

    Strayer
    Participant

    I assume BPO is Business Process Outsourcing. Say 90% is from one source and the other 10% is divided among several others. That’ll exaggerate the other 10% in your final score. Unweighted average of averages might be useful in some circumstances but I can’t think of one. Do you want the average quality of products/services they provide? It’ll be skewed. Do you want to compare the average from one source to the overall average? It’ll be skewed.

    0
    #209879

    Mike Carnell
    Participant

    I am with Strayer on this. You are averaging averages so basically you are really obscuring anything the data is telling you. You have a very benign look at central tendency and absolutely no clue about dispersion. What this means is if two companies have the same score your system says they are equal. If those same companies have a std deviation of 3 and the other one 10 you score still believes they are equal. Maybe that makes sense since I have no clue what the 41 items are but I would seriously doubt it. Think about the mechanics to variables Control Charts and consider how idiotic a subgroup size of 41 is.

    You can get into some exercise about weighting but before I even start that exercise I would start with two basic questions 1. What do I want to know? 2. How do I want to see it? Once you can understand those two responses then you figure out what calculations you need to make to get that.

    This is just going to be a guess so if I am wrong just ignore it. When this stuff starts it is typically driven by some executive that says “I don’t want to see all this I just want one number.” One number in this situation isn’t going to tell you anything. best case two. If you have an executive that cannot work with two numbers I would quit and find another job. You can bet your paycheck when they review budgets it is more than one number.

    Just my opinion.

    0
    #209886

    UL
    Participant

    Hello

    One should look at the outliers in the data and then take a decision whether you want to consider average or median as the measurement.

    0
    #209900

    Strayer
    Participant

    Maybe the real problem here is that you aren’t getting raw quality data from your outsourced providers. I was usually omitted from negotiation of such contracts and then challenged with evaluating them, individually and overall, without the data I needed since they weren’t obligated to provide it.

    0
    #209905

    Mike Carnell
    Participant

    This may be way off track. I have worked in places where the supplier score actually didn’t mean anything to anybody. In most places it is done by supplier quality people who for the most part have no real power in the decision making around suppliers. I am not sure how the system currently works but the last time I went through a Ford Q1 survey I was amazed to find out that Purchasing was who decided who was Q1 certified. They had input from Supplier Quality of course but not final say.

    The score was 4 measures: 1. Quality 2. On Time Delivery (OTD) 3. responsiveness to change (basically out scheduling was horrible so we needed them to respond promptly to our lack of forecasting skill) 4. I don’t remember this category but basically if this makes sense you select what means something to you. If anyone really wants to know what we used for #4 let me know and I will track someone down and figure it out.

    This is how it worked and this is the important part. If you were perfect your score was 100%. Pretty easy concept. As you got worse your score got higher. So if I had a delivery issue then my score became something greater than 100%.

    Lets say that delivery issue got you a score of 125%. Intuitively it makes no sense. A higher score for worse performance. What that number represented was a factor that was used to adjust your prices you submitted for future business. So if you have a delivery problem and you submit a bid of $10 per part, we understand there is additional costs for late delivery so we multiply the $10 per part X 125% your supplier score so your quoted price now becomes $12.50 per part. Now the Supplier will get busy and start to care about their Supplier Score and Purchasing will get involved and actually try to get their favorite suppliers to improve their score.

    Just having a system that creates a number is a pretty much just mental masturbation. Make the system matter so you get something that helps you not just something that gives you something to do.

    Just my opinion.

    0
    #209968

    Strayer
    Participant

    You hit the nail on the head, Mike. I would note that so often we have complaints about quality from outsourced providers but they say they’re satisfying the service level agreement. And it may be true. SLA’s are usually negotiated by purchasing and executives without involving quality professionals, so we tend to get averaged/massaged data and have no insight into what’s really happening. CMMI (Capability Maturity Model Integration) and other frameworks insist that quality departments from the contractor and outsourced provider coordinate and work together, but that may not get into the contract. We don’t seem to understand the difference between a supplier of materials or services that are inputs to our processes vs. BPO which takes over our processes. To get back to the anonymous questioner, if you aren’t getting raw data from your outsourced providers you really can’t evaluate them or come up with an overall quality rating, whether or not you use weighting. When I was asked to do that I told management that I couldn’t do much more than compare how well they claim to be meeting their SLA’s and you already know that. Maybe I could provide some helpful internal data. For instance when we outsourced our IT help desk I could tell them how many complaints and reports of unresolved problems we were receiving. But since I had no insight into their raw data all I could say is make of this what you will.

    0
    #209970

    Chris Seider
    Participant

    Transparency is an enigma. :)

    0
    #210005

    MBBinWI
    Participant

    All good points. For something like this, I would choose a spider chart. Each of your main categories becomes a spoke from a central point. You plot the score from each along this spoke and then connect the lines. This helps to identify strength areas and weak areas. You can do the same for the details within each category. This way, you get the overall perspective, and have more detailed info to dive into for improvement.

    0
Viewing 10 posts - 1 through 10 (of 10 total)

You must be logged in to reply to this topic.