iSixSigma

Measurement Metrics in a Subjective Scenario

Six Sigma – iSixSigma Forums Old Forums General Measurement Metrics in a Subjective Scenario

This topic contains 8 replies, has 8 voices, and was last updated by  Salomon 14 years, 8 months ago.

Viewing 9 posts - 1 through 9 (of 9 total)
  • Author
    Posts
  • #37282

    yadagri
    Member

     Problem Statement: Lack of defined measurement metrics
     Business Case: The approach and results of a business research analysts’ work is often subjective and therefore difficult to measure
     The Background: 
    I am looking to identify and measure metrics such as performance, quality, deadlines, productivity and utilization for an FTE team doing business research off the Internet.
     The job involves looking for requested information on the Internet and delivering it to the client. The problem is that there cannot be standards and times set for this work as it is very subjective. For e.g. if the information requested is direct and easily available on the Internet, then the time taken is less. On the other hand, if the information is much harder to look for then it takes a longer time. Example: If a person is looking for news on the current market trends in the automotive industry, this may not be that hard to find. On the other hand, if the person is looking for information on the current market trends on the dialysis industry, this may be much harder to find and the analyst will have to spend more time on it.
     Standard times cannot be set for this work, as it is subjective and depends on the type of information being looked for. Also, something like say the history of the dialysis industry over the last 5 or 10 years may take a very long time to find.
     In this case just because the business analyst takes a very long time to find the information, compared to a person looking for something direct, it does not mean the analyst is not good at his job.
     Any idea on how I may be able to come up with metrics to measure performance, quality, deadlines, productivity and utilization in such a case.
     
     

    0
    #109860

    Ganapathi
    Participant

    Dear Venkat,
    One way of looking at the problem:
    You will not build a control chart right from the first data. You would like to collect some 20 – 25 data points before start plotting.
    Similar way, you may NOT be able to define the target for a metric from day one. Collect data over a period of time, then arrive at an optimum target as bench mark for each metric you are interested  in. That means ‘ set your own standard’. Then compare the actual aginst the bench mark.
    Now the difficulty is : Which informatin you will look in , in order to set the bench mark since the client may ask any information. One suggestion is : Categorise the informatins generally the clients ask for. Like – info on automobiles; info on surgery; info on travel; info on books and literature etc. Targets set for one member in a category should be applicable to any member in the category, ofcourse with approximation – like data on a sample is applicable to the whole population. MG

    0
    #109861

    Wagh
    Participant

    Hi Venkat
    What you can do is ,There is certain information which is easy to search while certain which is difficult to search,so segregate the information depending on the level of difficulty and set service level agreements for both kind of informations.like for easy information it can be 1 day while for difficult it can be 2 or more depending on the level of difficulty.then you can measure the individual performances also.
    Prasad
    9845741360

    0
    #109863

    yadagri
    Member

    Prasad, thanks for your reply. This surely helps.
    The other problem we face is that how do I measure the quality of data that I send out. According to me, the data that I send out is 100% quality (else I will not send it out). Does that mean my quality level is 100% all the time?? Also, in a case where we do not receive feedback if what we have sent to the client is correct or not, how do you measure quality???? How can I rate my quality?

    0
    #109879

    Philip
    Participant

    A colleague sent me the link to your original message. 
    Can I ask a silly question? Why is the lack of defined measurement metrics a problem? What is that you hope to be able to do?
    Philip

    0
    #109904

    Hersey
    Participant

    I assume that you have identified all the metrics you are attempting to quantify as a result of House of Quality or similar activity indicating those that your customers feels are important and how much value they place on each.     The best source of data on your performance to these metrics are from the customers you serve.  A survey will provide the needed data in terms that are meaningfull to customer satisfaction and level of their delight in your service. 

    0
    #109923

    Henrique
    Participant

    Dear Venkat,
    We have here similar problems. Our experience is, if is difficult to mesure using “in process indicators”, ask to the customer. You can create a small report feedback where customers that asked for the reserch can evaluate the quality, time and actionability of work delivered.
    Best Regards
    Henrique

    0
    #109961

    Nishanth
    Participant

    Venkat
    Some thoughts here:
    Fundamentally, why do you have metrics? To measure whether you are below/meeting/exceeding a benchmark.
    In this special case, I would rather think the benchmark is what the customer wants. How can we find that? Simple – Constant VOC! Since each transaction is unique in its characteristic, there can be no standard. So, you can say you met your metric if the customer feels happy about it.
    Once you set up a Performance VOC collection system in place, which can ask Key Customer Performance Variables like, in your case, Quality of info provided (on a scale of 1 to 10), Tunraround time (scale of 1 to 10) etc, you have basic data to start analysis.
    As to performance monitoring, you can have, after observation for a considerable period, a P(85) or P(90) (Percentile scores) target set for each KCPV. You can set a target that P(85) should be, say, 8. This would mean, 85th percentile of your performance data score should be 8. A percentile mode would help you take out extreme outliers which are natural when customer rates (hence not taking mean)
    The challenge here is to collect feedback immediately after service delivery. But the abovesaid method should be quite seamless, as your CTQs are constantly mapped to Customer requirements online. ‘Expectation creep’ would be captured online and you can remap your targets immediately.
    Nishanth

    0
    #110075

    Salomon
    Member

    In addition, you can make your cualitative information very quantitative by using any of these:
    – Number of successes or failures to comply within a fixed standard
    – Time between failures
    – Average cycle time
    – % of occurrences within time standards
    – % of occurrences within (or outside) customer-defined “good, bad, medium”
    – Cost and/or revenue generated/lost due to appropriate/inappropriate service
    – etc

    0
Viewing 9 posts - 1 through 9 (of 9 total)

The forum ‘General’ is closed to new topics and replies.