iSixSigma

Determining Significant Cutoff Scores

Six Sigma – iSixSigma Forums General Forums General Determining Significant Cutoff Scores

Viewing 15 posts - 1 through 15 (of 15 total)
  • Author
    Posts
  • #53969

    JCH
    Participant

    My task is to determine the significance, if any, that Test Scores have on Tenure. Currently, we accept Test Scores of 16-23 as it was decided that these scores tend to indicate that one will attain higher tenure. I was asked to determine if the low-end cutoff score could be reduced to 11 without major risk. My problem is that none of the analysis I have done shows any significance whatsoever with scores and tenure. That is, test score doesn’t indicate eventual tenure. Scores are broken up into 5 categories, Weak low, Moderate, low, Strong, Moderate High, Weak High. The Strong category is what we want, supposedly.

    I have conducted various tests on the data from simple ANOVA to full scale Regression and have been unable to identify the significance of test scores as they relate to tenure. This could be due to my ignorance of testing methodology, or because there really is not significance to be found. I’ve attached my data in Minitab format. Any clarification as to how I might better determine statistical significance would be appreciated.

    JC, Houston

    0
    #192335

    JCH
    Participant

    I see Minitab is not allowed. Here is my data in Excel.

    0
    #192340

    JCH
    Participant

    Hmm, no one huh? Darn.

    0
    #192341

    Robert Butler
    Participant

    You have several negative values in the days of tenure column that are extremely negative and these are killing your comparisons. If you drop these you can get an overall significant result of test scores and tenure days, however, if you try to run all of the pairwise multiple comparisons of test score categories and adjust for these comparisons using something like Tukey-Kramer you do not have sufficient data to indicate any significant (P < .05) pairwise differences.

    If you use the grouping of Category 2 the overall significance disappears. If you boxplot your data with either category 2 or test score as the grouping variable the reason for lack of significance becomes apparent. Based on the data you have provided there is no apparent significant correlation between tenure days and test score.

    If the negative numbers are excluded and you plot tenure days against test score you get a very nice distribution (there won't be much difference in appearance between this and a boxplot where test score is the category). The plot suggests you might have 3 different groups of individuals – those that make up the fairly solid region of the plot, those that form sort of an immediate cloud above the more solid region and then a crew that sort of seems to be off by themselves starting around a test score of 18 and up to around 36 or so. It might be something and then it might not.

    0
    #192342

    JCH
    Participant

    Thanks Robert. Can you please tell me how you plotted the data so that your assertion in paragraph 3 becomes apparent. I’m trying to see the 18-36 range you mention, but my ignorance is screwing me up again.

    0
    #192343

    Robert Butler
    Participant

    Just do a scatterplot of tenure days (Y variable) vs. test score (X variable)and look at the patterns. (I added an attachment now we’ll have to see if it works)

    0
    #192344

    JCH
    Participant

    Ok. Thanks. Got it. Now I get it. I confused myself. Essentially what I think I have found is that reducing our low-end cut off to 13, should not impact our tenure as there is no significant difference among those scoring 13-15 and those scoring on what they think the sweet spot is of 17-23. Thanks for your guidance on this. I appreciate it. And the attachment made it through as well.

    0
    #192345

    Robert Butler
    Participant

    I had some time to take another look at your data. I won’t ask the why’s and wherefore’s of the data but I must admit that, with the exception of professions such as suicide jockeys or live transformer cleaning cowboys, I can’t think of a single occupation where one would expect or want the kinds of summary statistics one gets with this data.

    Across all of the data you have a median tenure days of 56, an average of 142 days and a standard deviation of 276. If you look at the distribution of medians by test score you find the 50% point of those to be 55 days…very curious.

    0
    #192349

    MBBinWI
    Participant

    @newbie6s – Dude: You post a query and expect a response less than 4 hours later? Hey, most of us have real jobs. If it were that critical, then most of us are for hire and would be happy to address your question on your timetable – for our price.

    0
    #192350

    MBBinWI
    Participant

    @rbutler – Robert: Given the angst of the poster to not getting an immediate response, I’m guessing that you got sucked in to a poorly crafted homework/test question.

    0
    #192353

    Robert Butler
    Participant

    Yes I know MBBinWI, the homework issue did cross my mind. If it is/was homework then the OP got lucky. I’d spent most of the day trying to explain the difference between usable and unusable data sets to one of my engineers and I was at a point where I needed a few moments of diversion so, homework or not, it was a welcome 10 minute break.

    0
    #192354

    JCH
    Participant

    Robert, You have spotted what is my conundrum. This data, and real data it is, is from a call center operation from 2008-2001, that has on staff over 600 full time employees at any given time. The scores are from a test that management has been convinced provides them a good indication of who will and will not remain employed for an extended period of time. And you are correct, the data is most unwanted, though not unexpected for this particular location. My concern was that I was missing something major, because none of the tests I ran indicated anything significant. My concern is that we are spending a huge sum of money per month, $10,000, for testing services that don’t do what they say they are doing.


    @MBBinWI
    . Dude, this is not homework. This is real-world work. If it is poorly crafted, that is due to my ignorance alone. As for my angst, yes it is there. My apologies for what was more of a rant than was indicated in the text. But it is real. And the costs are real. And the fact that our client is most concerned at the moment is quite real. So real that the project may in fact be moved, which would result in the termination of several hundred employees. These are the facts I’m dealing with.

    And lest you think I popped on here out of laziness; I have in front of me now 2 Business Statistics books, and several other primers. I also have a text given to me by a GE Master Black Belt-my mentor on the sly as it were. And yes, he provided input, but at such a high level as to be useless to me at this point. He is a client, and not at all supposed to see any data whatsoever from me. His guidance is coming into question. As I am very new to all this, I don’t know what I don’t know. The books, while helpful, were not as clear as Robert was.

    We are not a Six Sigma company. We are one that has 30 years of attitude ingrained into its psyche. But, perhaps, if this one project can at least eliminate a waste of 120K per year, and save a few hundred jobs, then just maybe we can get more action from the senior leaders. The fact is not lost on me that they chose me for the project, as I have no Six Sigma experience to speak of. But the consultants who wanted to charge several thousand dollars didn’t make a good pitch. So I’m left to my own devices. Right or wrong, that’s the facts. Not homework.

    JCH, Houston

    0
    #192355

    MBBinWI
    Participant

    @newbie6s – OK. I withdraw my homework accusation. You’re either being forthright, or the most detailed homework poster that I’ve seen for some time (you’d be amazed how often posts get on here that are clearly just test/homework by people too lazy or dumb to do their own work).
    What bothers me more about your post is that your mgmt team thinks that several thousand dollars is too much to address a problem that, as you scope it, is at least $120k and I would suspect even larger as naiive companies such as yours probably haven’t looked at the entire system costs. If they can’t make an investment decision on improving their company that has at least a 300% ROI, then I’d be looking for another place to call home.

    0
    #192356

    Robert Butler
    Participant

    Thanks for the clarification JCH. In that case – your job is to show management what the data clearly demonstrates – the test doesn’t do what is claimed. As someone once said – there’s nothing like a graph to make a point graphically so what you should do is provde a plot just like the one below. The overlay of the MEDIAN and MEAN tenure day values for each of the test scores will allow everyone to see what the data is telling them – namely that there is nothing here.

    It is important to show the relationship between the means and the medians and all of the data because this will cut down on “mental cherry picking” – the old – “I remember some of the data and I’m sure I saw a trend between test scores and tenure.”

    The reality is that there isn’t any cut point with respect to test score and tenure time and a plot like this should make that fact clear to anyone who can see.

    0
    #192358

    JCH
    Participant

    @Robert. You have hit the nail directly on its head. I’m in the process of writing my report now.


    @MBBinWI
    . While I try to thorough, I doubt I’m cool enough to come up with this one on my own. If it were from a book, I think the clarity on what I need to do would be lacking. We have done little in the way of assessing the true costs of this testing methodology, and the negative impacts of the turnover that it doesn’t seem to predict. As for finding a new home, easier said than done. I have tenure here and losing that at this point in my life would prove difficult. Because of all the hidden costs, ROI is a very bad word at my place. I kid you not, we have a vendor right now who claims that the ROI on our investment of 70K is 2.75 million annually. They can’t prove it. The reports they send us don’t show it. But that is their number. For us, the ROI on that investment is that we comply with a client requirement. We want their business, we shell out 70 large for some online tool they say we need. Thanks for the input and the help.

    I’ll be certain to be clearer, and more patient, should I request further assistance.

    JCH, Houston

    0
Viewing 15 posts - 1 through 15 (of 15 total)

You must be logged in to reply to this topic.