iSixSigma

before and after data.

Six Sigma – iSixSigma Forums Old Forums General before and after data.

Viewing 6 posts - 1 through 6 (of 6 total)
  • Author
    Posts
  • #52902

    NewBee
    Participant

    I’m conducting some Minitab training for interal folks.  I have an agenda for each class (roughly 10 topics per class).  To judge the effectiveness of the training, I’m giving each student the topics, and asking them to judge between 1 and 5 where they are. 1 being “this is totally new to me” and 5 being “I can teach this class”.  After the class I’ll ask them to re-judge themselves.
    So given 20 students and 10 topics, that 200 datapoints before and 200 after.
    They will all be 1,2,3,4 or 5.  What test can I run to determine if there was a statistical difference bettwen the before dataset and after dataset?

    0
    #186713

    Dharma Bum
    Participant

    What if the negative scoring is really a reflection on poor facilitation?Your survey method and Likert scale needs some serious beefing up.

    0
    #186718

    Mike Carnell
    Participant

    newbee,
    Instead of a smiles test that drives “entertainment” from the instructor, why don’t you give them a test and put some of the responsibility for learning on the students.
    Who cares how you analyze it if you data collection is inane.
    Just my opinion

    0
    #186723

    Kluttz
    Member

    With that methodology, you wouldn’t be measuring training “effectiveness” (however you wish to define it). 
    First, your training participants are not entering class with the same baseline understanding of statistics, excel, etc.  So their understanding of Minitab tools AFTER class is really going to be the sum of the understanding before class plus what they gained during class. 
    In addition, with your method, you actually have 20 distinct measurement systems.  Regardless of how well you define what a “2” or “3” means, you’re still adding significant measurement system variation.  We’re still not considering the fact that your data will almost certainly be clustered around the 3’s and 4’s.  
    Lastly, does it really matter what they think their proficiency level is immediately after class?  Shouldn’t the point of the training be to have them retain the info long enough to implement the tools? 
    So if you’re really interested in measuring the effectiveness of your training class (as opposed to just making some neat little boxplots to put on a powerpoint), I’d recommend doing a pre-training assessment test to determine baseline knowledge levels.  Then I’d follow that up with two post-training assessment tests (final day of training & 30-ish days after training).  Then I’d run paired t-tests (pre-training test vs last day test and last day test vs 30 day test) and pareto the the lowest scoring problems for follow-up.
    (Disclaimer: I’m deliberately avoiding any discussion regarding whether the test scores would be discrete or continuous)

    0
    #186726

    Stevo
    Member

    Newbee,
     
    Not sure why you need to prove a statistical difference.  If I was coaching you (and I’m not) this is what I would say.
     

    Practical – If before = 2 and after = 5 —– Stop
    Graphical – If obvious —– Stop
    Analytical — Run a hypo test —– Stop
     Easy way out – Run descriptive stats.
     

    Mean, Median, std dev, shape and spread.  Eyeball them – stop wasting time.
    Stevo
     

    0
    #186727

    StuW
    Member

    Newbee,
    Your main issue here is that the pre to post test comparison should show improvement since you are providing instruction to people that are taking the class presumably because they don’t have those skills, correct?  So then the question becomes how effective was your training?   In other words, your scores should improve by at least two or three units on your scale, maybe more, and that becomes a very subjective assessment.
    My suggestion is drop the pre-test and focus the post test on characteristics of instruction, such as clarity, quality of examples, etc.  You can then focus on areas where your scores are lower than your expectations.
    If you did want a pre-test than you have two blocking variables that need to be accounted for in the analysis.   The person becomes a block to account for, as well as the topic.

    0
Viewing 6 posts - 1 through 6 (of 6 total)

The forum ‘General’ is closed to new topics and replies.