before and after data.
Six Sigma – iSixSigma › Forums › Old Forums › General › before and after data.
- This topic has 5 replies, 6 voices, and was last updated 12 years, 7 months ago by
StuW.
-
AuthorPosts
-
November 9, 2009 at 10:18 pm #52902
I’m conducting some Minitab training for interal folks. I have an agenda for each class (roughly 10 topics per class). To judge the effectiveness of the training, I’m giving each student the topics, and asking them to judge between 1 and 5 where they are. 1 being “this is totally new to me” and 5 being “I can teach this class”. After the class I’ll ask them to re-judge themselves.
So given 20 students and 10 topics, that 200 datapoints before and 200 after.
They will all be 1,2,3,4 or 5. What test can I run to determine if there was a statistical difference bettwen the before dataset and after dataset?0November 9, 2009 at 10:20 pm #186713
Dharma BumParticipant@Dharma-BumInclude @Dharma-Bum in your post and this person will
be notified via email.What if the negative scoring is really a reflection on poor facilitation?Your survey method and Likert scale needs some serious beefing up.
0November 10, 2009 at 12:49 am #186718
Mike CarnellParticipant@Mike-CarnellInclude @Mike-Carnell in your post and this person will
be notified via email.newbee,
Instead of a smiles test that drives “entertainment” from the instructor, why don’t you give them a test and put some of the responsibility for learning on the students.
Who cares how you analyze it if you data collection is inane.
Just my opinion0November 10, 2009 at 1:10 pm #186723
KluttzMember@Union-of-Conjoined-ScientistsInclude @Union-of-Conjoined-Scientists in your post and this person will
be notified via email.With that methodology, you wouldn’t be measuring training “effectiveness” (however you wish to define it).
First, your training participants are not entering class with the same baseline understanding of statistics, excel, etc. So their understanding of Minitab tools AFTER class is really going to be the sum of the understanding before class plus what they gained during class.
In addition, with your method, you actually have 20 distinct measurement systems. Regardless of how well you define what a “2” or “3” means, you’re still adding significant measurement system variation. We’re still not considering the fact that your data will almost certainly be clustered around the 3’s and 4’s.
Lastly, does it really matter what they think their proficiency level is immediately after class? Shouldn’t the point of the training be to have them retain the info long enough to implement the tools?
So if you’re really interested in measuring the effectiveness of your training class (as opposed to just making some neat little boxplots to put on a powerpoint), I’d recommend doing a pre-training assessment test to determine baseline knowledge levels. Then I’d follow that up with two post-training assessment tests (final day of training & 30-ish days after training). Then I’d run paired t-tests (pre-training test vs last day test and last day test vs 30 day test) and pareto the the lowest scoring problems for follow-up.
(Disclaimer: I’m deliberately avoiding any discussion regarding whether the test scores would be discrete or continuous)0November 10, 2009 at 1:32 pm #186726Newbee,
Not sure why you need to prove a statistical difference. If I was coaching you (and Im not) this is what I would say.
Practical If before = 2 and after = 5 —– Stop
Graphical If obvious —– Stop
Analytical — Run a hypo test —– Stop
Easy way out Run descriptive stats.
Mean, Median, std dev, shape and spread. Eyeball them stop wasting time.
Stevo
0November 10, 2009 at 1:45 pm #186727Newbee,
Your main issue here is that the pre to post test comparison should show improvement since you are providing instruction to people that are taking the class presumably because they don’t have those skills, correct? So then the question becomes how effective was your training? In other words, your scores should improve by at least two or three units on your scale, maybe more, and that becomes a very subjective assessment.
My suggestion is drop the pre-test and focus the post test on characteristics of instruction, such as clarity, quality of examples, etc. You can then focus on areas where your scores are lower than your expectations.
If you did want a pre-test than you have two blocking variables that need to be accounted for in the analysis. The person becomes a block to account for, as well as the topic.0 -
AuthorPosts
The forum ‘General’ is closed to new topics and replies.