iSixSigma

How to Analyze Opinion Survey Comments?

Six Sigma – iSixSigma Forums Old Forums General How to Analyze Opinion Survey Comments?

Viewing 13 posts - 1 through 13 (of 13 total)
  • Author
    Posts
  • #48845

    Kyle QI Guy
    Participant

    Greetings colleagues and experts!
    I have been given a unique assignment (for me at least) in which I seek your expert minds for guidance. Recently a related company posted an online employee opinion survey to their employees. As with most surveys it utilized a common Likert scale from which data could be analyzed and reported. That part is easy, well relatively easy. The striking piece to this puzzle is the accompanying comments. I have been asked to analyze and report the results of the comments. Each of the 35 questions in this survey allowed for free-text comments. There are over 14,000 comments!! I suppose that really would not be too terrible if I had these data in some digital format, however all the survey company would release to me was the printed (or PDF of the printed report) comments and the related question.
    So here I sit with all of these comments and little idea how to begin to determine a way to scientifically analyze this information. This is a very frustrated workforce and has a lot to say about a lot of things. Even though they may comment about the question at hand, they often also continue to comment about both related and unrelated things. It is evident that these workers’ voices need to be heard. How can I make that happen in a scientific and analytical manner? There is so much subjective data!
    Does anyone have any experience with managing survey comment data?Thanks for any help you can offer. Have a great day!

    0
    #165720

    Brandon
    Participant

    Kyle, what a challenge & unfortunately I know of no wat to assist you.
    However, on another level, I would suggest mngt. read all the comments (with anonimity of submitter of course). There’s a lot of info there and reducing it to groups of comments and statistics will lose much of the message.
    If this is just a drill to pretend to care – they won’t do that – too bad.

    0
    #165729

    RickL
    Member

    Are you allow to use sampling on this project? I think you may consider to draw a sampling plan for data analysis.
    You are having 400 survery forms with 35 questions each? you may consider a sampling of 30 or 50 questionaire on random selection to start with. Then try to group the comments into categrory and continue with the data breakdown.

    0
    #165732

    George Chynoweth
    Participant

    There a couple of ways you can start thinking about this. Since you have such a large data set, focus on the extremes. 1. For each item, highlight the comments of those who gave an item the lowest rating, as well as those who gave it the highest. Select comments that make a suggestion for improvement, or provide some value. Ignore the comments without constructive value. Look for themes among the satisfied & dissatisfied comments. 2. If your 35-item survey is broken down into categories (e.g., work environment, leadership, communication, etc.), (and assuming you have a good Likert scale), add up the scores for each section. Within each section, select the responses that are in the upper & lower ranges, say in excess of +/- 1 s.d. As above, select the comments that have value – look for themes. Since the survey folks are being uncooperative, you’re going to have to do a bunch of clerical work – either one of these ideas should cut that load by 60%, at least, without suffering too much information loss. hth,
    george

    0
    #165837

    monts
    Participant

    you may want to use a an advanced technique called MTS ( Mahalanobis Taguchi System ). This helps when parameters are interdependant and normal hypothesis techniques will not work

    0
    #166073

    Dai
    Participant

    About a year ago one of our KM guys showed some software called Themescape that we were using to analyse customer feedback comments. It gave a sort of contour map as an output; words with most occurences got highest contours etc…

    0
    #166074

    Julisum
    Participant

    Kyle,
    go back to your survey provider and find out why they can’t give you the data electronically, after all you are the customer here.  I had a similar problem recently with an employee survey where I was told we couldn’t have the data electronically, after a little bit of probing I discovered that the reason was down to data protection and not because they didn’t want to give me the data.  In the end we came to a compromise whereby they removed all questions that could identify the employee and sent me the rest of the file.  If you can get them to agree to this, and explaining why you need it might help, you can then at least start to sort the comments into groups as suggestd in some of the other posts.

    0
    #166085

    Perryman
    Participant

    I agree with George here and would also add a couple of suggestions.
    If the comments are related to the Lickert score, concentrate first on the extremes:
    1. Find trends or themes in the comments.  Try grouping them with affinity techniques.  Any major issues pop up?
    2. Look at frequencies of recurring comments to build a type of pareto. ie. if everyone is saying the same thing when they are really satisfied or dissatsfied about something, chances are it is worth highlighting.
    For comments not related to a Lickert score – I don’t know how your survey was built but sometimes they just ask for general comments – follow the two steps above but unfortunately, for the second step, you have no way to link this to any quantifiable data from the scale so that part of the analysis cannot be done. However, you may still find some interesting information such as recurring suggestions for improvement, etc.
    I have done this type of analysis in the past – usually for training surveys – but never to the scale you have.  You have quite an undertaking, I suggest you get some help with the grouping exercise.  
    Good luck,
    Patch 

    0
    #166095

    Ron
    Member

    an affinity diagram is a easy tool to use in this case. The primary purpose of this tool is to co-locate similar comments into groupings. Once they are group in similar themes you can begin by refining the larger grouping etc etc.
    When you have compiled the statistics on how many comment fell within the broader group you cqan use more classic tools to analyze or report the results. 

    0
    #166099

    Kyle QI Guy
    Participant

    There have been many fabulous suggestions. It’s funny that I didn’t think Affinity diagram even though I use it so often, mostly for brainstorming and occasionally early stage root cause analysis. I can get started and see how things begin to pan out.
    Needless to say this is an extremely painful process. I literally retyped all weekend (2, 10hr days) and entered over 40,000 words into MS Word. The funny thing is that I only got through 6 questions, there’s 35!! These employees have a lot to say and the Likert score did not hold enough weight to them it seems. My personal opinion is that this was a poorly designed survey tool even though this a “professional” and experienced company.
    Regarding my role in this and why they will not give me the electronic results. I work for an organization that oversees the medical aspects of this service. We are not supposed to be involved in the operations of this organization and have been told time and time again to essentially mind our own business and let them manage their staff. Well, after a very long time of telling them that their employees are miserable with their leadership (plus, added pressure of our municipal leaders who hold their contract) they finally surveyed the staff and have been shocked at the results. Now, I’m trying to quanitify the findings because we have sensed that they’re downplaying and dismissing the results. All of that to essentially say, they still will not give me an electronic copy of the results because of this complex relationship and political crap.
    Having read through all of this I cannot allow their voices to go unheard.
    Thank you all for your guidance, you have been most helpful. I will continue to watch this posting if you have any other comments.
    Hope everyone has a happy and safe holiday season!
    -Kyle

    0
    #166158

    fevila
    Participant

    Kyle,
     
    You may want to try one of the many programs out there that can convert from PDF into other formats as MS Word.
    One I have used quite succesfully is Scan Soft.
    Best of luck

    0
    #166164

    annon
    Participant

    With over 14000 surveys (or was it questions) to go through, I would give some thought to using a team/based approach to help reduce the workload and begin building concensus. 
    For example, segmenting the surveys according to areas of control (ie product line, dept, function, region, etc) and applying a WorkOut format in conjunction with, say, an IRD exercise might be a useful way to start.
    Good luck.

    0
    #166181

    Deanb
    Participant

    Kyle,I agree with Brandon that statistics may not be your best path forward here. The highest goal of a survey IMHO is to point you to look deeper. Surveys give you aggregated sentiments, not absolute truths. WHY people gave the answers they gave is often more relevant than the answers themselves. A survey pro I once teamed with told me comments need to be evaluated on their face, not from a statistical lense. If they are true, relevant, and important, they represent a higher form of information than statistics can give. For example, if someone commented that their operation is knowingly shipping nonconforming product and the company was about to lose its biggest customer, wouldn’t that warrant a deeper look, regardless of the survey statistics? Good luck.

    0
Viewing 13 posts - 1 through 13 (of 13 total)

The forum ‘General’ is closed to new topics and replies.