|
|
 |
Customer Satisfaction Survey Advice
|
|
|
|
|
|
Posted by: george chynoweth
Posted on: Friday, 28th September 2007, 2:06 PM.
I have to differ with much has been said here. A good survey is reliable and valid and can provide actionable and strategic information that will tell you how and how not to allocate your resources. However, on the face, it is difficult to tell a good survey from a bad one, and most people think they can develop a survey – and often do, with disastrous results. A poorly designed survey can actually provide misinformation or even disinformation. It understandable why surveys are regarded as untrustworthy as a source of real information – and it is an unnecessary state of affairs. Below are some points to consider.Regarding some potential origins of the questionnaire:
1. A typical starting point for a customer or employee satisfaction questionnaire for a quality organization would be the section of the strategic business plan that deals with customers and employees. Look at the key business drivers and derive your metrics from them. Develop questions you think will measure theses metrics. Don’t do this in the dark – ask managers & employees what they think of the questions you develop. Edit as necessary. Use common sense.
2. In developing “an internal research tool” for establishing baselines, ask your internal customers what works & what doesn’t. You’ll get varying and possibly contradictory opinions, but your not looking for consistency, you’re looking an item pool for your questionnaire. I worked for DoD for many years, and the best commanders I saw were the ones who got out from behind their desks and into the trenches with their soldiers. They went to the motor pools, firing ranges, security perimeters etc., and asked questions of individual soldiers: "How's it going? What do you need? What problems are you having?". Do the same. As implied elsewhere, individual attention provides credibility and PR value, as well as information for your item pool.
3. Be sure to add at least one “bottom line” question (which will serve as the dependent variable in the regression analysis), such as “Are you satisfied with …”. When running an external customer survey, add two more questions like “Value for the money”, and “Will you recommend us to …”. These will serve as dependent variables for two additional regressions - run 3 separate regressions, one for each dependent var. These additional questions will allow you to get at customer loyalty and customer value in addition to satisfaction.
4. Include a qualitative item such as “Give us just one idea on how to improve …”. This is very focused, and almost everyone can come up with one idea. You'll get a lot of data.
5. Once you have an item pool, run it by some colleagues/managers who are familiar with the metrics you’ve chosen. Get their input regarding which items best measure what you’re after. Keep in mind that you need to balance your objectives with respondent burden (time & thoughtfulness). Keep the questionnaire short & simple. Scaling: This is where many surveys really get it wrong.
1. Data contain information, they are not information themselves. E.g., 200 PSI is twice the pressure as 100 PSI, but 200 degrees Fahrenheit does not have twice the heat as 100 degrees F. The difference is due to scaling, and the lesson is, you need to design the type of data you need before you start collecting it.
2. Likert developed his scale in 1929 using 5 points with a descriptor for each point. Looks easy. However, Likert’s scale is balanced (an equal number of positive and negative ratings), and each rating point is visually equidistant from its neighbor. The equal distance requirement supposedly provides interval level data as opposed to ordinal, thus allowing the more powerful parametric analyses to be used. Naïve survey developers usually overlook these FUNDAMENTAL characteristics, and introduce systematic bias into their analyses which just wreaks uncontrolled havoc on explained variance. A particularly deadly scale is something like “Excellent – Very Good – Good – Fair – Poor”. It has 4 positive valuations and 1 negative. It is rarely equidistant, and it truncates the response spectrum – it is akin to saying “Do you agree with me, or do you AGREE with me.” Trash. We’ve come a long way since 1929, but most folks continue using, and even bastardizing, Likert’s scale.
3. When it comes to measuring opinions, attitudes, beliefs, etc., we have been using a natural rating scale for decades that everyone understands – they even made a movie about it: “10”. As a kid we used to rate July 4th fireworks on a 10-point scale – I’m sure most have done something similar. I don’t understand the continued debate about how many points a survey should have, and whether or not it should contain a neutral point. A 10-point scale provides a broad enough response range that a neutral point is unnecessary. There are also 2 key points with this scale: the rating points must be equidistant, and you should use only 2 descriptors, one for each end of the scale (e.g., Excellent … Terrible). The respondents will be able to fill in the blanks, so to speak, without intervening value judgments provided by the survey developer. This scale has been shown to improve reliability considerably. In my own work, I have never had an internal reliability index (Chronbach’s alpha) below .90 – a typical survey is between .7 and .8. Reliability, in this context can be thought of as Precision of Measurement. 90% measurement precision for a survey is excellent.Data Analyses:
1. Of course, examine the data and run the standard descriptives. Look for outliers, unusual groupings, normality, etc. Get a feel for your data before you ever start to analyze it – you will better understand the results, and be more attuned to problems should they arise.
2. Three basic analyses will provide a wealth of information. (1) Convert the item means to percentages and consider them as Performance data (from the respondent’s perspective). (2) Calculate the Coefficient of Variation (standard deviation / mean) – this will allow comparison of variation among all items. (3) Run a regression analysis on each dependent variable. The regression will weight each item regarding its impact on the dependent var – this is “Critical to Satisfaction” information (and Critical to Quality, and Value). You now have Performance, Variation, and CTS (CTQ) data. Is this starting to look familiar?
3. Analyze the qualitative data looking for common themes. Once you have themes, examine them using the Performance, Variation and CTS data. If you’ve collected any demographic data, you can sort your target groups with the same procedure. You’ll find what’s important, what’s not, and to whom it matters.
4. When you run the regression(s), you’ll have a statistic called the Multiple R Square. This tells you how much variance in the dependent var was explained by the independent vars. It is a measure of validity. Convert this number to a percentage, and you have an estimate of accuracy. A good survey will be at 85% or higher – most don’t reach 50%. So, if you have an accuracy of say, 88%, this means that 12% of the variance in the dependent var was not explained by the independents. This is due to measurement error, sampling error, scaling error, etc. If you’ve done good developmental work, these errors (residuals) will be normally distributed, with a mean of 0 and a standard deviation of 1. Be sure to examine these residuals. If they aren’t normally distributed, you’ve introduced bias, or uncontrolled error, into you results. This is bad news as we don’t know where or how this error is affecting the results – which is misinformation, making the results untrustworthy.Finally, go back to the beginning. Include an “informed consent” with the survey. Tell them whose doing the survey, why, and what will happen with the results. Tell them if it’s confidential or not. Who can they contact if the have questions? AND, provide feedback – tell them where they find the results (website, newsletter, personal notification, etc.). This is great PR and will help considerably towards the success of the next survey.“Anything that exists, exists in some quantity, and therefore can be measured.” LL Thorndike, 1932. I agree completely. :) Enjoy.
Message Thread: 
Return To Discussion ForumPost A New MessageRead the Forum Guide to Good Etiquette
"The Bottom Line" Links
|
|
|
