Surveys

Dear Melanie,
I am not sure what you mean by a “test”, but here is how I would do the comparison of the data.
Since you said you expected the number of responses of 1 to be 1000, I assume you made some assumption about how many responses you would get.  If you got the number of responses you expected, good, but if not, you need to use the percentage method shown below.
First, turn your expected values into % of responses, that will account for the fact that the number of responses you got is not equal to the number you expected.
Now, do a histogram on the responses you got.
Take the number of responses in each bin of the histogram and turn it into % of responses.
Now you can do a direct comparison of the % of responses expected to be a 1 against the actual % of “1″ responses.
I’ll have to do more research on the “test” question.  Do you mean a test to see if the actual is statistically different from the expected?
I’ll get back to you if I find anything.
Sincere regards,
Jim Shelor

Jim:
Thank you for your response. 
The data set looks like this:
Score — Expected — Actual
1 
I followed your advice, and I now have 10 percentages, one per score 1-10.  I charted that in a bar chart and it looks like it has a normal distribution. 

Melanie, this looks like you are barking up the wrong tree. What are you trying to prove? If you are trying to show that 10 is statistically different than 1000, that’s a no brainer. You are taking percentages and treating them as continuous data by using histograms and mentioning normality. You have counts, which are discrete. We don’t know enough yet if it can be treated as continuous. Again, what is your problem? If you are trying to show that the proportion of actual responses is the same/different than the proportion of expected, then you use proportion tests or chi square. You have a bigger problem to deal with because it looks like you have a very small response rate given you expected 1000 and got 10. Jim, don’t send her off on any more wild goose chases until she clarifies what the heck she is trying to do…unless you have some mindmeld that makes it perfectly clear what she means. I for one haven’t got a clue.

Sorry Jim, I have a problem with you recommending all this proportion and percentage stuff and recommending she do a T test. What happened to using tests for proportions? Melanie still doesn’t understand what she is trying to accomplish.

Dear Darth and Emperor,
I think I know what Melanie is trying to accomplish with her analysis.  If I am wrong, Melanie is capable of telling me that.  I do not believe Melanie is devoid of knowledge, as you two have suggested.  I believe she is trying to confirm here knowledge and seeking help from this forum to learn.  There is no more appropriate use of this forum than trying to learn and it deserves you respect, not your snide remarks.
I believe the ordinal data Melanie has collected lends itself to the method of analysis I have recommended to answer the questions she really wants to know.
The company has predicted the number of responses they expect to get for each category of answer (# of 1′s, # of 2′s,…etc).  These numbers are obviously based on a total number of survey responses (just add up the numbers) and a predicted distribution of the responses.
Converting the expected response data into something that can be compared to data from an actual survey that may or may not have the number of responses expected, demands that the data be converted to something that can be directly compared (e.g., % of 1′s or proportation of 1′s, which is the same thing in my opinion).
She wants to compare the grades of 1 expected to the grades of 1 actually received in the survey.  Secondly, she wants to know if the actual survey results indicated the company is performing statistically significantly better or worse than expected.
Having converted each grade category into proportions (% of total), a direct comparison can be made for each grade level.
Plotting a histogram with fit of the porportions of exected versus can graphically show the expected versus actual results.
She stated that the actual results looked like a normal distribution.  Assuming the expected results are also a normal distribution, a T test is an appropriate way to determine if the company’s actual overall performance is statistically significantly better or worse than the expected performance.
The overall performance is the only statistical difference I would be concerned about because the tells me something where a statistical difference between each grade doesn’t mean that much to me.
Obviously, since the counts of 1′s through 10′s are as high as Melanie says for the expected data, the ordinal data can be reasonably assumed to be continuous data and this method can produce the answers they are looking for.
That having been said, please provide some value added input on why this approach is wrong and your suggestions on how to proceed to produce the correct the method and desired answers.  Telling Melanie and I we are incapable of performing this analysis correctly because we are insufficiently educated to know the difference between what is right and what is wrong does not add value to the discussion.
If the suggested approach will work and will produce the desired answers, what difference can it make that a different approach would also work but would also produce essentially the same result?
Have a good day,
Respectfully
Jim Shelor

Dear Melanie,
From the expected numbers you have stated, the expected number of responses your numbers are based on must be in the 15,000 to 20,000 range.  Is this true?
How many actual responses have you received?  I was assuming you have at least somewhere on the order of 10,000.
Sincerely,
Jim Shelor

Gentlemen,
 
I greatly appreciate the 5 plus pages of discussions and take the blame for not being clearer in my question.  You are right that I am here to learn, like many of us.  
 
The expectation set my management was that we would receive over 20,000 responses. They also went a step further in estimating that over 50% of the customers would put us in the 7-8 score area (10 being the best).  The extremes (1, 2, 9, 10) were estimated to receive under 5% each.   
 
When the actuals came in, the response rate is low – under 10%, so this makes it clear that a non-response bias analysis is a must via a follow up survey.  But before doing that, we tried to determine if the actual responses met the expected values.  At a glance, by simply comparing the percentages of 1’s, 2’s, 3’s etc into the total actual count, shows that the estimations were a little far from reality.  The extremes (1, 2, 9, 10) are not around 5% but around 2%.  The bulk (75%) seems to be between 6, 7, and 8.
 
For some scores we obtained a higher percentage than we thought, and viceversa.  Before I do the follow-up survey, is there a way to report any statistical results on these results, understandly that the follow-up survey is a must.
 
In a nutshell, Jim, you put it very well:
“She wants to compare the grades of 1 expected to the grades of 1 actually received in the survey.  Secondly, she wants to know if the actual survey results indicated the company is performing statistically significantly better or worse than expected.”
From your suggestions, thanks again I tried the following (good exercise):

Normality – none of the actual or expected percentages (or normal counts) are normal distributions.  The Anderson Darling P values are low.
Darth’s Two Proportion test for the two columns of percentages (expected and actual).  Not sure why I am getting this error….  Looking into it – Test and CI for Two Proportions: Expected, Actual  * ERROR * Column contains more than two distinct values. 
Darth’s Chi Square test – expected versus actual counts by score (total 20 values, 10 for actual, 10 for expected by score 1-10).  P-value is 0.000.  Now what?
Jim’s Outline and groups histogram – with percentages it is just a big block with no bar on 20 and 30.  Not sure how to interpret.
Another one I tried was the 1-Proportion Test to calculate the upper and lower confidence interval for the proportion of respondents who gave each score.  Then what?

Melanie, thanks for the clarification. Expectation put it quite clearly that organizations luv to do their surveys without sufficient planning ahead of time. Most are fishing expeditions. To summarize what I understand to date. First, you got a 10% response rate so the issue of non response bias might be a concern. As Expectation pointed out, if you have historical data and it leads you to the conclusion that response and nonresponse have not varied significantly in the past, then maybe you don’t have to do a followup survey. If you don’t have that data, then you will have to consider the followup survey. Second, the tests that you have run so far seem to indicate that there is a difference between the proportions you expected and what you received. Ok, so what? Since the actuals represent reality and the expected represent management wet dreams you go with the actual data and then figure out what the hell management was thinking. Third, there is no magical statistical solution at this point. You report the proportions that you actually received and they are what they are. Figure out why they may be lower than you want and start planning on how you will improve customer perceptions. If you have really good operational definitions of what the questions were searching for you might have some direction. If the questions were not worded well then you will still be unsure what to do with the results. Finally, I would stop worrying about any differences between actual versus expected. The only time I might consider it is if the expected was based upon another survey or source of customer perception and the inconsistency might need to be explained. If it is just management expectation, stop wasting your time.

It looks like we are finally over the fenciful statistical stuff, and can move on to what can be done given the quality of the given data.
1. Create simple bar charts of response frequency by category 1 – 10.
2. Review if the categories 9 and 10, and the categories 1 – 4 respectively exceed 5% of responses.
If 9 and 10 have > 5% of responses, good for you, because these responses typically result in loyalty. If the categories 1 – 4 exceed 5% then you need to review that because these are clients at risk. Clients in the categories 5 – 8 are relatively satisfied and the correlations to actions/behaviors are typically low (they have multiple suppliers and switch and/or distribute volumes = share of wallet based on price or convenience factors). 
3. Finally, satisfaction scores in B2B settings have lower predictive validity than satisfactation scores in C2B settings. The main reason is that in many C2B settings the relationship between customer and vendor/supplier is based on simple transactions and switching behavior has low costs. Customers “hedge” by having multiple suppliers and decisions are driven by considerations such as price, convenience of location or other factors. You will need a very high level of satisfaction to induce loyalty or create bundled services and work on contractual relationships.
4. If you want to identify key drivers of satisfaction, correlate the attribute satisfaction scores with the overall satisfaction scores.
5. To get a rough idea of the relationship between level of satisfaction and loyalty, correlate the overall satisfaction score with a scale measuring “recommendation”, “likelihood to continue using the service” and/or “likelihood to purchase other products”.
6. Please go to amazon and buy a basic introductory text on conducting customer surveys. Any of the available books will be helpful to you and your company!

Jim,
You are making me curious – how would you perform a comparison of actuals to the results of a previous survey?
Thanks,
Mel

Mel,
 
The method would be essentially the same one derived here.
But if you have two actual results for essentially the same survey that are 6 months to a year apart, that comparison will tell you if you are getting better or worse.
Sincere regards
Jim

“One shot surveys are useless because they only give you a snapshot in time” … at some point these “useless” surveys had a name: “cross-sectional”. It’s almost too ironic how confused even experienced six sigma professionals are about the measurement of their most critical dependent variable:  customer satisfaction. So much time spent on statitistics and so little knowledge about interpreting changes in those highly revered “satisfaction scores”. There is, after all, a healthy dose of pop culture in this all too scientific enterprise: Six Sigma.

Dear Melanie,
Surveys that cover the same or similiar projects can be used to compute a performance index, such as the mean of the survey responses (only for the similiar questions).
Now, instead of just looking at the data, you can test to see if the change is significant or just random noise.
Say the mean of the new survey is higher than the mean of the last survey.  Conclusion, we are getting better.  Maybe, Maybe not.
If you run a simple T test, you can determine is the means at statistically, significantly different.  If the means are not significantly different, you have insufficient evidence to conclude that you are getting better.  Accordingly, you have insignifican evidence that the improvement actions you have done so far have done you any good and you need to work harder on improvement.
Sincere regards,
Jim Shelor

I’ll leave it up to you and every six sigma professional to dig into the strand of literature that started with Cadoza’s article in the Journal of Marketing (1965), and read up on the last 42 years of literature on the antecedents, determinants and consequences of satisfaction. It is just amazing to what degree quality and six sigma have missed this whole strand of literature and still religiously repeat Kano’s conceptual model from the 1950s. I hope that on its way to six sigmatizing marketing, six sigma may learn a thing or two from an adjacent strand of business research. What a heretic idea that six sigma may actually benefit from learning something about its most cherished assumptions. I assume that we agree and disagree.

Spreadin’ the love…that’s what it’s all about.
 
Glad to see you’re still yourself.

Darth,
That was really not one of your most witty responses (… sometimes you show your emotions too obviously, which only underlines your lack of knowledge in the subject-matter). That little turn back to Deming is even more amusing. By citing Deming you show that your knowledge has not increased since 1950! A great position to lecture this site on the issue!  
Aside from that, I’ll just site the two statements that most blatantly show your lack of more current knowledge of the topic. Again, I’ll leave it up to you and the rest of the site to see what’s so funny about these two statements. As you know it all, you’ll figure it out.
One shot surveys are useless because they only give you a snapshot in time
 
Given how fickle and quickly customers can change perceptions due to a slippage in customer service or a competitor that’s finally got it together, any one time survey has little value except for a specific window of time
 

The one thing I’d add to the discussion picks up on the earlier comment about the 9′s and 10′s. Harvard Business Review ran an article a couple of years back about using something called the Net Promoter score to gauge customer satisfaction.Basically, you ask customers if they would recommend your firm to others. Take the % of 9′s and 10′s, subtract the % of 1, 2, 3′s and track THAT number over time. Some firms actually measure managers’ performance using this metric (Enterprise Car Rental, for example)The other thing I always emphasized to our management when I analyzed surveys were the verbatim comments. Often these were far more rich than the scores themselves.Good luck – sounds like your managers may be wanting some “validation” of whatever it is they’re doing, and the survey was chosen as the weapon of choice.

The net promoter score was a nice tool years ago. It suffers from the same problem that it tried to remedy: method-method artifacts. The area has moved on to measure retention and share of wallet. Today, the net promoter score is a proxy when a company cannot or will not tie the score to actual behavior.

if it is was worth my time, i would give an explanation. however, 1 + 1 = 2. what is there to explain when a complete novice calculates it as 1 + 1 = 3? … isn’t that the standard answer that you give to those novices who ask those very basic questions about stats? i can’t even disagree with you because why answer the obvious? cheers, it’s a beautiful day where i am now. keep up the good work. you all seem to be enlightening each other quite happily.

have a wonderful day!

Darth,
So, glad you responded. I had the same thought (minus the “wet dream”). Why do we try to overcomplicate things? What difference does the expected response make to the actual response? All that is shows is how out of touch with reality managment may be regarding it’s customers.
Thanks for the reality check.

Melanie:I am in the midst of defining some rigour around the analysis of NPS. I am very interested in talking with you.Cheers, BTDT6SigmaGuru(at)gmail(dot)com