Expert Commentary

The Psychometrics of Customer Feedback

Survey structure can trigger undesirable psychological phenomena, but asking the right questions first can limit these risks.

By Jason Lee

Published on 七月 12, 2016
min read


In surveys, the sequence of questions matters—a lot. We often begin our marketing and customer research surveys with Net Promoter Score® questions. These are overall brand rating questions that ask customers “How likely are you to recommend…?” on a 0-to-10 scale, and are followed by one or two open-ended questions. A client recently wondered why we ask the NPS® question before any other questions about the customer’s experience with, or perception of, the brand. “Wouldn’t asking the NPS question later in the survey, when the respondent has had more time to think about the brand, provide more meaningful and richer answers?” This client’s intuition is only partially correct. Read More Advanced Analytics Expert Commentary Success with advanced analytics requires both technical know-how and a thoughtful approach. In this series, Bain's experts offer practical advice on some of the most common data issues. Best-practice survey design sets the question order to help respondents reflect on and recall the information that a company is most interested in learning. As the client suggested, we usually ask customers to report past purchases, occasions, and decisions before asking them to describe what matters to them in making those decisions or what they might do in the future. This order helps respondents frame the topic, activate memories and reflect on the broad range of their experience with the product or service. What makes the NPS question different? Survey structure can trigger some undesirable psychological phenomena as well. Priming occurs when the thoughts and context of an earlier question influences how people interpret or reflect on subsequent questions. Anchoring occurs when respondents adjust their responses to subsequent questions using an earlier response as a reference. Rationalizing tends to increase as customers answer questions about their opinions, past experiences and decisions. Many of our surveys include sections that ask respondents to recall and report on specific experiences, such as using a product, talking with a service representative, visiting a store or branch, paying a bill or getting a product repaired. Imagine what happens when we follow these sections with a question on overall satisfaction or quality. We will have primed the respondents with memories that may have been forgotten, or at least were not at the top of their minds. Respondents may reflect on their answers to these experience-specific questions and find a rational “overall” rating that summarizes these experiences. They may anchor the “overall” rating up or down relative to the rating they gave on an earlier question. We ask the NPS question to gain insight into customer loyalty, as it might influence future decisions and word-of-mouth behavior. And we ask them first, because we want to hear customers’ top-of-mind impressions. By asking NPS questions early, we intentionally avoid memories of experiences that do not come to mind immediately. We also limit the risks of priming, anchoring and rationalization. To test these effects, we recently collaborated with J.D. Power & Associates to do a research-on-research study of US credit card brands. Half of the respondents provided overall brand ratings at the start, and the other half at the end of a survey that also probed specific elements of the product and service experience. We analyzed the results to identify which specific experiences best correlated with overall brand ratings. The results were largely the same whether we asked about overall brand ratings at the start or at the end of the survey. However, we did see that less memorable experiences, such as billing, had higher correlations with overall ratings when asked at the end of the survey. More emotional or brand-related factors, such as reputation and customer orientation, showed a higher correlation with overall rating when asked at the front of the survey. To sum up, asking the NPS question at the front of a survey better captures memorable experiences and overall loyalty to a brand. Asking NPS at the end tends to capture a summary evaluation of experiences that may be important, but not at the top of the customer’s mind. Jason Lee is a principal with Bain & Company’s Advanced Analytics practice. He is based in Los Angeles. Net Promoter Score® and NPS® are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.

In surveys, the sequence of questions matters—a lot. We often begin our marketing and customer research surveys with Net Promoter Score® questions. These are overall brand rating questions that ask customers “How likely are you to recommend…?” on a 0-to-10 scale, and are followed by one or two open-ended questions.

A client recently wondered why we ask the NPS® question before any other questions about the customer’s experience with, or perception of, the brand. “Wouldn’t asking the NPS question later in the survey, when the respondent has had more time to think about the brand, provide more meaningful and richer answers?” This client’s intuition is only partially correct.

Advanced Analytics Expert Commentary

Success with advanced analytics requires both technical know-how and a thoughtful approach. In this series, Bain's experts offer practical advice on some of the most common data issues.

Best-practice survey design sets the question order to help respondents reflect on and recall the information that a company is most interested in learning. As the client suggested, we usually ask customers to report past purchases, occasions, and decisions before asking them to describe what matters to them in making those decisions or what they might do in the future. This order helps respondents frame the topic, activate memories and reflect on the broad range of their experience with the product or service. What makes the NPS question different?

Survey structure can trigger some undesirable psychological phenomena as well. Priming occurs when the thoughts and context of an earlier question influences how people interpret or reflect on subsequent questions. Anchoring occurs when respondents adjust their responses to subsequent questions using an earlier response as a reference. Rationalizing tends to increase as customers answer questions about their opinions, past experiences and decisions.

Many of our surveys include sections that ask respondents to recall and report on specific experiences, such as using a product, talking with a service representative, visiting a store or branch, paying a bill or getting a product repaired. Imagine what happens when we follow these sections with a question on overall satisfaction or quality. We will have primed the respondents with memories that may have been forgotten, or at least were not at the top of their minds. Respondents may reflect on their answers to these experience-specific questions and find a rational “overall” rating that summarizes these experiences. They may anchor the “overall” rating up or down relative to the rating they gave on an earlier question.

We ask the NPS question to gain insight into customer loyalty, as it might influence future decisions and word-of-mouth behavior. And we ask them first, because we want to hear customers’ top-of-mind impressions. By asking NPS questions early, we intentionally avoid memories of experiences that do not come to mind immediately. We also limit the risks of priming, anchoring and rationalization.

To test these effects, we recently collaborated with J.D. Power & Associates to do a research-on-research study of US credit card brands. Half of the respondents provided overall brand ratings at the start, and the other half at the end of a survey that also probed specific elements of the product and service experience. We analyzed the results to identify which specific experiences best correlated with overall brand ratings. The results were largely the same whether we asked about overall brand ratings at the start or at the end of the survey. However, we did see that less memorable experiences, such as billing, had higher correlations with overall ratings when asked at the end of the survey. More emotional or brand-related factors, such as reputation and customer orientation, showed a higher correlation with overall rating when asked at the front of the survey.

To sum up, asking the NPS question at the front of a survey better captures memorable experiences and overall loyalty to a brand. Asking NPS at the end tends to capture a summary evaluation of experiences that may be important, but not at the top of the customer’s mind.

Jason Lee is a principal with Bain & Company’s Advanced Analytics practice. He is based in Los Angeles.

Net Promoter Score® and NPS® are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.