Rating vs. ranking: which question type is best for your data?

When it comes to surveys, how you ask each question can be just as important as what you ask. Here we look at two of the most popular question types, their pros and cons, and when to use them.

Rating scale vs. ranking scale – what’s the difference?

Rating and ranking are similar ideas, but they’re not interchangeable. Here’s a summary of the differences between rating and ranking scales in research surveys.

• A ranking question asks the respondent to put items in order, usually order of preference. Only one item can occupy each rank, so the respondent has to think carefully and choose which one gets the top spot.

• A rating question is where the respondent assigns a score to each item. The scores can be assigned more than once. So if your respondent thinks that more than one item on your list of options deserves a 10 out of 10, they can express that.

Rating scale questions

In surveys, the most commonly used question types are rating scale questions, where respondents are asked to indicate their personal levels on things such as agreement, satisfaction or frequency.

Rating scale questions are best used when you want to measure your respondents’ attitude toward something. Questions that include “how much…” or “how likely…” are best when differentiation between desirable things is not necessary.

One of the most familiar types of rating scale is the Likert scale, which gives the respondent a range of options to indicate their agreement with a statement and the strength and positivity or negativity of their feelings.

For example

I love corn chips

Strongly agree / agree / neither agree nor disagree / disagree / strongly disagree

You can also use rating scales that express a respondent’s position on a continuum between polar opposite points.

For example:

How hot was your food when it arrived?

Cold / mostly cold / neither hot nor cold / warm / hot

A classic example of a rating scale question is NPS (Net Promoter Score):

On a scale of one to ten, how likely are you to recommend us to a friend or colleague?

very unlikely - 1 2 3 4 5 6 7 8 9 10 - very likely

Rating scale pros and cons

Pro:

Rating scales are familiar and comfortable for a lot of survey participants. They’re relatively low effort to complete, meaning that you can add a number of them to your survey without worrying about causing survey fatigue.

They don’t require the respondent to express themselves in words, which means data is easier to collect and collate.

They lend themselves well to research carried out in offline channels like phone or face to face.

They produce data which is consistent in form and easy to process at scale

Rating questions can be graphical, e.g. rating using stars, which means they can function despite language barriers.

Con:

Rating scales are often not helpful in eliciting the fine-grained data that researchers need to make decisions.

They can be prone to satisficing, which means respondents shortcut the optimal response process by choosing any acceptable answer. For example when rating a selection of desserts, they may find everything equally appealing, or just not care enough to think hard about which one they like best, and as result everything gets the same score.

This sort of non-differentiation can produce data that are not very useful, especially if you’re trying to generate a ranked set of preferences. If you’re asking your respondents about things that may all be potentially desirable, a rating question may not work.

Rating scales rely on the options or statements you have selected being the appropriate ones for your respondents. For example, if you showed a range of classic dessert options to someone with gluten intolerance, they might rate most of them poorly, but for reasons other than taste preference. Had you provided an open field answer instead of or as well as the rating scale, you could have picked up the requirement for gluten-free options on your menu.

Learn more about rating scale questions

Ranking scale questions

Ranking questions allow respondents to identify which items from a list are most and least preferred.

This is particularly helpful when you are forcing your respondents to choose between two things they might otherwise not put in priority order. While some people may like everything on the dessert menu, most will settle for whichever single dessert they prefer most when you ask them to really think about it.

If your respondents will face real-world choices among sets of items, it’s a good idea to allow them to rank their choices in your survey. You can always follow up the ranking question with a rating question to evaluate the strength of the preference.

For example, you could ask people to assign their choices a position on a numbered list:

Please rank the following according to preference, where number 1 is the best and 5 the worst.

1. Ice cream sundae
2. Mississippi mud pie
3. Key lime pie
4. Churros
5. Lemon drizzle cake

Another option is to ask respondents to assign a descriptive category to each item which equates to a rating. For example

Where would you rather take a beach vacation?

• Florida
  most preferred / neutral / least preferred
• Hawaii
  most preferred / neutral / least preferred
• Southern California
  most preferred / neutral / least preferred

A popular format for ranking questions is drag and drop, where items can be physically moved around by the participant in a list to show their rank order. This is an easy and intuitive way for respondents to express their choices.

Ranking scale pros and cons

Pro:

Ranking scales aren’t prone to satisficing in the same way as rating questions, since it’s not possible to give every item the same score. It’s still possible for respondents to straight-line their answers (putting the first item in first place for every list, for example) but if they engage with the question at all they’re likely to make a meaningful choice.

Ranking scales produce consistent numerical data across all respondents. You can express the data as a percentage, which is helpful for reporting and for statistical analysis.

Con:

Ranking scales are of limited use when you have large numbers of items to research. When the scale is too large, the items in the middle can produce unreliable or non-meaningful data, since respondents won’t feel strongly about these middle ranks, and if the list is longer than 10 they quickly become meaningless. It’s the top and bottom of the scale where strong preferences are expressed.

All of the items must be things the respondents have familiar knowledge of. Asking a respondent to rank things they don’t have experience of or proper context for will result in data of limited value. For this reason, ranking may perform better in some contexts – such as a survey of existing customers – than others – for example market research on a brand new product.

Ranking data can tell you what was preferred by most people, but not why they preferred it. It may be difficult or frustrating for participants to weigh up which item they prefer, especially if they like two things equally. It’s relatively high-effort too, so researchers must take care not to overuse this question type.

Learn more about ranking in research methodology, including details on different types of ranking systems.