Usability testing questions: How to make your offerings better

Building a successful product, service or experience starts with solving a problem or meeting a need for the end user. Once your product has been developed, new usability problems may come to light and more testing and iteration is needed to iron them out.

But what if we start right at the beginning – what is usability testing? Why is it important? And how can you develop the right questions, usability testing routines and observational skills to get the answers you need?

What is usability testing?

Usability testing helps you understand how easy it is for your product, service or experience to be used by its audience. Can they understand how to put its features to use to meet their goals and solve their problems? And can they actually use those features?

For example, imagine a new design for a doorhandle that looks like a square of metal on the centre of a door. Do the people who want to open that door understand how to use the handle? Do they need to twist it, pull it, or push it? And if they understand how to manipulate it, can they physically interact with it successfully, or is it too large and sharp to turn with bare hands?

The square doorhandle is unlikely to be a usable product, but it’s helpful for illustrating how usability testing helps designers and product owners better meet the needs of users.

At this point, it’s handy to recap the definition of usability too.

In their Usability 101: Introduction to usability, Nielsen Norman Group identify 5 ‘quality components’ of usability. We’ll go through them in turn, applying the doorhandle example:

• Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
  The square doorhandle might not do very well on this criterion, since it’s not familiar or intuitive for the user base.
• Efficiency: Once users have learned the design, how quickly can they perform tasks?
  If the doorhandle’s target audience can physically turn the handle, they should be able to open the door pretty quickly.
• Memorability: When users return to the design after a period of not using it, how easily can they reestablish proficiency?
  Will they remember how to approach the door with the strange handle next time they enter this room?
• Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
  A door they can’t open, a sore hand or a sense of deep frustration might hamper the overall user experience, as well as taking time to recover from.
• Satisfaction: How pleasant is it to use the design?
  This measure is somewhat subjective, but if something is as confusing and uncomfortable to use as a large square doorhandle, it’s unlikely to be pleasant.

Running a usability test involves getting the actual users of a product or service to interact with it so that you can assess how usable they find it.

Usability testing methods usually include asking the test subjects to complete a set of standardised, specific tasks while UX designers watch, listen and take notes or record what happens.

For example, a website usability test could involve a user completing tasks such as clicking through website pages, submitting landing page forms, and testing calls-to-action (CTAs). You might ask them to complete tasks that are more representative of realistic scenarios in a typical user journey, such as making a purchase on an e-commerce website, or finding a given piece of information such as a company address or product price.

Whether you use simple, specific tasks or complex flows, the goal of a usability test is to identify problems, collect qualitative and quantitative data, and determine just how usable your design is once it’s in the hands of its target audience.

Is usability testing just for websites?

Website usability tests are how many people learn about usability testing, and it’s become quite common for web development companies to run usability tests on their products. This might be because our modern understanding of user experience and user satisfaction has been developed through close links between the world of tech and computers and the emerging discipline of user experience and UX design.

However, usability testing can be used for all kinds of products and experiences, digital and non digital. It can be used to find usability issues in a software application, a video conferencing service, an operating system, or even a vehicle’s UI (user interface) system, not just websites. Physical products – like doorhandles – can be tested for usability, as can experiences such as a journey by airplane or the process of booking a restaurant table.

If, however, all usability testing takes place on a website, then it’s referred to as website usability testing.

Why is usability testing important?

Very often, the people who design and develop a product, solution, or service are so familiar with their blind to issues – both large and small. This can also happen outside the company, when customers have a strong, positive view of a brand or business, or have been using it for so long that it seems like its overall usability is higher than it is.

Usability testing aims to get around this by recruiting real users – or those who are representative users in the target audience – who are unfamiliar with the products or services undergoing usability testing. These would-be end users are much more likely to uncover problems – minor or major – than those who have built or used the product or service for some time.

By incorporating usability testing into any product, service, or solution, you can get a more rounded and representative view of the ways in which they suit users and where they need to be improved.

Download our free idea screening and product concept testing ebook

What are the core benefits of usability testing?

Usability testing is used to:

1. Improve the user experience

What do users think of your offering? Is it easy to use? Is it efficient? Are they motivated to explore it further or do they readily turn to alternative options? At the same time, what do they expect? The observations you make during a usability test will help to chart a course for what your offering could be, not what it is right now.

2. Identify pain points

What frustrates your users? Discerning their needs and concerns at the earliest opportunity will help you to reduce development costs and time, all while building a more user-targeted offering.

3. Measure your progress

Running usability testing determines whether your design is getting better over time. Successive usability tests during iteration will – hopefully – show improvement as the design process continues, in the form of easier user experiences and fewer moments of confusion or frustration.

What are the differences between usability testing and user testing?

Usability testing is about whether or not users can use your offering successfully, while user testing is whether or not users need it.

In other words, user testing is testing the utility and necessity of your idea, application, or offering during early development. It’s about understanding how people do something currently and if there’s an alternative (or better) way of doing that same activity that you can help them with.

Usability testing also deals with user behaviour, but it looks at how well the UX design of a product or concept fits in with people’s natural tendencies and habits. It may come later in the design process than user testing, as the target audience has already been established and the concept proven.

Where can I find usability testing participants?

The answer depends on which usability testing method you’re using, and what kind of product or service is being evaluated. For something like website usability testing, you may be able to recruit people for quantitative testing directly from your website traffic.

For something that requires showing users design ideas in detail or asking them to navigate a user interface with complex flows, you’re asking for much more of their time and effort – and investing more of your own. One option is to use a recruiter who will help screen potential participants for you, although they will charge an additional fee. The good news is that you can find 85% of usability problems with the help of just 5 users, according to Jakob Nielsen.

What are some misconceptions about usability testing?

Usability testing is a relatively young discipline and there are a few misunderstandings about its role.

It’s very important to distinguish usability testing and other forms of UX research from user feedback. What real users have to tell you about usability issues is very valuable, but it’s not the same thing as usability evaluation. If you rely only on user comments, even about things users enjoy, you’ll simply end up turning the end user into the designer.

During usability studies the usability researcher will be observing the participants as they work through the usability testing script, paying attention to what they notice, what they miss, which things they find confusing or whether they misinterpret information that isn’t sufficiently clear. The user may be asked about their intentions and strategies as they complete tasks, but their overall opinions aren’t the most important data yielded by the testing session. Leave feedback for the focus group.

Another point to note is that neither usability testing nor user testing are about testing the participants. During test sessions, participants may feel nervous that they are getting things ‘wrong’. It’s part of the UX designer’s role to emphasise that the product or service is under development and that the test subject is helping turn something like a rough draft at an early stage into something more user friendly.

How much usability testing is enough?

If you only conduct usability studies once, you’re likely to be missing out on a host of qualitative research insights. Usability testing is an iterative process, with each new participant helping to reveal more potential additions to your development process. The iterations allow UX designers to devise and test possible solutions to the user problems previous test results have revealed, and to discover whether their new design ideas are successful.

The number of times you’re able to conduct usability testing depends somewhat on your budget. But thanks to remote testing and the wide range of different methods now available, it’s easier than it has been in the past to dedicate multiple rounds of usability testing to your design.

What’s the best kind of usability testing?

To get the most accurate and actionable results while conducting usability testing, you need to think carefully about the main usability testing methods you’ll employ. Each testing method comes with its own pros and cons.

Qualitative or quantitative?

Qualitative usability testing involves learning about individual participants’ experiences, problems and challenges during the usability testing session. It’s helpful for identifying problems with the design.

Quantitative usability testing methods focus on a metric that’s tracked across multiple participants, such as time to complete tasks or success rate. It’s helpful for establishing how well things are working before and after a change, and for tracking progress over time. Data from quantitative testing can be helpful for proving the ROI of testing.

Formative or summative?

Formative research focused on identifying and resolving usability issues, while summative research is about measuring, both qualitatively and quantitatively, how tasks are completed.   

Moderated or unmoderated?

In moderated usability testing, you can follow up on what participants do, allowing you to get instantaneous feedback. Of course, you don’t want to bombard them with questions. Let them relax and do things at their own pace.

If you’re not moderating the usability test, you’ll have to write questions in advance. Make sure to test these questions on others before you send them to your users, as you won’t be there to clarify things if they get stuck.

In-person or remote usability testing?

Another thing to think about is how you want your usability testing to be done. While usability tests used to take place in person, with digital technologies remote usability testing can be used to get results faster. Another benefit of remote usability testing is that you can recruit test subjects from a wider geographical area, and may find participants more willing to participate as it’s more convenient for them. Note however that with remote usability testing you may miss out on some of the finer details such as the user’s facial expressions and body language.

Examples of usability testing questions:

Below are some examples of usability testing questions. Some will be more applicable to moderated environments.

Screening questions — these will ensure you have the right audience for your study

• Can you tell us a little about yourself?
• What is your current occupation?
• What’s your level of education?
• What industry do you operate in?
• What’s your household income?
• Have you used any products in the [insert your industry] before?
• What device do you normally use for [insert solution, e.g. experience management]?

Pre-test questions — these will help to establish the background of the participants

• What kind of device do you use normally?
• How familiar are you with our offering?
  • Not familiar
  • A little familiar
  • Somewhat familiar
  • Very familiar
  • Use it all the time
• Do you use any competitor solutions?

Testing questions — these will allow you to get the most out of the process

• What stops you from completing a task?
• What convinces you to take action?
• What features do you find most valuable and why?
• What features do you find least valuable and why?
• How long did it take you to complete [x] task?
• What are your thoughts on the language used?
• If you were looking for information, where would you expect to find it?
• What motivated you to [specific action] while using our offering?
• What are your thoughts on the design and layout?
• Which features would you use the most?
• How would you prefer to do [insert action, e.g. updating dashboards] instead?

Post-testing questions — these will provide a rounded view of what participants think

• How easy or difficult was it to navigate through our offering? (0-10)
  • What could be improved?
• How likely are you to recommend our offering to a friend or colleague? (0-10)
• Would you choose a competitor offering?
  • If so, why?
• What would you like to see added to the [product, service, or solution]?
• How frequently would you use our offering?
  • Never
  • Rarely (2-3 times a month)
  • Occasionally (2 times a week)
  • Frequently (1-2 times a day)
  • Very frequently (3+ times a day)
• What is your overall opinion of the offering?

How can we help you?

If you’re thinking about putting together a list of usability questions to present to a select panel of participants, our survey templates and idea screening guides can provide you with a foundation.

Completely free and designed to cover four key areas of experience management: customer, product, employee, and brand, our survey templates can help you to uncover the information you need to drive breakthrough change.

As for our idea screening guide, check it out for yourself. In it we cover:

• How to filter out viable product ideas before you invest in them further
• A three-step process for effective concept testing studies with single or multiple ideas
• Usability questions to ask and how to interpret the results.