Your guide to user testing

What is user testing?

User testing is the practice of putting end users into contact with your products or services and taking note of their responses and reactions.

By conducting user testing, you can gauge how positive the average customer experience is and whether or not the product or service is performing as you intend. This is invaluable for finding flaws and targeting improvements. It can also help you discover ways of using your products that you may not have had in mind at first, but that users find beneficial.

User testing can be done even before you have a real product prototype or service design in hand. To do this successfully involves testing at the product prototyping stage, or even as part of product concept testing when ideas first emerge.

The practice of user testing can also be used once a product is live and on sale. Instances of this include general user feedback, which can guide and influence a re-release or upgrade.

UX, usability, UAT… what’s in a name?

When you start delving into the subject of user testing, you’ll encounter lots of similar-sounding terms, some of them used almost interchangeably. To keep yourself focused, it’s worth knowing a little about the different definitions so that when you start researching the topic you can hone in on the area you’re interested in.

When someone mentions user testing, they might mean:

1. User research

This phrase usually means discovering the user base for a product or service. It may be something that happens at the idea stage as a way of testing whether there is an appetite for the idea, and if so, what the eventual end-user wants to get out of the eventual product.

Learn more about user experience research

2. Usability testing

Exposing a product, prototype or work-in-progress to its intended users as a way of gauging how easy it is to use. The aim is to find areas where usability could be better and to direct product development towards these areas.

Learn more about usability testing

3. UX testing

UX (user experience) is everything about a product or service that has a direct impact on its end-user, with usability being just one part of the picture. ‘UX testing’ is often used interchangeably with ‘usability testing,’ but it’s actually a much broader scope of discovery.

4. User acceptance testing

User acceptance testing (UAT) has the clearest definition of the bunch. It happens towards the end of a product design process and can be seen as a kind of QA testing. Users test products or services and researchers gauge the overall product performance against a predefined set of criteria. For example, UAT for a new ice-cream dessert might involve checking standards for flavour, texture, and aroma have been met.

Why so many names?

One reason for the plethora of names for user testing is that user experience is a relatively new discipline that’s not yet widely understood. Another factor is the field is constantly developing – it hasn’t yet arrived at a common definition for the different practices and ideas that feed into user-centred design. On top of that, different UX professionals in different businesses have their own ideas about what true usability is, and how to achieve it.

Throughout this article we’re going to use the term ‘user testing’ to describe a range of user research methods, since – naturally – many kinds of user-focused tests can be done using Qualtrics tools and software.

Why do user testing?

As we’ve outlined there are lots of benefits user testing brings, whether you employ it at the first stages of idea generation or when you’re revamping a much-loved hero product.

Here’s a deeper dive into some of them.

1. Save money on production

User testing provides valuable insights at the early stages of product development before more substantial investment is made. It could make the difference between starting full-scale production based on an untested set of assumptions – which turn out to be wrong – or launching a product users want and need.

2. Make sure your product is fit for purpose

As a product’s owner, you’re its biggest advocate – and that’s the way it should be. But with user testing you can make the concept stronger, find more compelling messaging to market it with, and maybe even discover new uses and benefits by taking on board the feedback of unbiased end-users.

3. Identify usability issues before they cause widespread problems

Product sales and income are one thing, but brand equity is something much more difficult to achieve. It’s also more powerful and lasting when you secure it. Usability issues with your products and services can create negative experiences for users that result in lost loyalty and even create detractors of your brand. Thorough usability testing allows you to discover these pitfalls before they do widespread damage to your brand.

4. Test the success of a change or upgrade

Collecting user feedback after changing a product or service can help you gauge how successful the move has been. It can also help you assess your ROI, particularly if you collect user feedback over time and compare it to other indicators like sales, social ranking, and NPS.

5. Capture emotional reactions to different aspects of the product

Feedback from real people is emotional by nature. User testing can give you a picture of the kind of sentiments your product or service creates in its users, and help you gauge whether those feelings are in tune with your brand values and marketing approach.

Types of user testing

Here are a few common methods of user testing, and how to implement them.

Method: A/B testing

Also known as split-testing, this method involves dividing your users into two or more groups and exposing them to different variants of something so you can see which performs best. Most commonly, A/B testing is done with elements of a website, email or other digital experience, such as two different headlines or CTA buttons in different sizes.

Requires: At least two versions of the thing you’re testing, and a user environment (usually a web page or email) where all the other variables are the same.

It’s for you if: You want to optimise a single element of something as a way to fine-tune your approach, and you have the resources to create different versions.

Strengths: Can be used across a large sample size without increasing your costs. Gives quantitative data about performance that’s easy to compare.

Limitations: Not suitable for new ideas, complex changes or in-depth assessments of user experience. Works at a granular level. It’s best for making small tweaks to an otherwise confirmed approach.

Method: Prototypes

A useful method at the idea generation and concept testing stage. Researchers and designers create prototypes, which can be virtual, mocked up in 3D or even written and scamped out on paper. They gather feedback from target users via interview, survey, or using a tool like Qualtrics Concept Testing Tool.

Requires: Time and resources to create a prototype, and a pre-selected group of users to test it with.

It’s for you if: You have an idea you feel confident and excited about and you want to build a stronger case for putting it into production.

Strengths: Could save time and money by providing useful insights early on.

Limitations: It’s dependent on the prototype being a fair representation of the end product, so there needs to be rigorous attention paid to getting key features and ideas across to the user clearly.

Method: Usability lab studies

A UX professional works one-on-one with the user as they experience the product or service – usually on a computer or mobile device. The user acts either with direction from the researcher or using their own initiative to navigate through a website or app and complete a task. The UX professional notes down their observations of the user’s behaviour as well as their feedback.

Requires: UX lab, equipment and researcher time, suitable participants.

It’s for you if: You’re looking for an in-depth analysis of an experience and a deep understanding of your end-user.

Strengths: Working one to one can provide rich information to the researcher and allow them to capture factors and ways of thinking that may not come across in other methods.

Limitations: Quite labor-intensive and can be costly. Difficult to achieve a large sample size without prohibitive time and money being spent.

Method: Remote testing

This type of research works in a similar way to usability lab testing, but is done remotely using software that allows users and UX researchers to remotely share a desktop. They communicate via phone, chat or video. There are also variants of this type of testing where the user works alone and then reports their findings.

Requires: Remote testing software and a suitable group of participants.

It’s for you if: You’re looking for detailed human input without the outlay of a usability lab, or you want to test your product with users across a geographically broad area.

Strengths: Cheaper for the business and more convenient for users compared with in-person usability testing, and can also be more versatile in terms of time and location.

Limitations: Researchers don’t get the same level of insight and empathy with users, and some detail may be lost from their findings.

Method: Surveys

A questionnaire is circulated to a group of users or potential users to gather their feedback. It can then be analysed to develop overall findings.

Requires: A survey platform and a sample of suitable users.

It’s for you if: You want to gather data across a broad range of users, and combine different forms of data including qualitative and quantitative answers.

Strengths: Surveys are an affordable way to get a relatively in-depth insight from a large number of users, and you can cover a number of topics and ideas in a single session.

Limitations: Asynchronous, relying on a user’s memory of their experience which may not be accurate. You only get the user’s interpretation of their experience rather than a researcher’s direct observation.

Method: Intercepts

A contextual popup or window on a website that appears at a strategic point in the user journey. If the user engages with it, it poses messages or questions to collect their feedback.

Requires: Suitable site intercept software.

It’s for you if: Your research is specifically website-based, and you’re looking to gather feedback from certain user groups at a specific point in the journey.

Strengths: Powerful targeting capability that allows you to hone in on a particular user segment based on existing customer knowledge. Appears in context, so the responses aren’t reliant on user memory.

Limitations: Not all users will engage. As with other remote forms of research, the data is based on user feedback rather than researcher observations.

Method: Focus groups

One of the longest-established forms of user research, this method involves gathering a representative group of users for a guided discussion of your product or concept.

Requires: A space to host discussion, a suitable group of participants, and the time and expertise of a facilitator and/or researcher.

It’s for you if: You’re looking for a deep dive into your users’ thoughts, feelings, and opinions about a product, and how it fits into the context of their lives.

Strengths: The format is dynamic and flexible, allowing free exploration of relevant topics as they arise – including things which may not have been foreseen at the start of the discussion.

Limitations: Participants influence and interact with one another and will also be affected by their environment, and the sample number is very small, so it can be difficult to achieve a representative picture of your wider user base. The person moderating the group must be skilful to avoid the conversation going off track.

The value of human input

Whichever form of user testing you choose, you’ll benefit from the views, insights, and feedback of real people – the same real people who will ultimately be instrumental in taking your business forward. By taking note of their experiences at all stages of your product journey, from early prototypes to newly rolled-out products to old favourites, you’ll have the potential to create true and lasting value for everyone who comes into contact with your company.