Conducting a Usability Test

What is a usability test?

A usability test is an empirical method for evaluating the usability of a product.

The focus is not whether an interface appears understandable from the perspective of designers or developers.

The decisive question is:

Can real users achieve their relevant goals successfully with the product?

Participants perform typical tasks. During the session, researchers can observe, for example:

• whether tasks are completed successfully,
• which errors occur,
• how much time is required,
• where uncertainty arises,
• what users expect,
• and how users experience the interaction.

Usability tests are therefore particularly suitable for making concrete problems in the user experience visible.

How does a usability test work?

A usability test can be divided into seven basic steps:

1. Define the research objective
2. Determine the context of use and target group
3. Develop realistic tasks
4. Recruit suitable participants
5. Prepare and pilot the test procedure
6. Conduct the usability test
7. Evaluate and prioritize the results

1. Define the usability test objective

The process starts with the question:

What should the usability test investigate?

Possible research questions include:

• Do users understand the registration process?
• Can users find a specific product?
• Which problems occur during the ordering process?
• Is the navigation understandable?
• Can users use a central function without assistance?

A clearly defined research objective makes all subsequent steps easier.

It helps in particular to:

• select relevant tasks,
• determine suitable participants,
• define appropriate measures,
• and interpret the results meaningfully later on.

2. Determine the context of use and target group

A usability test should reflect the actual conditions of use.

First define:

Who are the relevant users?

Possible characteristics include:

• experience with comparable products,
• relevant subject knowledge,
• frequency of use,
• technical skills,
• and specific needs of the target group.

What goals do users pursue?

A test should focus on relevant goals.

For an online shop, for example, the evaluation could examine whether people can:

• find a specific product,
• compare products,
• find shipping and return information,
• and successfully complete a purchase.

Under which conditions is the product used?

Relevant conditions can include:

• desktop or smartphone use,
• a quiet or distracting environment,
• private or professional use,
• time pressure,
• and specific technical requirements.

3. Develop realistic tasks

The quality of the tasks is one of the most important factors in a meaningful usability test.

A good task should:

• reflect a realistic goal,
• be relevant to the target group,
• be phrased as neutrally as possible,
• and not prescribe a solution path.

For example, this task is less suitable:

“Click on ‘Book tickets’ and then select Berlin.”

This wording already reveals the solution path.

A better formulation would be:

“Imagine that you want to book a train journey to Berlin for next week. How would you proceed?”

The second formulation makes it possible to observe the participant’s actual behavior.

4. Select participants

Participants should match the actual target group as closely as possible.

The number of participants is not the only important factor.

Also consider:

• How heterogeneous is the target group?
• Which different usage scenarios exist?
• Is the study exploratory?
• Should quantitative measures be estimated or groups compared?

Small qualitative usability tests can already reveal important problems.

However, larger samples are generally required when precise quantitative statements are needed, variants are compared or differences between user groups are investigated.

5. Prepare and pilot the test procedure

The complete test procedure should be prepared before data collection begins.

This includes:

• greeting and introduction,
• explaining the test procedure,
• tasks and scenarios,
• success criteria,
• observation criteria,
• technical preparation,
• recording the session,
• and closing questions.

A pilot test is particularly important.

Piloting can reveal, for example:

• unclear task wording,
• technical problems,
• unsuitable time limits,
• missing instructions,
• and ambiguous success criteria.

6. Conduct the usability test

During the test, participants work on the prepared tasks.

Different types of data can be collected.

Behavior

Researchers can observe, for example:

• navigation,
• clicking behavior,
• search strategies,
• errors,
• abandonment,
• and detours.

Performance

Possible measures include:

• task success,
• task success rate,
• completion time,
• number of critical errors,
• and need for assistance.

Subjective experience

Researchers can also record:

• perceived ease of use,
• satisfaction,
• trust,
• frustration,
• expectations and feedback.

Think-aloud

In a think-aloud study, participants are asked to verbalize their thoughts while completing the task.

This can provide insight into:

• what users expect,
• how they interpret information,
• and why they make particular decisions.

However, the method should not be equated with natural usage behavior, because verbalizing thoughts can itself influence the interaction process.

Moderated or unmoderated?

Moderated usability tests

In a moderated test, a moderator accompanies the session.

Advantages:

• follow-up questions are possible,
• observations can be explored in greater depth,
• misunderstandings can be interpreted more effectively,
• and complex tasks can be examined.

Disadvantages:

• greater organizational effort,
• longer implementation time,
• and possible influence from the moderator.

AI

AI-generated image

Unmoderated usability tests

In unmoderated tests, participants complete the tasks independently.

Advantages:

• flexible implementation,
• less moderation effort,
• larger samples are easier to reach,
• and good scalability.

Disadvantages:

• no immediate opportunity for follow-up questions,
• less context for observed behavior,
• and greater demands on task clarity.

7. Evaluate the results

The evaluation should go beyond individual observations.

A systematic process can include the following steps:

Identify problems

First collect situations in which:

• tasks were not completed successfully,
• errors occurred,
• uncertainty became visible,
• unnecessary detours occurred,
• or expectations were not met.

Describe problems

Each problem should be documented as concretely as possible.

For example:

Users cannot find the shipping cost information because they do not expect it under “Delivery”.

This description is more useful than:

“Improve navigation.”

Analyze causes

Then investigate:

• Why does the problem occur?
• Which interface elements are involved?
• Which expectations are violated?
• Does the problem affect multiple users?

Prioritize problems

Not every problem is equally relevant.

Prioritization can consider:

• problem severity,
• frequency,
• number of people affected,
• impact on goal achievement,
• and impact on central product goals.

What does a usability test measure?

A usability test can examine different dimensions of usability.

This structure connects directly to the framework of ISO 9241-11.

Typical mistakes in usability tests

Unsuitable participants

If participants do not match the target group, the results can only be transferred to actual usage to a limited extent.

Leading tasks

Tasks should not already contain hints about the expected solution path.

Missing success criteria

Without predefined criteria, it is difficult to decide whether a task was completed successfully.

Missing piloting

Problems with tasks, technology or the procedure may then only become visible during the actual study.

Vague problem definitions

Observations should be described as concretely as possible and kept separate from the interpretation process itself.

Overinterpreting individual observations

Not every individual statement or action by a participant automatically represents a general usability problem.

Interpretation should consider:

• recurring patterns,
• context,
• different user groups,
• and other available data.

Planning a usability test: Key questions

Before starting a usability test, you should be able to answer these questions:

1. What should be investigated?
2. Which users are relevant?
3. Which goals and tasks should be tested?
4. How will success be measured?
5. Which data should be collected?
6. How will participants be selected?
7. How will results be documented and prioritized?

Conclusion

A usability test produces particularly meaningful results when it is not understood as a spontaneous collection of individual user opinions.

A systematic study defines:

• relevant users,
• concrete goals,
• realistic tasks,
• success criteria,
• suitable measures,
• and a traceable evaluation process.

This makes it possible to identify and prioritize concrete usability problems and translate them into evidence-based improvements.

Last modified: 17 September 2026