Usability Testing Guide: Choose Participants, Run Tasks, and Record Findings

A billing portal can calculate every total correctly while leaving customers unable to find the invoice they need. A functional test may verify that the download works. It does not show whether a person recognises the right document, understands its status, or knows what to do next.

Usability testing examines that interaction by observing representative people attempting realistic tasks. In a moderated session, a facilitator presents a goal, watches the participant work, and asks neutral follow-up questions. The useful evidence is the combination of what happened, the conditions in which it happened, and the participant’s explanation. General opinions about whether a screen looks good cannot replace that evidence.

Start With a Decision You Can Test

Define the decision before choosing tasks or recruiting participants. “Check whether the portal is easy to use” is too broad to guide a study. “Can recurring billing administrators distinguish a paid invoice from an outstanding payment request?” identifies a user group, a task, and a design uncertainty.

Connect each research question to something the team can change: document labels, navigation, instructions, error recovery, or the confirmation after an action. Record the prototype or build being evaluated and what it can actually do. If an unfinished prototype cannot save an address, reaching that limitation is different from failing to understand the form.

Keep the evidence boundaries explicit:

• Functional correctness: check specified behavior, such as whether the correct invoice file is returned and access restrictions work. A usability session may expose a defect, but it does not systematically verify every business rule.
• User acceptance testing: keep the business acceptance decision with its authorised owner and agreed criteria. A participant completing a task does not, by itself, approve a release.
• Accessibility conformance: combine research with disabled people with a separate evaluation against the applicable accessibility standard. Successful sessions with some users cannot establish conformance or represent every disability and technology combination.

For a first qualitative study, focus on discovering problems and understanding their causes. A quantitative benchmark needs a different measurement plan, including a defensible sample and consistent conditions. Do not turn a small problem-finding study into a claim about the percentage of all customers who will succeed. There is no participant count that guarantees complete usability coverage for every product and audience.

Choose Participants for the Work They Need to Do

Representative participants resemble the actual or likely users in ways that affect the task. For a billing portal, relevant criteria might include responsibility for invoices, familiarity with billing terminology, how often they perform the work, and whether they are new to this particular interface. A demographic profile alone does not establish that fit.

Write a recruitment brief that links each criterion to a research question. If the concern is first-use navigation, include people unfamiliar with the product. If the concern is a specialist workflow, include people who understand the work rather than expecting novices to invent domain knowledge. Describe gaps honestly when some important groups are absent.

Convenient colleagues can help pilot the script and identify broken prototype paths. Do not automatically treat their performance as representative: people who know the design may recognise labels and routes that customers have never seen. Where employees are genuinely the target users, select for the relevant work and experience rather than assuming every employee is equivalent.

Plan accessible participation, including the location or remote setup, usable materials, assistance, and the participant’s usual assistive technology where appropriate. One screen-reader user cannot speak for all screen-reader users, much less everyone with a disability. Record the combinations and experience actually represented instead of claiming broad coverage.

Protect Consent and Research Data

Before a real session, explain its purpose, what participation involves, who will observe, whether it will be recorded, what data will be collected, and how the material will be used and shared. Make voluntary participation and the ability to stop clear. Provide the information in a format the participant can use, and retain evidence of the consent obtained under your organisation’s process.

Collect only what the research needs. Use approved storage, restrict access, agree retention and deletion procedures, and check any recording, transcription, or other processing service before sending material to it. A participant code can help organise notes, but does not make a recognisable voice, face, or detailed personal story anonymous. Keep identifying extracts out of broadly shared findings unless their use is appropriately authorised and protected.

Have your privacy or legal adviser review the process for your setting. These practices support careful research; they are not a compliance certificate. The study design below uses fictional data and does not require customer records, real payments, or production access.

Write Tasks Without Giving Away the Route

A useful task presents a believable situation and a goal. It leaves the route to the participant. “Open Documents, choose September, and press Download” tests whether someone can follow instructions. “Your finance colleague needs the paid invoice for September. Find a copy you could send them” lets you observe discovery and document interpretation.

Keep a separate researcher-only completion check. Without it, observers may disagree about whether opening a payment reminder counts as finding an invoice. The check defines the required end state without requiring one particular sequence of clicks. Do not read that check aloud or expose it on the participant’s task card.

The following is a fictional, unexecuted study plan for a billing prototype. Assume its document area contains a September paid invoice and a separate reminder, invoice statuses distinguish paid from overdue, the address form saves fictional details, and support requests end at a prototype confirmation. These are design assumptions, not observed product behavior or participant results.

Task	Participant Prompt	Researcher-Only Completion Check	What to Observe
T1: Retrieve an Invoice	Your finance colleague needs the paid invoice for September. Find a copy you could send them.	The September paid invoice is opened or downloaded, not a payment reminder.	Search route, date interpretation, document choice, and any help.
T2: Check an Amount Due	You want to know whether anything still needs paying. Show what you would check.	The participant identifies the invoice marked overdue as unpaid.	Status interpretation and whether the participant confuses paid with due.
T3: Correct Billing Details	Your company has moved. Prepare the next invoice to use the new fictional address on your task card.	The new address is saved in the prototype billing record.	Where the participant looks, field meaning, and confirmation visibility.
T4: Ask About a Charge	A charge appears twice and you need someone to investigate. Show how you would ask for help.	The support request reaches the prototype confirmation without a real submission.	Help discovery, information requested, and confidence that the request is received.

For a real study, verify that each scenario makes sense to the intended participants and that the environment supports its completion check. Supply all required fictional task data without revealing the navigation path. Dummy data can reduce privacy exposure, but may also remove contextual details that affect real work; include that limitation when interpreting the findings.

Pilot the script and prototype with a colleague. Check wording, missing information, reset steps, recording permissions, and whether the observer can distinguish the intended document from a plausible wrong one. A pilot checks the study setup; it is not evidence that representative users find the product usable.

Consider task order. Earlier tasks may teach participants where documents or help controls live. Use a logical sequence where the workflow requires it, and record that dependency. If independent discovery is the question, avoid treating a later, already-learned route as first-use evidence. Prepare a consistent reset so one session does not inherit another session’s changes.

Run a Neutral Moderated Session

Use a discussion guide containing the introduction, tasks, planned prompts, completion checks, and stopping conditions. Assign a facilitator and, where possible, a separate note-taker. Observers should know the research questions and privacy boundaries, and should not interrupt with hints or defend the design.

A practical session sequence is:

1. Confirm consent, observation and recording arrangements. Explain that the service is being evaluated, not the participant’s ability. Make it clear that they can pause or stop.
2. Establish relevant background without giving a product tour. Ask how the participant normally handles the work, then introduce the first task in neutral language.
3. Invite the participant to describe what they are looking for, expecting, or thinking. Mostly watch and listen rather than filling every silence.
4. Use open prompts when clarification is needed, such as “What were you expecting there?” Avoid questions that reveal the answer, such as “Did you notice the Documents tab?”
5. Close with questions about observed moments and any final concerns. Explain what will happen with the research and protect the notes or recordings afterward.

Neutrality does not require leaving someone distressed or indefinitely stuck. Decide in advance when to offer help, move to another task, or stop. If assistance is necessary to continue learning, record exactly what the facilitator supplied and when. Reaching the end after a navigation hint is useful evidence, but is not unassisted completion.

Moderated testing is useful when you need to explore an unexpected interpretation or understand a recovery attempt. Remote sessions also require a workable connection, accessible materials, and a clear plan for what may appear on screen. Unmoderated testing removes live facilitator follow-up, so task wording and instructions must stand on their own. Choose the method for the question rather than assuming that a platform’s recording or task label explains why someone struggled.

Record Outcomes Before Interpreting Them

Capture the action and context first. “Opened the reminder, returned to the document list, then asked which file finance needed” is an observation. “The user does not understand billing” is an interpretation that may be wrong. Where you record an exact quotation, preserve its wording and enough context to explain what prompted it.

A useful observation record identifies the research round, participant code, task, prototype version, relevant device or assistive setup, observed action, outcome, and facilitator assistance. Add an evidence reference that authorised reviewers can locate. Keep unnecessary identifying information out of the working findings log.

Use consistent local outcome labels, agreed before sessions:

• Unassisted completion: the defined end state was reached without route or answer hints.
• Assisted completion: the end state was reached after recorded help.
• Not completed: the end state was not reached under the agreed stopping rule.
• Blocked by test setup: a prototype, connection, or fixture problem prevented a meaningful attempt.
• Not attempted: the task was not presented or could not be started.

These are proposed recording conventions, not universal tool statuses. Preserve setup-blocked and not-attempted records rather than counting them as ordinary participant failures. A person who confidently chooses the wrong document also differs from someone who recognises uncertainty and asks for help; the outcome label alone loses that distinction.

If timing is useful, define start and stop points and record interruptions and assistance. Think-aloud discussion and moderator questions affect the circumstances of a timed task. Do not compare those durations with another study as though the protocols were identical, or present descriptive observations as a measured improvement without a suitable comparison.

Turn Observations Into Owned Changes

Review the evidence while the sessions are fresh. Group related observations, then write findings that explain what the evidence suggests. Keep raw observations separate from the interpretation and the proposed design response. A team can agree that a problem exists while still needing to test why it occurs.

For example, suppose a future session records a participant opening the reminder and calling it the invoice. This is a hypothetical finding illustration, not a session conducted for this guide:

• Observation: the reminder was opened and mistaken for the requested invoice.
• Interpretation: document labels may not communicate their different purposes.
• Alternative explanation: task wording or unfamiliar billing terminology may have contributed.
• Proposed change: try clearer labels and document descriptions, subject to product-owner review.
• Follow-up: retest the same retrieval goal without revealing the correct route or document name in advance.

A real finding should include its affected task and user context, supporting evidence, consequence, uncertainty, proposed next action, and an owner. Keep a recommendation distinguishable from an accepted implementation commitment. If a finding reveals incorrect calculation or missing access control, route it for functional investigation as well as considering the interaction problem.

Prioritise using the consequence for the user, recurrence within the tested scope, ability to recover, and confidence in the explanation. A serious isolated barrier can justify investigation even if it did not recur. Conversely, repeated hesitation is not automatically a release blocker without understanding its effect. Avoid presenting qualitative recurrence as population prevalence, and keep unresolved explanations visible instead of turning every observation into a certain root cause.

Use the Next Round to Resolve the Remaining Risk

After a design change, check whether comparable participants can reach the intended outcome under comparable task conditions. Do not demonstrate the fix and then treat the taught route as independent discovery. Record the revised prototype and any changes to participant criteria, wording, or assistance so reviewers can understand what is being compared.

Continue functional checks, business acceptance and accessibility evaluation on their own terms. Usability evidence adds an account of how people experience the workflow; it does not replace those other decisions.

Before planning sessions, take one important user goal and review its task card beside its researcher-only completion check. Remove the route hints, confirm that the intended participants recognise the situation, and make sure the outcome can be observed. That gives the next study a decision it can genuinely inform.