What testing with disabled people finds that an audit doesn't

The two tests ask different questions, and neither answers the other's:

W3C's page on involving users in evaluation says that evaluating with disabled and older users "identifies usability issues that are not discovered by conformance evaluation alone", and also that user involvement "alone cannot determine if a website is accessible". It advises combining the two. WCAG-EM, W3C's method for conformance testing, doesn't require testing with users, but strongly recommends involving people with a wide range of abilities.

A usability test aimed at accessibility runs differently from a general one. W3C notes that it usually uses thinking aloud with more facilitator involvement, collects data on errors caused by accessibility barriers rather than on time taken or satisfaction, and puts its tasks where barriers are suspected. It also tends to turn up problems that affect every user, disabled or not.

WCAG 2.2 says in its introduction that its guidelines "are not able to address the needs of people with all types, degrees, and combinations of disability", and that even content conforming at AAA won't be accessible to all of them, "particularly in the cognitive, language, and learning areas". These are things people meet on pages that can pass at AA:

W3C's answer to the gap is supplemental guidance: "additional ways to improve accessibility, not required to meet WCAG". Its cognitive accessibility guidance comes from the Working Group Note Making Content Usable for People with Cognitive and Learning Disabilities (29 April 2021), and is grouped under eight objectives:

• Help users understand what things are and how to use them
• Help users find what they need
• Use clear and understandable content
• Help users avoid mistakes and know how to correct them
• Help users focus
• Ensure processes do not rely on memory
• Provide help and support
• Support adaptation and personalization

The Note says plainly that following it "is not required for conformance to WCAG", and that usability testing is the best way to know whether content and design work for people with cognitive and learning disabilities. It also warns against relying on automated testing alone, since automated tools often can't tell whether these users can use the content.

W3C advises bringing users in from the start of a project, not only at the end: "Conducting informal evaluations throughout development is more effective than only formal usability testing at the end of a project." Its two pages on involving users describe a part for them at each stage:

• Research: learn how people with disabilities use the web, find people with a range of characteristics, and ask them to show you products like yours that work well for them, and ones that don't. Include disabled and older people in use cases, personas, scenarios and workflows.
• Design: when weighing a design choice, such as navigation that expands and collapses, find products that already do it and explore with users what works. Include them in design walkthroughs.
• Prototypes: give people specific tasks on prototypes. The Note on cognitive accessibility adds that testing wireframes, rough drawings of the interface or template pages can find problems early.
• Before launch: a formal usability test, following an established protocol, with representative users doing specific tasks.

Before any session, W3C advises an initial review by someone who knows accessibility, to fix the known barriers first and to focus the sessions on the likely areas of concern. A session spent on a missing label that an audit would have found learns little the audit couldn't. The Note on cognitive accessibility also asks for people with cognitive and learning disabilities in the design and development team itself.

W3C stresses that disabled people are as diverse as any people: different disabilities, sometimes more than one, and wide variation inside each. Visual disability alone runs from total blindness since birth to blurred vision from a recent injury. The cognitive accessibility Note gives an example of why one group isn't enough: content with fewer words and more numbers can suit some autistic and dyslexic users, and shut out people with dyscalculia. It suggests including people with memory, attention, learning, numeric, language and communication, and intellectual disabilities.

Skill with assistive technology matters as much as the disability. Whether to recruit novice, average or advanced users depends on who will use the site: W3C's example is advanced users for an internal application for accountants, and novice users for a public site to apply for disability benefits. A user with too little experience may not be able to use their assistive technology effectively, and a very advanced one may know work-arounds the average user couldn't manage. In the early stages, while the team is still learning how people use the web, W3C suggests people with fairly high experience.

On finding people and treating them well, the sources say:

• Recruiting: disability organisations and self-help groups, social media groups, people the team already knows, and staff who help colleagues get assistive technology or other adjustments. Some projects work with peer researchers who have the disabilities in question. There are also organisations that specialise in recruiting disabled participants.
• Paying: "respect their time and provide appropriate compensation", as for any other participants.
• Informed consent: participants know what the project is for, what they will do, any risks, what personal details are collected and how they are used, and that they can stop at any time. Use a consent form people can understand, check that they have understood it, and get consent from a guardian as well where a participant has one.
• Adjustments: ask beforehand what support people need, such as a quiet room or breaks, and which methods suit them. A participant may bring a carer or friend, who must not do the tasks for them.

Then the caution both W3C pages repeat: "Avoid assuming that input from one person with a disability applies to all people with disabilities." One person doesn't necessarily know how others with the same disability use the web, and may know little about other disabilities.

A task is what someone comes to the site to do, given in their terms and not the interface's: "Buy two seats next to each other for the 14 March concert", not "Try the seat map". The Note on cognitive accessibility asks testers not just to ask questions but to have people "complete an action that demonstrates usability". A feature named in the task tells people where to go, and the test no longer shows whether they would have found it.

While people work through the tasks, the Note suggests recording tasks completed and errors made, the time each task takes (relative times say more than absolute ones), and mood and comments. By the end, the team should be able to say what stopped people finishing a task, what confused them, what led to errors, when they got frustrated, and how those problems could be avoided. Make sure participants know they can't do anything wrong: it is the site being tested, not them.

When something goes wrong, W3C lists the places the cause can lie:

• the page's markup or code;
• the browser or media player handling it badly;
• the person's assistive technology handling it badly;
• the person not knowing a feature of their browser, player or assistive technology;
• a design that is hard for everyone, disabled or not.

Only the first and last are faults in the site itself, and a test that can't tell them apart can send a team after the wrong problem. The report then states the scope of the study, the methods and who took part. W3C's example: a study with only blind participants should say that it didn't evaluate conformance and doesn't apply to all people with disabilities. Small studies are useful, but they are not statistically significant.