Automated accessibility testing tools, such as axe-core by Deque, WAVE, Lighthouse are bit like a spellcheck for web accessibility. They are really useful for identifying and resolving many common accessibility issues quickly.
There are a whole range of tools that provide similar services, a way to detect some of the most common accessibility issues across a page.
The problem with automated accessibility scores
The problem is the way their reporting gives a score of out 100%. It gives the impression to the uneducated that an automated scoring once it reaches 80 or 90% is pretty good. However, these scores can be deeply misleading.
Automated tests typically detect…
Automated accessibility testing tools, such as axe-core by Deque, WAVE, Lighthouse are bit like a spellcheck for web accessibility. They are really useful for identifying and resolving many common accessibility issues quickly.
There are a whole range of tools that provide similar services, a way to detect some of the most common accessibility issues across a page.
The problem with automated accessibility scores
The problem is the way their reporting gives a score of out 100%. It gives the impression to the uneducated that an automated scoring once it reaches 80 or 90% is pretty good. However, these scores can be deeply misleading.
Automated tests typically detect only 20% to 40% of real accessibility issues. What about with AI I hear you scream? I’m sure that will increase but for now let’s pause that for this post. Like a spell-checker that flags spelling mistakes but cannot understand meaning or context, it can’t tell you if the book makes sense. These tools identify technical errors but miss many barriers that only humans can detect.
Deque’s own marketing materials claim they can detect up to 57% of issues, although at the time of writing I find it hard to review how they’re arrived at this. Which websites? How was this tested etc? Are there user testing videos?
How this scoring misleads those in power
I was sat in a presentation recently, cringing, where a Product Owner and Lead Designer proudly assert their automated score of 70% suggesting their “almost there” when they are so far away from the reality…
Suddenly there was another epic piece of work to educate certain stakeholders about this misleading nature of this score.
A site scoring 70% might appear nearly compliant but if we accept the marketing claims of 57% then a “70%” score equates to roughly 39.9% of actual accessibility compliance. This discrepancy leads people to believe that accessibility work is largely complete, when in fact the majority of blockers remain unresolved.
| Automated score (%) | Approx. % of actual issues detected (57%) |
|---|---|
| 30 | 17.1 |
| 40 | 22.8 |
| 50 | 28.5 |
| 60 | 34.2 |
| 70 | 39.9 |
| 80 | 45.6 |
| 90 | 51.3 |
| 100 | 57 |
The wider consequences
When teams focus on improving their automated score, accessibility becomes a checkbox exercise rather than a genuine effort to create accessible experiences. Developers start “fixing for the tool” instead of fixing for disabled users. The whole goal is to simply get the tooling to give a green light.
This has several negative effects:
- Teams make superficial somewhat performative, changes to satisfy tooling rather unblock disabled people.
- Businesses suddenly think they are compliant when they are not, giving them a sense of false confidence.
- Leadership tend to use these scores to justify reducing investment in accessibility.
- Most importantly, disabled users remain unable to complete tasks such as checking out, navigating menus, or using interactive features.
Why automated tools still matter
Don’t get me wrong, automated accessibility tools should not be dismissed, They are excellent for identifying obvious issues and ensuring consistency across large codebases. However, they are only a starting point, not a comprehensive solution. They are not a replacement for testing with real disabled users.
The things below can’t be skipped
- Manual testing with assistive technologies
- User testing with people with disabilities
Without these, even a “perfect” automated score is somewhat meaningless.
Time to get uncomfortable
The uncomfortable truth is that, in many organisations Accessibility isn’t treated as a commitment to unblocking people, it’s a risk management piece. For some leaders, it’s not about people, it’s about protection.
They invest in automated tools, chase high Accessibility scores because if they’re ever challenged legally, they can point to those numbers as “evidence” of compliance, hoping no one looks too closely.
Sometimes the companies selling these Accessibility testing tools also have a vested interest in keeping those scores high. Their products are compared against other platforms, and a higher “score” looks better in sales demos. They get their subscription fees whether or not disabled people can actually use the product or service.
A Call for action
I would love for these tools to update their scoring metrics.
Change their metrics, imagine if axe-core or Lighthouse had a maximum score of 57%. There was no way to get to 100%, that would shift the understanding instantly.
Misunderstanding these scores can give an organisations a dangerous illusion of compliance and may not actually improve the experience for disabled people.