How Conforma tests a website
Every number this product publishes comes from the process below. It is written down so you can check it, quote it, and disagree with it.
What happens during a scan
-
Load the page in a real browser
A headless Chrome instance opens the address and waits for the page to settle, so client-rendered markup, dialogs and hydrated components are all present. Testing the HTML a server returns would miss most of a modern site.
-
Find the rest of the site
The crawler reads the sitemap where there is one and follows same-origin links where there is not. It stays on the host it landed on after redirects, so an apex-to-www redirect does not silently reduce the crawl to a single page.
-
Run axe-core at desktop and phone width
Deque's axe-core is the same engine behind most browser accessibility extensions. Each page is tested at a desktop width of 1366 pixels, then resized to a phone width of 390 pixels on the same load and tested again, because navigation and cookie banners often change on a small screen. A failure at either width counts. Rules are selected by the conformance level set for the site: A, AA (the default) or AAA.
-
Record every failing element
Each failure is stored with its rule, its WCAG success criterion, its severity, the CSS selector of the element, the markup that failed and the widths it failed at, so a developer can go straight to it.
-
Score, and say what the score covers
The page score is computed from the distinct failing rules. If the crawl stopped at the plan cap or ran out of time, the result says how many pages were discovered versus tested, and the score is described as covering the pages actually tested. Elements axe cannot decide, such as text over an image where contrast cannot be measured, do not change the score. The result counts them as needing a manual look.
The free scan on the homepage tests the one page you enter. A free account crawls up to 25 pages of a site, and the deepest paid plan crawls up to 500. Pages can be excluded by path, and the conformance level is per site.
Scanning a site behind a firewall
Some firewalls and bot protection services, Cloudflare, Vercel, AWS WAF and Akamai among them, answer a headless browser with a block page or an HTTP 403. The scan then stops and says so, because a block page scored as if it were the site would be a false result. Every request the scanner makes ends its user agent with this token:
ConformaScanner/1.0 (+https://conformahq.com/how-we-test)
To scan such a site, add a rule to its firewall that lets requests through when the user agent
contains ConformaScanner,
then run the scan again. Anyone can send a user agent, so remove the rule once you no longer scan the site.
How the score is calculated
A page starts at 100 and loses points for each distinct failing rule, weighted by severity. Ten images missing alt text cost the same as one, because they are one fix. A site's score is the mean of its page scores, rounded down, so a site scores 100 only when no tested page has a WCAG failure. The arithmetic is fixed, so the same markup always produces the same number and the number only moves when the markup does.
Points lost per failing rule
- Critical
- −15
- Serious
- −10
- Moderate
- −5
- Minor
- −2
Grades
- A
- 90-100
- B
- 75-89
- C
- 60-74
- D
- 40-59
- F
- 0-39
Only rules that map to a WCAG success criterion count towards the score. axe-core also ships best-practice rules with no criterion behind them; Conforma runs and reports those, because they are useful, but they are advice rather than a conformance verdict.
What this cannot tell you
Automated testing covers about a third of the WCAG success criteria. A scan can demonstrate that a page fails; no scan can demonstrate that a page passes. These are the things it does not judge:
- Whether alt text describes the right thing, or merely exists
- Whether a keyboard user can complete a purchase, a booking or a sign-up from end to end
- Whether the reading order and focus order make sense to somebody who cannot see the layout
- Whether an error message explains what to do about the error
- Whether a video has captions that match what is said, or an audio description at all
- Whether a component that is technically labelled is actually understandable
Full conformance with WCAG 2.2 AA, the web part of EN 301 549 V4.1.1, needs a manual review by somebody who tests with a keyboard and a screen reader. Conforma's job is to clear everything a machine can decide, and to leave a dated record that you did.
How the claims on this site are sourced
Legal and enforcement claims name their source on the page that makes them. Where sources conflict or a case is under appeal, both sides are stated. Where nothing has happened, that is stated too: no EU regulator has issued a fine under the EAA to date, and this site says so rather than implying otherwise.
Country pages cite the transposing statute by number and name the enforcing body. The Carrefour write-up lists the decision, the reporting and the case that went the other way. Nothing here is legal advice.
Reports published about other people's sites
A scan of a public website produces a public report at /accessibility-report/{host}. Those pages always show the newest scan of the domain's home page, they state what the scan covered, and every one of them carries a removal link that works with no account and no wait. If a report about your site is wrong or unwanted, take it down from the page itself.
Questions about the method
How much of WCAG can an automated test actually check?
About a third of the success criteria, and only the machine-checkable part of those. A scanner can prove a page fails; it can never prove a page passes. Contrast ratios, missing text alternatives, unlabelled controls, language attributes and document structure are decidable by a machine. Whether the alt text describes the right thing, whether a keyboard user can complete a purchase, and whether an error message makes sense are not.
Why does the score not drop for every single failing element?
Because ten images missing alt text are one fix, not ten. The score charges for each distinct failing rule, weighted by severity, so it tracks the work in front of you rather than the size of your page. A page with one broken pattern repeated a thousand times and a page with it once score the same, which is the honest reading of the situation.
Do best-practice findings count against the score?
No. axe-core ships rules that are good advice but carry no WCAG success criterion behind them. Conforma runs them and shows them, because they are useful, but it does not count them as conformance failures. A site whose only finding is a missing main landmark is not failing WCAG, and a score that said otherwise would be inventing a compliance problem.
What happens when a crawl cannot finish?
The scan says so. A crawl that hit the page cap or ran out of time reports how many pages it discovered and how many it tested, and the score is described as covering those pages rather than the whole site. Reporting a partial crawl as if it were a complete one is the single easiest way for a tool like this to lie.
Can I have a published report about my site removed?
Yes, from a link on the report itself, with no account and no wait. Reports are generated only for domains somebody actually scanned, they always show the newest scan, and they carry the removal link at the foot of the page.
Run the method against your own site
Free, no sign-up, and the result is yours to keep or share.
Related: the fix guide for every rule we report, what the EU Accessibility Act requires, and Conforma's own accessibility statement.