1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217# Scanning Services
Equalify's accessibility scanning is performed by a collection of microservices that work together to scan HTML pages and PDF documents at scale.
## How Equalify Scans: URL-Based vs Crawling
Many accessibility tools (such as SiteImprove, Pope Tech, or Monsido) use a **crawling** approach: you provide a root domain, and the tool automatically discovers pages by following links, parsing sitemaps, and spidering the site. While convenient, crawling has notable drawbacks:
- **Unpredictable scope**: Crawlers may miss pages behind JavaScript navigation or follow links to external sites, leading to incomplete or noisy results.
- **Slow discovery**: A full crawl of a large site can take hours before scanning even begins.
- **No PDF coverage**: Most crawlers focus on HTML pages and ignore linked PDF documents.
- **Redundant scans**: Crawlers often re-scan unchanged pages, wasting resources.
Equalify takes a **URL-based** approach instead. Users provide an explicit list of URLs — either entered manually or uploaded via CSV — and Equalify scans exactly those pages. This design offers several advantages:
| | Crawling Tools | Equalify (URL-Based) |
|---|---|---|
| **Scope** | Automatic discovery; may miss or over-include pages | Explicit — you choose exactly what's scanned |
| **PDF support** | Typically limited or absent | First-class PDF scanning via veraPDF |
| **Speed** | Crawl + scan (slow for large sites) | Scan only (no discovery overhead) |
| **Repeatability** | Results vary based on crawl path | Deterministic — same URLs every time |
| **Gated content** | Requires authenticated crawling setup | Scan any publicly accessible URL directly |
This means Equalify does not automatically discover pages on your site. If you add a new page, you need to add its URL to your audit. The trade-off is full control over what gets scanned and consistent, reproducible results across scan runs.
> **Tip**: For large sites, use the CSV upload feature to bulk-add URLs. You can export a URL list from your CMS, sitemap.xml, or analytics tool and import it directly into an Equalify audit.
## Overview
The scanning architecture uses AWS SQS FIFO queues to distribute work across scanner Lambdas, ensuring reliable delivery and ordered processing per audit.
```
┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌───────────┐
│ Backend │────▶│ SQS Router │────▶│ SQS Queue │────▶│ Scanner │
│ API │ │ Lambda │ │ (FIFO) │ │ Lambda │
└─────────────┘ └──────────────┘ └─────────────┘ └───────────┘
│
▼
┌───────────┐
│ Webhook │
│ (Backend) │
└───────────┘
```
## Service Components
### SQS Router (`aws-lambda-scan-sqs-router`)
**Purpose**: Receives scan requests and routes URLs to appropriate SQS queues based on content type.
**Input Schema** (Zod validated):
```typescript
const scansSchema = z.object({
urls: z.array(
z.object({
auditId: z.string(),
scanId: z.string(),
urlId: z.string(),
url: z.string(),
type: z.string(), // "html" or "pdf"
isStaging: z.boolean().optional(),
})
),
});
```
**Behavior**:
1. Parse and validate incoming URL list
2. Separate URLs by type (HTML vs PDF)
3. Batch URLs into groups of 10 (SQS limit)
4. Send batches to appropriate FIFO queues
**Queue Configuration**:
- HTML Queue: `scanHtml.fifo`
- PDF Queue: `scanPdf.fifo`
- Message Group ID: `auditId` (ensures ordered processing per audit)
- Deduplication ID: `urlId` (prevents duplicate scans)
### HTML Scanner (`aws-lambda-scan-html`)
**Purpose**: Scans web pages for accessibility issues using a headless Chromium browser and axe-core.
**Key Features**:
- Uses AWS Lambda Powertools for metrics and logging
- Processes SQS records with partial batch failure handling
- 2-minute timeout per scan to prevent Lambda hangs
- Converts axe-core results to EqualifyV2 format
**Scan Flow**:
1. Receive SQS message with URL details
2. Launch headless Chromium browser
3. Navigate to URL and wait for page load
4. Inject and run axe-core accessibility tests
5. Convert results to EqualifyV2 format
6. POST results to webhook endpoint
**Error Handling**:
- Timeout errors: Logged and reported as failures
- Network errors: Caught and reported with error type
- Partial failures: Uses AWS Powertools batch processor
### PDF Scanner (`aws-lambda-scan-pdf`)
**Purpose**: Orchestrates PDF accessibility scanning using veraPDF.
**Architecture**:
- TypeScript Lambda receives SQS message
- Invokes Java-based veraPDF Lambda
- Converts veraPDF results to EqualifyV2 format
- Reports results via webhook
### veraPDF Interface (`aws-lambda-verapdf-interface`)
**Purpose**: Java Lambda that performs actual PDF scanning using veraPDF library.
**Features**:
- Native Java Lambda for performance
- Downloads PDF from URL
- Executes veraPDF accessibility checks
- Returns raw JSON results
## Result Format (EqualifyV2)
All scanners convert their results to a unified format:
```typescript
interface StreamResults {
status: string; // "complete" or "failed"
auditId: string;
scanId: string;
urlId: string;
url?: string;
blockers: Blocker[];
date: string;
message: string;
}
interface Blocker {
source: string; // "axe-core" | "editoria11y" | "pdf-scan"
test: string; // Rule ID (e.g., "color-contrast")
tags?: string[]; // WCAG tags (e.g., ["wcag2aa", "wcag143"])
description: string; // Human-readable rule description
summary: string; // Specific failure details
node: string | null; // HTML snippet or null for PDFs
}
```
## Axe-Core Conversion
The `AxeToEqualify2` converter transforms axe-core results:
```typescript
function convertToEqualifyV2(axeResult: AxeResults, job: any): StreamResults {
const blockers: Blocker[] = [];
// Process violations and incomplete results
[axeResult.incomplete, axeResult.violations].forEach(results => {
results?.forEach(rule => {
rule.nodes?.forEach(node => {
blockers.push({
source: "axe-core",
test: rule.id,
tags: rule.tags,
description: `${rule.description}. ${rule.help}`,
summary: node.failureSummary ?? "",
node: node.html
});
});
});
});
return { auditId, scanId, urlId, url, blockers, status: "complete", ... };
}
```
## Webhook Integration
Results are sent to the backend API webhook:
| Environment | Endpoint |
|-------------|----------|
| Production | `https://api.equalifyapp.com/public/scanWebhook` |
| Staging | `https://api-staging.equalifyapp.com/public/scanWebhook` |
**Webhook Payload**:
```json
{
"auditId": "uuid",
"scanId": "uuid",
"urlId": "uuid",
"url": "https://example.com",
"status": "complete",
"blockers": [...]
}
```
## Metrics & Monitoring
Using AWS Lambda Powertools:
- `scansStarted`: Count of scans initiated
- `ScanDuration`: Time to complete each scan (milliseconds)
- Cold start metrics captured automatically
## Adding New Scanners
To add a new scanner type:
1. Create a new service in `services/`
2. Implement SQS message consumption
3. Create a converter in `shared/convertors/`
4. Add queue routing in `aws-lambda-scan-sqs-router`
5. Register the new type in the scan schema
---
*For database schema details, see the Database Guide.*