Scanning somebody else’s website on demand is a hostile workload. Pages are slow, hosts are unreliable, redirects loop, and every request is an invitation to abuse. The pipeline is designed around those constraints rather than pretending they do not exist.
The budget
- DNS + fetch, including redirects — 15s hard cap, 5 redirects maximum.
- HTML body — capped at 2MB. Anything larger is truncated and reported.
- Parse and extract — cheerio, single pass, no selector backtracks.
- Rule engine — 15 weighted checks, pure functions, no I/O.
- Image validation — 5MB cap, HEAD first and a ranged GET fallback.
- Result cache — 10 minutes in memory, 500 entries, LRU eviction.
Fetch safely or not at all
The first rule is that user input never reaches the network unchecked. Every URL is parsed, the host is resolved, and private ranges are rejected before a socket opens.
- Only http and https schemes are accepted.
- Loopback, link-local, private and unique-local addresses are blocked.
- Cloud metadata endpoints are blocked by hostname and by resolved IP.
- Redirects are followed manually so each hop is re-validated against the same rules.
- A custom user agent identifies the bot and links back to the project, so operators can see who is fetching.
Deterministic scoring
Scoring is a weighted sum over pure functions — the same input always produces the same score, and the same code powers the website, the REST API, the MCP server and the test suite. Each check declares its own weight, so “og:image missing” can cost far more than “twitter:site missing”.
score = Σ (earned / weight) × 100
// pass → full weight
// warn → partial credit
// fail → zeroCache the boring part
Anonymous checks are served from a 10-minute in-memory cache keyed by normalised URL. It is deliberately short: long enough to absorb a user refreshing a report, short enough that a fix is visible almost immediately. Authenticated runs bypass the cache and persist to your dashboard.
Fail loudly, but usefully
When a site times out, the report still renders — with the checks it could run and an explicit issue explaining what could not be verified. A partial report is far more useful than a spinner that ends in a generic error, and it keeps the score honest by withholding weight from checks that never ran.
Never invent a score. Withhold the weight you could not verify.
All of this is observable in the product: run a URL, then open the report’s check table. Every row shows the weight it carries, the status it earned, and the rule that produced it.