Available nowPro

Read the whole site, not just the homepage

Deep Crawl discovers a site’s pages, groups them by template and samples each group. You get merged tokens, a component inventory, the places where pages disagree, and a markdown cleanup plan.

Start a Deep CrawlSee pricing
A site's pages fan out from one root into three template groups, with a few pages sampled from each.example.comtemplate 1template 2template 33 sampled per template
50 to 200pages per crawl
1 creditper 10 pages
3 per templatepages sampled

How it works

From a saved project to a result, in four steps.

  1. Discover

    Sitemaps and same-origin links give a list of pages. Every redirect is checked again for safety.

  2. Group by template

    Pages are probed and hashed by structure, so a thousand product pages count once, not a thousand times.

  3. Extract in parallel

    Three pages are read at a time, with hard caps on time, bytes and requests. A job stops cleanly at the first cap and delivers what it has.

  4. Report

    You get merged tokens, clusters of buttons, cards and inputs with their one-offs, a list of inconsistencies, screenshots and a cleanup plan in markdown.

Where it helps

Audits

Start an engagement with the real inventory instead of a spreadsheet.

Redesign scoping

See how many button styles exist before promising to unify them.

Consistency checks

Find the pages that stopped following the system.

Limits. Deep Crawl reads public pages and does not log in. A job is capped at 40 minutes and 96 MB of probes and screenshots, and the report says where it stopped.

Questions, answered

What does it cost?
One credit per 10 pages, rounded up. A 50-page crawl is 5 credits and a 200-page crawl is 20.
Does it read every page?
No. It samples up to three pages per shared template, which is why 200 pages can describe a much larger site.
Can I read pages behind a login?
Not in Deep Crawl. Authenticated reads are an Agency feature, and the CLI accepts cookies on your own machine.
Where do the results go?
To a Vault snapshot of kind “deep” and a report page, with the cleanup plan as a markdown download.

More from designlang

All features
Pro

Time Machine

Read a site’s design system from the Internet Archive and see every rebrand on a timeline.

Pro

Design Debt Score

A 0 to 100 score for how much of a site’s styling resolves to a design token, with a ranked fix list and a trend line.

Pro

Monitoring

Scheduled checks that email you only when a confirmed design change happens.

Try Deep Crawl now

Free accounts get 3 live runs. Pro is $9 a month with 200 credits.

Start a Deep CrawlRead the guide