The loop, and its caveats

You already know the one-sentence version from how internet search works: a crawler fetches a page, reads its links, and adds them to a list of pages to visit next. That loop is correct and it is trivial. Everything that makes crawling a real engineering problem lives in the exceptions to it - which URL to fetch next out of billions, whether it is polite to fetch it right now, whether you have already seen its content under another address, and whether the links it hands you lead somewhere real or into an infinite pit.

This article assumes you have read the robots.txt and sitemaps piece; that is the site owner's side of the conversation. This is the crawler's side.

The URL frontier

The "list of pages to visit next" has a name - the frontier - and it is not a simple queue. It is a prioritized structure that answers two questions at once: what should be crawled next, and when is it allowed to be. Priority is driven by signals like a page's estimated importance (its PageRank is a natural input), how often it changes, and how deep it sits in a site.

Seeded with a set of known URLs, the frontier grows as pages are parsed and new links discovered. The design tension is that a pure "most important first" ordering would hammer a handful of popular hosts, so the frontier is really two-tiered: one layer decides priority, another enforces per-host spacing so no single site is overwhelmed. That second layer is politeness, and it constrains everything.

Politeness

A crawler that fetched as fast as it could would be indistinguishable from a denial-of-service attack. Politeness is the set of rules that keep a crawler a guest rather than a threat:

  • Obey robots.txt. Fetched and cached before anything else on a host, it is the first gate - covered in robots.txt and sitemaps.
  • Rate-limit per host. Cap concurrent connections and space out requests to a single server - often just one request at a time, seconds apart - even while crawling thousands of other hosts in parallel.
  • Honor Crawl-delay and back off. Some sites request a delay; a 429 Too Many Requests or 503 means slow down, and a well-behaved crawler backs off exponentially.
  • Identify yourself. A truthful User-Agent with a contact URL lets an operator tell you apart from an attacker.

Politeness is the constraint that turns crawling from a bandwidth problem into a scheduling problem. You are not limited by how fast you can fetch; you are limited by how fast you are allowed to fetch each host, so throughput comes from breadth - many hosts at once - not depth.

Freshness and re-crawl

The web is not a snapshot; it changes constantly, so a crawler is never done. The freshness problem is deciding when to return to a page you have already fetched. Re-crawl a news homepage every few minutes and a static archive page once a year - guess wrong in either direction and you either waste your crawl budget or serve stale results.

The lever is adaptive scheduling: estimate each page's change rate from its history and re-crawl proportionally. Sitemaps help by advertising lastmod, but the efficient move is the HTTP conditional request - ask the server whether anything changed before downloading the body:

GET /article HTTP/1.1
If-Modified-Since: Wed, 01 Jul 2026 12:00:00 GMT
If-None-Match: "a1b2c3"

# unchanged? the server saves you the download:
HTTP/1.1 304 Not Modified

A 304 costs a round-trip and almost no bandwidth, so conditional requests let a crawler check far more pages for change than it could ever re-download - the difference between checking freshness and re-fetching the web.

Duplicates and traps

Two structural problems eat naive crawlers. The first is duplication: the same content lives at many URLs - http vs https, trailing slashes, session IDs, and the UTM parameters that make one page look like dozens. Crawlers defend with URL normalization (canonicalizing the address before queuing it), the page's declared rel="canonical", and near-duplicate content detection - fingerprinting techniques like SimHash to notice two pages are substantially the same even when the bytes differ.

The second is the crawler trap: a region that generates infinite unique URLs. A calendar with a "next month" link forever, or faceted navigation whose filter combinations explode - both mint new addresses without new content, and a crawler that follows them never comes back. Defenses are budget limits per host, depth caps, and pattern detection that recognizes a page factory. This is also why the robots.txt Disallow on faceted paths matters as much to the crawler as to the site.

Crawling at scale

Everything above multiplies by tens of billions of pages, so a production crawler is a distributed system. Work is partitioned by host - all of one site's URLs handled by one node - which keeps politeness state (rate limits, robots rules) local and avoids two machines hammering the same server. Around that sit the unglamorous necessities:

  • DNS caching. Resolving a hostname per fetch would make DNS the bottleneck; crawlers cache aggressively.
  • A "seen URLs" set at web scale, so a page is not queued twice - often a space-efficient structure like a Bloom filter to avoid storing billions of full URLs.
  • Fault tolerance. A crawl runs for days; nodes die, and the frontier must be checkpointed so a restart does not lose progress.

This is the architecture the classic Mercator crawler design laid out, and it is still the shape of the thing: a prioritized, politeness-aware, checkpointed frontier feeding a fleet of fetchers partitioned by host.

Being crawl-friendly

Knowing the internals tells you exactly how to help the crawler that visits you - which is the same work as helping it rank you:

  • Advertise change accurately. Support conditional requests (ETag, Last-Modified) and honest sitemap lastmod so the crawler re-fetches only when it should.
  • Collapse duplicates yourself. Use rel="canonical" and consistent URLs so your crawl budget is not spent re-fetching the same content.
  • Do not build traps. Keep infinite spaces - calendars, filters - behind robots.txt so the crawler spends its budget on pages that matter.

The theme of the whole crawl side: a crawler has a finite budget per site, and every wasted fetch is a real page that did not get crawled. Being crawl-friendly is just refusing to waste that budget.

References