Two files, opposite jobs

Before a crawler like Googlebot fetches anything on your site, it looks for two files at the root. They both talk to crawlers and they both sit in plain sight, but they pull in opposite directions. This builds directly on the crawling step from how internet search works - if you have not read that, start there.

  • robots.txt is a set of rules that says where a crawler should not go. It is a boundary.
  • The sitemap is a list that says which pages are worth visiting. It is an invitation.

Neither is required, and a small, well-linked site is often crawled fine without either. But once a site grows, or has areas that should stay out of results, these two files are how you steer.

robots.txt: the rules

The file lives at exactly one place: the root of the domain, at /robots.txt. A crawler reads it before anything else. It is grouped by user-agent - the name of the crawler - with rules under each group:

# applies to every crawler
User-agent: *
Disallow: /admin/          # skip the whole admin area
Disallow: /search          # skip internal search result pages
Allow: /

# a specific bot, with its own rules
User-agent: BadBot
Disallow: /                # this one gets nothing

The pieces are few:

  • User-agent names the crawler the rules apply to. * means all of them.
  • Disallow lists a path prefix the crawler should not fetch. Disallow: / blocks the whole site; an empty Disallow: blocks nothing.
  • Allow carves an exception back out of a broader Disallow.

A crawler picks the group that matches its own name, or falls back to *. Rules are matched by longest path, so a specific Allow can override a broader Disallow.

What robots.txt cannot do

This is the part that trips people up, so it is worth being blunt. robots.txt is a request, not a lock. Reputable engines obey it; nothing forces a crawler to, and anyone can read the file to see exactly which paths you would rather hide.

More subtly: blocking crawling is not the same as blocking indexing. If you Disallow a page but other sites link to it, a search engine can still list the URL in results - it just will not have read the contents. To actually keep a page out of the index, you let the crawler fetch it and put a directive on the page itself:

<!-- in the page's <head> -->
<meta name="robots" content="noindex">

The tradeoff to hold in your head: robots.txt controls crawling; the noindex meta controls indexing; and neither is security. Anything that truly must be private belongs behind a login, not behind a rule in a public text file.

Sitemaps: the invitation

Where robots.txt fences the crawler out, a sitemap pulls it in. It is an XML file listing the pages you want found, so the engine does not have to discover every one by following links. That matters most for pages that are new, buried deep, or not linked from anywhere obvious.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-07-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/how-internet-search-works</loc>
    <lastmod>2026-07-02</lastmod>
  </url>
</urlset>

Each <url> needs a <loc> - the full address. <lastmod> tells the crawler when the page last changed, which helps it decide what to re-fetch. A sitemap is a hint about what exists and what is fresh, not a command - listing a page does not guarantee it gets indexed, and leaving one out does not hide it.

Big sites: the sitemap index

A single sitemap is capped - 50,000 URLs or 50 MB uncompressed. Large sites split their pages across many sitemap files and publish a sitemap index: a sitemap of sitemaps that points to each one.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-articles.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-authors.xml</loc></sitemap>
</sitemapindex>

Most sites never hit the limit and never need this. It is worth knowing only so the structure makes sense when you see it on a large site.

Connecting the two

The two files meet in one line. You point crawlers at your sitemap by adding a Sitemap: directive to robots.txt - which is why the crawler, reading robots.txt first, finds the sitemap without being told where it is:

Sitemap: https://example.com/sitemap.xml

Beyond that, you can submit the sitemap directly in a search engine's webmaster tools - Google Search Console or Bing Webmaster Tools - which also reports which listed pages were indexed and which were skipped, and why.

When each one matters

You do not always need to touch these files, and it helps to know when you do:

  • Reach for robots.txt when there are areas a crawler wastes time on or should stay out of - admin panels, faceted-search URLs, staging paths, endless filter combinations.
  • Reach for a sitemap when the site is large, new, or heavy on pages that are not well linked - the cases where relying on link-following alone leaves pages undiscovered.
  • Reach for noindex, not robots.txt, when a page should be crawlable but kept out of results.

A small, tidy, well-linked site can ship without either and be crawled perfectly well. Both files are steering wheels, not engines - useful the moment the default path is not the one you want. Next in the series, SEO basics puts these alongside the rest of the work that makes a page rank - and How web crawlers work takes the crawler's own side apart.

References