What this is
Search is the front door of the web: most visits to most pages start with a query. This map is the route through StackNova's search series - what happens inside a search engine, how you steer its crawler around your own site, and how you earn a spot in the results, then how you measure the traffic that arrives - and finally, how AI answer engines sit on top of the whole thing. Eleven articles across the Web and AI paths, in the order they build on each other.
Like every Map, this is a living index, not new knowledge - each stage links to the article that carries the depth. The series is young; as it grows, new articles slot into these layers.
First understand the machine, then steer it, then earn its trust - then measure what it sends.- how to read this map
The engine
Everything else in this map makes sense only after you can picture the pipeline inside the engine - and the single fact that reframes it all: you never search the live web, you search a prebuilt copy of it.
Crawling discovers pages, indexing stores them as a word-to-pages lookup, and ranking orders the matches while you wait. One article builds that whole mental model, with a worked inverted-index example; its companion breaks the URL itself into its five named parts - the vocabulary every stage speaks. A 201 takes the query side deep - intent and semantic matching - and a 401 takes the ranking side to the linear algebra.
- → How internet search works LEVEL 101 - start here
- → What is a URL? LEVEL 101 - the vocabulary
- → What is a URL shortener? LEVEL 101 - URLs & redirects
- → Query understanding LEVEL 201 - the query side
- → How PageRank works LEVEL 401 - the internals
Steering the crawler
The crawl is the one stage of the pipeline a site owner can talk to directly. Two plain files at the root of a site do that talking.
robots.txt is the boundary - where a crawler should not go. The sitemap is the invitation - the pages worth finding. Its 101 covers both, plus the crawl-vs-index distinction and the noindex directive; a 301 companion takes the crawler apart - the frontier, politeness, freshness, and traps.
- → robots.txt and sitemaps LEVEL 101
- → How web crawlers work LEVEL 301 - the internals
Earning the ranking
Once the engine can reach and index a page, ranking decides whether anyone ever sees it. This is the deliberate side of feeding the ranking signals honest input.
SEO without the tricks: be findable, match what people actually search, write titles and headings that say what the page is, earn links rather than buy them, and read what Search Console tells you. Then, one level up, structured data - the schema.org markup that earns rich results - and Core Web Vitals, the performance side of ranking.
- → SEO basics LEVEL 101
- → Structured data and rich results LEVEL 201 - earning SERP features
- → Core Web Vitals LEVEL 201 - the speed signal
Measuring the traffic
Ranking earns the click; measurement tells you which links actually delivered it. The last stage of the map is the tagging convention that lets analytics credit every visit to its origin.
UTM parameters - five tags that ride on a URL's query string so a visit arrives labeled with its source, channel, and campaign. Plus the naming discipline that keeps reports honest, the canonical tag that keeps tagged URLs from polluting the index, and the campaign object those tags roll up into.
- → What is UTM? LEVEL 101
- → What are campaigns? LEVEL 101 - what utm_campaign names
The AI layer
The newest turn, and the one that looks like a replacement but is not: answer engines that write a paragraph instead of returning links. This is where the map crosses into the AI path.
ChatGPT, Google's AI Overviews, Perplexity, and Gemini add a generation step on top of crawl-index-rank: they retrieve the top results and write one answer from them. The pipeline is not replaced - it is consumed. One article covers how that works; its companion covers GEO - what you do to become a source the answer cites.
- → AI answers and search LEVEL 101 - AI path
- → What is GEO? LEVEL 201 - optimizing for it
Two reading orders
Fourteen articles, but two different reasons to be here.
You want to understand how search works
You have pages that should rank
- SEO basics - the checklist and the mindset
- What is UTM? - tag the links you control
- robots.txt and sitemaps - the technical floor
- How internet search works - the why behind both
- What is GEO? - be the source an AI cites
The map at a glance
If you read only this section, here is the route:
- The engine: crawl → index → rank, in How internet search works; the URL vocabulary in What is a URL? and What is a URL shortener?; the query side in Query understanding; the ranking math in How PageRank works.
- Steering the crawler: the boundary and the invitation in robots.txt and sitemaps; the crawler internals in How web crawlers work.
- Earning the ranking: findable, understandable, trusted, in SEO basics; richer SERP slots via structured data; the speed signal in Core Web Vitals.
- Measuring the traffic: five tags and one naming rule, in What is UTM?; the marketing object they roll up into, in What are campaigns?.
- The AI layer: answer engines on top of the pipeline, in AI answers and search; optimizing for them in What is GEO?.
Each article cross-links the others, so wherever you enter, the series carries you through.
Every article in the map
Every article in the map, by stage, with a one-line reason to read.
| Stage | Article | Why read it |
|---|---|---|
| The engine | How internet search works 101 | The mental model everything else stands on - crawling, the inverted index, ranking, and why results come back in half a second. |
| The engine | What is a URL? 101 | The anatomy - scheme, host, path, query, fragment - the vocabulary every other article in the map leans on. |
| The engine | What is a URL shortener? 101 | A lookup table and a redirect - how a long URL becomes a short one, why the 301-vs-302 choice matters, and the risks. |
| The engine | Query understanding 201 | How an engine works out what you meant - tokenizing, correcting, intent, and the shift to embeddings and semantic search. |
| The engine | How PageRank works 401 | The authority signal to the linear algebra - the link graph as a stochastic matrix, power iteration, and the damping factor that makes it converge. |
| Crawling | robots.txt and sitemaps 101 | The two files that steer a crawler - what each does, what neither can do, and when either one matters. |
| Crawling | How web crawlers work 301 | The crawler internals - the URL frontier, politeness and rate-limiting, freshness and re-crawl, and the duplicate content and traps that break naive crawlers. |
| Ranking | SEO basics 101 | The honest fundamentals of ranking well - findability, intent, titles, structure, links, and what not to do. |
| Ranking | Structured data and rich results 201 | schema.org markup and the JSON-LD that earns star ratings, breadcrumbs, and other SERP features - plus what no longer pays. |
| Ranking | Core Web Vitals 201 | LCP, INP, and CLS - the performance metrics that became a ranking signal, measured on real users, and how much they actually weigh. |
| Measurement | What is UTM? 101 | Query parameters as campaign tracking - how utm_source and friends ride along on links and land in analytics. |
| Measurement | What are campaigns? 101 | The advertising campaign hierarchy - the object utm_campaign names, and the aggregate root marketing tooling is built around. |
| AI layer | AI answers and search 101 | How LLM answer engines add a generation step on top of crawl-index-rank - retrieval, the zero-click shift, and what stays the same. |
| AI layer | What is GEO? 201 | Generative engine optimization - becoming the source an AI answer cites, and why most of it is SEO done well. |