# SEO and GEO: Search in the Age of Answers > A complete course on how search works now, built to leave you genuinely expert rather than merely briefed. - **Canonical URL:** https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/experience/seo-geo - **Category:** Education & Learning - **Curated by:** SKEMR AI - **Contents:** 5 journeys, 17 chapters, 85 sources - **Last updated:** 2026-10-14 ## What this is A curated transmedia experience: existing web sources selected and ordered by an editor. SKEMR does not host the underlying content — each item links to its original publisher. Journeys are alternative routes through the same material. ## Journeys ### The Complete Course Best for becoming genuinely expert rather than briefed. All seventeen chapters in order: the fundamentals that still hold, the answer layer and what it did to clicks, the actual mechanics of training versus retrieval, the llms.txt evidence, what Google says to do, the tactics that survive scrutiny, measurement, the tooling market, the crawl economy, and the research frontier. Around twelve hours if you read the primary documents, which is the point. Nothing here is a summary of a summary. - Steps: 85 - Machine-readable: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j1 - Briefing for a chat: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j1/briefing.md ### The Marketer's Route Best for someone who owns the channel and needs to act on Monday. Skips the implementation detail and the research frontier, and concentrates on what changed, what the evidence actually supports, what Google says to do, which tactics survive scrutiny, and how to measure any of it. Ends at the vendor market with the questions worth asking before signing anything. Around three hours. - Steps: 22 - Machine-readable: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j2 - Briefing for a chat: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j2/briefing.md ### The Technical Deep Dive Best for engineers and technical SEOs who have to implement rather than advise. Heavy on specifications, crawler documentation, structured data syntax and retrieval mechanics; light on strategy. Includes the server-rendering failure that makes a site invisible to models while remaining perfectly visible in Search, which is the single most common expensive mistake in this area. Around six hours with the specs. - Steps: 40 - Machine-readable: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j3 - Briefing for a chat: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j3/briefing.md ### The Business Leader's Route Best for someone deciding budget, headcount or vendors rather than implementing. Shortest of the deep routes. Covers what actually changed and how confident we can be about it, what Google's own position is, what the tooling market genuinely sells, the commercial question of paid crawl access, and the governance risks. Written to leave you able to interrogate a proposal rather than evaluate a tactic. Around two and a half hours. - Steps: 22 - Machine-readable: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j4 - Briefing for a chat: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j4/briefing.md ### The Sceptic's Route Best for working out what is actually known, and the route to take before spending money. It sets every strong claim against its best counter-evidence: the GEO paper against what its headline figure actually measures, Pew against Google's rebuttal, the llms.txt proposal against two studies that found almost nobody reads the file, vendor capability claims against Google's statement that no third-party tool sees its internal systems. The conclusion it builds to is not that the field is fake. It is that the foundations are well evidenced and the tactics mostly are not. - Steps: 31 - Machine-readable: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j5 - Briefing for a chat: https://prodcms-skemr-monolith-d3fzb9bva4fsfbbj.northeurope-01.azurewebsites.net/api/ai-context/seo-geo/journeys/j5/briefing.md ## Chapters - **What Search Actually Does** — Three distinct operations that get collapsed into one word. Crawling is discovery: software follows links and fetches pages. Indexing is storage and understanding: the fetched page is parsed, its content and meaning recorded. Ranking is selection: for a given query, deciding which of billions of indexed documents to show and in what order. Almost every practical problem in this field is really a failure at one specific stage, and diagnosing which one is most of the job. - **The Fundamentals That Still Hold** — Google's own position is that AI features are built on its core ranking and quality systems, so the groundwork is unchanged: be crawlable, be indexable, be fast enough, have one canonical URL per thing, and tell the crawler where everything is. None of this is glamorous and all of it is prerequisite. A page that is not indexed and eligible for a snippet cannot appear in an AI Overview, which makes technical hygiene the floor rather than the ceiling. - **Quality, and How It Gets Judged** — Google publishes two things worth reading closely: its guidance on creating helpful content, and the Search Quality Rater Guidelines, a document of nearly two hundred pages used to train the human raters who evaluate search results. The raters do not change rankings directly; their judgements are used to evaluate changes to the systems that do. E-E-A-T — experience, expertise, authoritativeness, trustworthiness — comes from that document, and is a description of what raters are asked to look for rather than a score any algorithm computes. - **Structured Data** — Machine-readable markup describing what a page is about, usually schema.org vocabulary expressed as JSON-LD in a script tag. It does not make a page rank better. What it does is make a page eligible for specific presentation features — review stars, recipes, events, products — and give machines an unambiguous statement of entities and relationships rather than a guess parsed from prose. Google is explicit that it is not required for AI features, and equally explicit that it still recommends it as part of general SEO. - **The Answer Layer Arrives** — Between 2023 and 2026 a layer settled on top of search that answers rather than lists. Google shipped generative summaries and then AI Mode, a full conversational surface. ChatGPT gained live web search. Perplexity built an entire product around cited answers. The common structural change is that the user's question is now answered on the page they are already on, with links as supporting evidence rather than as the destination. Everything downstream in this course follows from that one shift. - **What It Did To Clicks** — The most-cited evidence is Pew's July 2025 study of 900 US adults and 68,879 searches: where an AI summary appeared, users clicked a traditional result in 8% of visits against 15% where none appeared, just 1% clicked a source cited inside the summary, and 26% ended the session without clicking anything against 16% otherwise. Google disputes it, calling the methodology flawed and the query set unrepresentative. Both positions belong in your head at once — the study is the best independent measurement available and it is one panel, one month, one country. - **GEO, AEO, and What the Words Mean** — Generative Engine Optimization has a real academic origin: a paper by Aggarwal and colleagues presented at KDD 2024, which proposed the term, built a benchmark, and reported visibility gains of up to 40% from specific content modifications. Read the paper before quoting the number, because it is measured against their own visibility metric on their own benchmark, not against revenue or traffic. Answer Engine Optimization is used roughly interchangeably in industry. Neither term describes a system any search provider has confirmed exists as a distinct ranking surface. - **Training Data Versus Live Retrieval** — The single most important distinction in this field, and the one most often muddled. A model's training data is a frozen snapshot: content absorbed during training, unattributed, unchangeable after the fact, and impossible to update by editing your site. Live retrieval is different — the system runs a search at question time, fetches current pages, and generates an answer grounded in them, usually with links. Publishing changes can affect retrieval today. They cannot affect a model already trained. Almost every piece of GEO advice is implicitly about one or the other, and rarely says which. - **Who Is Actually Crawling You** — Several distinct agents with different purposes, and conflating them produces bad decisions. Googlebot crawls for Search. Google-Extended is a separate control governing use in Gemini and other AI products without affecting Search ranking. OpenAI runs distinct agents for training, for live user-initiated browsing, and for search indexing — blocking one does not block the others. Anthropic and Perplexity publish their own. Your server logs are the authoritative record of which of these actually visit you, and they frequently contradict what you assume. - **How Citation Actually Happens** — In a retrieval system the chain runs: the question is converted to a vector, candidate passages are retrieved by similarity from an index, a subset is selected to fit a context window, and the model generates an answer conditioned on them. Being cited therefore requires being retrievable, being selected, and being useful enough to survive into the generated text. This is why a page that answers one specific question cleanly outperforms a longer page that covers it among nine other topics — the unit of retrieval is the passage, not the site. - **The llms.txt Question** — A proposed convention: a Markdown file at the site root summarising the site for language models, by analogy with robots.txt. It is a proposal with no formal standard behind it, and the evidence on it is unkind. Google states that Search ignores the file entirely and that creating one will neither harm nor help. Ahrefs examined 137,000 domains and found that of those publishing a valid file, 97% received zero requests for it in May 2026. SE Ranking looked at roughly 300,000 domains and found no relationship between having one and being cited. Note the two studies disagree on adoption — 28% against about 10% — which is itself a lesson about sample selection. - **What Google Actually Tells You To Do** — Google published a dedicated optimisation guide for generative AI features in May 2026, and it is shorter and more deflationary than the industry built around it. Its position: SEO best practices continue to be relevant because AI features run on the same core systems; create content yourself based on what you actually know; you do not need new machine-readable files, AI text files, markup or Markdown; there is no requirement to chunk content; structured data is not required though still recommended generally; and no third-party tool has access to Google's internal ranking or AI systems. Read the primary document rather than anyone's summary of it, including this one. - **Tactics That Survive Scrutiny** — What is left once the unevidenced advice is stripped out. Answer one question per page, in the first paragraph, in the words people actually use. Be the primary source for something rather than a summary of other people's work. Keep facts current and dated, because retrieval favours pages that look maintained. Make the page render its content server-side, since a crawler that does not execute JavaScript sees nothing otherwise. Earn genuine mentions on sites that get retrieved. None of this is new, and that is the point — the tactics that work for answer engines are substantially the tactics that worked before, applied with more precision. - **Measurement** — The part that improved most in 2026. Google added generative AI performance reports to Search Console, showing how often your URLs appeared in AI features in Search and Discover — rolled out to a subset of sites first. Beyond that, your server logs tell you which AI agents fetch what and how often, which is the only first-party evidence available and costs nothing. Everything else on the market is sampled estimation: useful for direction, not for a number you would put in a board paper. - **The Tooling Market** — A real category with real money in it. Profound is the best-known, offering prompt-volume data, citation and sentiment monitoring, agent analytics and content tooling; Scrunch AI, Otterly, AthenaHQ and Peec occupy adjacent positions with differing emphases and price points. What these products genuinely do is sample: run prompts against models repeatedly and record what comes back. That is a legitimate and useful measurement technique. It is not access to anybody's ranking system, and Google says so directly. Judge any vendor claim by whether it is a sampled observation or an assertion about internal mechanics. - **The Crawl Economy** — The commercial question underneath all of it: AI systems consume content and increasingly do not send traffic back, which breaks the implicit bargain the open web ran on. Cloudflare moved in 2025 to block AI crawlers by default for new domains and to build a marketplace where publishers can charge for access. Whether that becomes normal or stays niche is unresolved, and it matters more to most businesses than any tactic in this course — if access becomes negotiated rather than open, visibility becomes a procurement question rather than a content one. - **Where This Is Going** — The research has moved past describing GEO to interrogating it. Recent work examines adversarial or malicious optimisation — deliberately crafted content designed to capture generative answers — and defences against it, along with attempts to build manipulation-resistant scoring. A 2026 position paper argues the real risks are concentration, disclosure and academic blind spots rather than individual tactics. The reasonable expectation is that generative surfaces will develop the same adversarial dynamic search did, that the countermeasures will be undocumented, and that anybody promising a durable edge is describing a window rather than a moat. ## Sources - [Google Search Essentials](https://developers.google.com/search/docs/essentials): The shortest statement of what Google requires, and the right first read. Three parts: technical requirements for a page to be eligible at all, spam policies describing what gets you removed, and key best practices. Everything else in Google's documentation elaborates on this page. If a tactic contr... - [SEO Starter Guide](https://developers.google.com/search/docs/fundamentals/seo-starter-guide): Google's own introduction, which is notably more restrained than most paid courses on the same material. Covers getting indexed, helping Google understand content, managing crawling, and organising a site. Read after Search Essentials. Worth noting what it does not contain: no keyword density, no wo... - [Sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview): A machine-readable list of your URLs with optional metadata about when each changed. It does not guarantee indexing and does not improve ranking; it improves discovery, which matters most for large sites, new sites, and pages that are poorly linked internally. Practical note: a sitemap listing URLs... - [Canonicalisation](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls): How to tell a search engine which URL is the real one when the same content is reachable several ways — tracking parameters, http and https, trailing slashes, print views. Unresolved duplication splits signals across versions and leaves the engine to pick for you. One of the most common and least... - [Page experience](https://developers.google.com/search/docs/appearance/page-experience): Google's framing of loading performance, interactivity, visual stability, mobile usability and HTTPS. Read it with the proportion Google itself gives it: these are tie-breakers between comparable pages rather than primary ranking forces. Fixing a slow site is worth doing; expecting it to outrank bet... - [Creating helpful, reliable, people-first content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content): Google's guidance on quality, structured as questions to ask about your own work: does it provide original information, does it demonstrate first-hand expertise, would someone bookmark it. The most useful section is the self-assessment list, which is uncomfortable to run honestly against most commer... - [Search Quality Rater Guidelines](https://guidelines.raterhub.com/searchqualityevaluatorguidelines.pdf): The full document Google gives the human raters who evaluate search results, published openly and running to well over a hundred pages. This is where E-E-A-T is actually defined, along with Your Money or Your Life topics and the page quality scale. Read after the helpful-content guidance. Critical c... - [Introduction to structured data](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data): Google's explanation of what markup does and does not do. The essential framing: structured data makes a page eligible for specific result features, it does not improve ranking, and marking up content that is not visible on the page is a policy violation. Start here before touching schema.org itself... - [Structured data search gallery](https://developers.google.com/search/docs/appearance/structured-data/search-gallery): The definitive list of which markup types Google actually supports and what each produces in results — articles, breadcrumbs, events, FAQs, products, reviews, videos and the rest. Read after the introduction. This is the page that answers the real question: of the hundreds of schema.org types, whi... - [Rich Results Test](https://search.google.com/test/rich-results): Google's own validator: paste a URL or code and see exactly which rich result types the page qualifies for and which errors block it. Practical rather than theoretical, and the fastest way to settle an argument about whether markup is working. Use it before and after any structured data change. - [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features): Google's reference page on how AI Overviews and AI Mode interact with sites, including the eligibility requirement that a page be indexed and able to show a snippet. Read alongside the Pew findings as the other side of the argument: Google's stated position is that AI features surface a wider range... - [Top ways to ensure your content performs well in AI experiences](https://developers.google.com/search/blog/2025/05/succeeding-in-ai-search): Google's May 2025 blog post, and the earliest clear statement of its position: focus on unique, satisfying content for people rather than on anything AI-specific. Useful chronologically — it shows the advice was consistent a full year before the formal optimisation guide, which undercuts the idea... - [A new resource for optimizing for generative AI](https://developers.google.com/search/blog/2026/05/a-new-resource-for-optimizing): The May 2026 announcement of the optimisation guide. Short, and worth reading for the framing rather than the content: Google chose to publish a dedicated AI optimisation resource whose central message is that you do not need AI-specific optimisation. That tension is the most informative thing about... - [Google crawlers overview](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers): Every user agent Google operates and what each is for, including the distinction that matters most here: Googlebot crawls for Search, while Google-Extended is a separate control for whether your content is used in Gemini and other AI products. Blocking Google-Extended does not remove you from Search... - [Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro): How to control crawling, and the limits of doing so. Two points people get wrong constantly: robots.txt controls crawling rather than indexing, so a disallowed URL can still appear in results if linked from elsewhere, and the file is a request that well-behaved crawlers honour rather than an access... - [Google's guide to optimizing for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide): The keystone document of this entire course, published May 2026. Its actual statements: best practices for SEO continue to be relevant because AI features are built on core Search systems; you do not need to create new machine readable files, AI text files, markup, or Markdown; there is no requireme... - [Snippets and how to control them](https://developers.google.com/search/docs/appearance/snippet): The mechanics behind eligibility for AI features, since a page must be able to show a snippet to be used. Covers nosnippet, data-nosnippet and max-snippet, which let you exclude specific passages rather than whole pages. The underrated control here is data-nosnippet, which can keep boilerplate or pa... - [The Search generative AI control](https://support.google.com/webmasters/answer/16908024): Search Console's setting for excluding your site's links and content from AI Overviews, AI Mode and Discover's generative features. Note what it costs: exclusion applies to the generative surfaces specifically, takes a few days to take effect, and is a genuine trade-off rather than a free safety mea... - [Google's approach to website controls for AI features](https://blog.google/products-and-platforms/products/search/search-ai-features-controls/): The policy reasoning behind those controls, from Google's own blog. Useful for anyone who has to explain to a board why the company is or is not visible in AI answers: this is the official account of what site owners can and cannot decide, in Google's words rather than a commentator's. - [Ecommerce SEO guidance](https://developers.google.com/search/docs/specialty/ecommerce): Google's specialty documentation for product sites, included because ecommerce is where structured data genuinely earns its keep — product, price, availability and review markup feed shopping surfaces directly. If you sell things, this page is worth more than any amount of general advice about con... - [Search generative AI performance reports](https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports): Google's June 2026 announcement of reporting that shows how often your URLs appeared in generative AI features across Search and Discover. The most significant measurement development in the field, because until this existed there was no first-party data at all. Note the caveat in the announcement:... - [Getting started with Search Console](https://developers.google.com/search/docs/monitor-debug/search-console-start): The tool itself, free, and the only source of first-party data about how Google sees your site: indexing status, queries, impressions, clicks, structured data errors. Read after the generative reports announcement. Anyone discussing their search performance without reference to this is speculating. - [Search Off the Record (podcast)](https://developers.google.com/search/podcasts/search-off-the-record): Google's own Search Relations podcast, where Mueller, Splitt and Illyes discuss these questions at length and with more candour than the documentation allows. The best listening in the field, and the place where the llms.txt scepticism was argued through rather than merely stated. Audio, and genuine... - [Google Search Central (video)](https://www.youtube.com/@GoogleSearchCentral): The official video channel: office-hours recordings, explainers and conference talks from the Search Relations team. Video, and the most efficient way to absorb the reasoning behind a documented position when the document itself is terse. - [Search engine optimization](https://en.wikipedia.org/wiki/Search_engine_optimization): The field's full history and scope, from the mid-1990s onward, including the long arms race between optimisation and anti-spam systems. Read it for the pattern rather than the tactics: every technique that worked mechanically was eventually detected and neutralised, and the only durable strategy has... - [Web crawler](https://en.wikipedia.org/wiki/Web_crawler): How discovery actually works: a frontier of URLs, politeness rules, revisit policies, and the economics of deciding what to fetch and how often. Read after the SEO overview. The concept to take away is crawl budget — crawling is not free, so a crawler decides how much of your site is worth its tim... - [Google Search](https://en.wikipedia.org/wiki/Google_Search): The product itself and how it has changed: the index, the ranking signals, the long sequence of named updates, and the progressive addition of answer surfaces above the links. Useful mainly for scale and context — the point to absorb is that results pages have been getting less link-shaped for fif... - [PageRank](https://en.wikipedia.org/wiki/PageRank): The original link-based ranking algorithm, and worth understanding as an idea rather than as current mechanics: importance inferred from the structure of citations rather than from the content itself. Modern ranking uses hundreds of signals and PageRank is one small input, but the underlying intuiti... - [Search engine results page](https://en.wikipedia.org/wiki/Search_engine_results_page): The anatomy of the page itself: organic results, ads, knowledge panels, featured snippets, local packs, and now generative answers. Optional, and useful for a concrete sense of how little of a modern results page is actually the ten blue links. This is the visual context for every click-through stat... - [Backlinks](https://en.wikipedia.org/wiki/Backlink): What links are as a ranking signal, how link schemes work, and why they remain both effective and dangerous. The honest position: links still matter, buying them violates Google's spam policies, and most link-building services sell exactly what the policies prohibit. Read it before approving any age... - [JSON-LD](https://en.wikipedia.org/wiki/JSON-LD): The syntax structured data is actually written in: JSON with a context that maps keys to a shared vocabulary, embedded in a script tag so it does not affect rendering. The practical advantage over older formats is that it sits in one block rather than being woven through the HTML, which means it can... - [ChatGPT](https://en.wikipedia.org/wiki/ChatGPT): The product that made conversational answers mainstream, and now a search surface in its own right with live browsing. Relevant here for one structural reason: a question answered in a chat window has no results page, no position one, and no impression count — which is why the entire measurement a... - [Gemini](): Google's model family and assistant, and the system Google-Extended governs your content's use in. Worth keeping distinct from AI Overviews in Search: they are different surfaces with different controls, and conflating them leads people to block the wrong thing. Optional, and the clearest illustrati... - [Perplexity](https://en.wikipedia.org/wiki/Perplexity_AI): The product built from the start around cited answers rather than retrofitting them, which makes it the clearest working example of what a citation-first engine looks like. Study its output format specifically: numbered inline citations, a visible source list, and follow-up questions. That shape is... - [Featured snippets](https://en.wikipedia.org/wiki/Featured_snippet): The precursor everybody forgot. Google has been extracting direct answers onto results pages since 2014, and the entire zero-click debate was rehearsed then with the same arguments and much the same data. Useful historical grounding: the answer layer is not new, it has become general rather than app... - [Answer engine optimization](https://en.wikipedia.org/wiki/Answer_engine_optimization): The companion term, used broadly interchangeably with GEO in industry while sometimes being distinguished as focusing on direct-answer surfaces specifically. Read it mainly to inoculate yourself: the proliferation of near-synonymous acronyms in this field is a marketing phenomenon rather than a conc... - [Retrieval-augmented generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation): The architecture behind every cited AI answer, and the most important technical concept in this course. The model does not recall your page; a retrieval step fetches candidate documents at question time and the model generates conditioned on them. The whole of practical GEO reduces to being retrieva... - [Large language models](https://en.wikipedia.org/wiki/Large_language_model): What the model is and how training works, at the level needed to reason about this field properly. The decisive fact for marketers: training is a batch process over a fixed corpus, so a model cannot learn about your rebrand by you publishing a page. If your subject matters to a model's baseline know... - [Hallucination in AI](): Why generated answers about your business can be confidently wrong, and why that is a content problem as much as a model problem. Where the record is thin, contradictory or out of date, the model fills the gap plausibly. The practical implication is defensive: a clear, current, unambiguous statement... - [robots.txt](https://en.wikipedia.org/wiki/Robots.txt): The convention itself, its history since 1994, and its genuine limits. Read alongside Google's documentation for the part Google does not emphasise: compliance is voluntary, there is no enforcement, and the growth of AI crawling has put real strain on a protocol built on good manners between a handf... - [Semantic search](https://en.wikipedia.org/wiki/Semantic_search): Matching on meaning rather than on literal terms, which is why keyword-stuffing stopped working and why writing naturally started to. Read after retrieval-augmented generation. The practical consequence is that you no longer need to guess exact phrasings — you need to cover a concept thoroughly en... - [Embeddings](https://en.wikipedia.org/wiki/Word_embedding): How meaning becomes arithmetic: text mapped into a high-dimensional vector space where proximity approximates similarity. This is the mechanism underneath semantic retrieval, and understanding it explains a lot of otherwise mysterious behaviour — including why a passage on a page about something e... - [Vector databases](https://en.wikipedia.org/wiki/Vector_database): The infrastructure that makes similarity search fast at scale, using approximate nearest-neighbour indexes. Optional for marketers and essential for anyone implementing retrieval. Included because it makes concrete what retrieval is doing: your content is not being read, it is being measured for pro... - [Knowledge graphs](https://en.wikipedia.org/wiki/Knowledge_graph): Entities and relationships held as structured facts rather than as prose, which is how a system knows your company and a person with the same name are different things. Relevant to citation because entity disambiguation determines whether answers about you are actually about you, and consistent nami... - [Schema.org](https://en.wikipedia.org/wiki/Schema.org): The vocabulary's own history and governance, founded in 2011 by Google, Microsoft, Yahoo and Yandex. Included in the closing chapter deliberately: it is the last time the major search players agreed a shared standard for machine-readable content, and the obvious question for the next decade is wheth... - [Web Vitals](https://web.dev/articles/vitals): Google's performance metrics defined precisely — what each measures, the thresholds, and how to improve them. Read for the measurement discipline rather than the targets: this is a rare case where the industry has agreed definitions, so performance arguments can be settled with numbers instead of... - [Schema.org vocabulary](https://schema.org/): The vocabulary itself: hundreds of types and thousands of properties, maintained collaboratively and used far beyond search. Browse it rather than reading it. The practical approach is to find the single type that matches your page, implement that properly, and ignore the rest — over-marking is a... - [JSON-LD 1.1 specification](https://www.w3.org/TR/json-ld11/): The W3C standard itself, for anyone implementing rather than specifying. Dense and authoritative. Worth having to hand for the parts that trip people up in practice — context resolution, @id and @type semantics, and how nesting actually serialises. - [The Robots Exclusion Protocol](https://www.robotstxt.org/): The original specification site for robots.txt, documenting a convention that predates almost everything else in this course and still governs who may read your content. Read it for the realisation that the entire economy of web crawling rests on a voluntary text file agreed informally in 1994. - [The llms.txt proposal](https://llmstxt.org/): The specification itself, and the thing to read before forming a view. It proposes a Markdown file at the site root giving a model a curated map of the site, by analogy with robots.txt and sitemap.xml. Note its own framing: it is a proposal, open for feedback, with no standards body behind it and no... - [Schema Markup Validator](https://validator.schema.org/): The vocabulary-level validator, as distinct from Google's rich results test: this one checks whether your markup is valid schema.org regardless of whether Google happens to use that type. The right tool when you are marking up for consumers other than Google Search, which is increasingly the point. - [IndexNow](https://www.indexnow.org/): A protocol letting sites notify participating search engines immediately when content changes, supported by Bing and Yandex among others. Practical and underused: for sites where freshness matters, this converts discovery from polling to pushing. Google has not adopted it, which is the honest caveat... - [Bing Webmaster Guidelines](https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a): The other major index, and worth reading precisely because it is not Google. Bing's guidelines are more explicit in places about what it values, and Bing's index underpins other products — which makes it disproportionately important relative to its direct search share. - [Pew: users click less when an AI summary appears](https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/): The headline independent study, July 2025. Method: browsing data from 900 US adults on Pew's KnowledgePanel, 68,879 searches in March 2025, of which about 18% produced an AI Overview. Findings: 8% click rate on traditional results with a summary present against 15% without; 1% of visits included a c... - [Reuters Institute Digital News Report](https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2026): The annual survey of news consumption across dozens of markets, run by Oxford's Reuters Institute and the most methodologically serious recurring dataset on how people actually find information. Read after Pew for scale and trend rather than a single snapshot. For publishers specifically, this is th... - [GEO: Generative Engine Optimization (KDD 2024)](https://arxiv.org/abs/2311.09735): The paper that named the field, by Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan and Deshpande, presented at KDD 2024. It proposes GEO as a black-box optimisation framework, introduces GEO-bench across multiple domains, and reports visibility improvements of up to 40%. Two things to hold: that... - [Ahrefs: 97% of llms.txt files never get read](https://ahrefs.com/blog/llmstxt-study/): An analysis of 137,000 domains in Ahrefs' own web analytics, May 2026. Of those publishing a valid llms.txt, 97% received zero requests for it that month. Caveats worth stating: it is a vendor study, the sample is sites using one analytics product, and it measures whether files are fetched rather th... - [SE Ranking: why llms.txt does not work](https://seranking.com/blog/llms-txt/): A second independent analysis, roughly 300,000 domains, asking the different and more useful question: does having the file correlate with being cited by LLMs. It found no relationship. Read alongside Ahrefs and note that the two disagree on adoption — about 10% here against 28% there — which te... - [GEO creates underexamined risks (position paper)](https://arxiv.org/abs/2606.12439): A 2026 position paper arguing that the governance questions around generative engine optimisation — concentration of influence, disclosure of paid placement, and gaps in academic attention — matter more than the tactical questions the industry focuses on. The most useful single document for anyo... - [Defending generative search against malicious GEO](https://arxiv.org/abs/2609.02964): Research on adversarial optimisation: content deliberately engineered to capture generative answers, and the defences against it. Read after the position paper. This is the clearest signal of where the field goes next — the same detection-and-evasion dynamic that defined twenty years of search spa... - [Manipulation-resistant content scoring](https://arxiv.org/abs/2609.07559): Work on scoring content for generative engines in a way that resists gaming, without needing access to the engine itself. Technical and early. Included because it points at the likely end state: if durable scoring becomes possible, the advantage shifts back to genuinely better content, exactly as it... - [Common Crawl](https://commoncrawl.org/): The open repository of web crawl data that has underpinned a great deal of model training, and the most concrete answer to where does training data come from. Practically useful: if you want to understand whether your content plausibly entered a training corpus, this is the dataset to reason about r... - [OpenAI's crawlers](https://developers.openai.com/api/docs/bots): OpenAI's own documentation of its user agents and what each does. The essential detail is that they are distinct: one for training-data collection, one for live fetches triggered by a user's question, and one for search indexing. Blocking the training crawler while permitting the user-triggered one... - [Anthropic's web crawler](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler): Anthropic's documentation on its crawling and how site owners can control it. Read alongside OpenAI's for the comparison: the providers have converged on broadly similar conventions — named user agents, robots.txt compliance, published ranges — without any formal standard requiring it. That volu... - [Perplexity's crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers): Perplexity's documentation of its user agents, including the distinction between its index crawler and fetches made in response to a live user question. Relevant because Perplexity is the purest citation-driven product, so understanding what it fetches and when is the cleanest window into retrieval... - [Cloudflare: no AI crawl without compensation](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/): Cloudflare's July 2025 announcement that it would block AI crawlers by default for new domains and build a marketplace for paid access. Whatever you think of it, this is the single most consequential infrastructure decision in the period this course covers — a company sitting in front of a large s... - [Cloudflare bot documentation](https://developers.cloudflare.com/bots/concepts/bot/): The operational detail behind the policy: how bots are identified, categorised and verified, and what controls are available. Read after the announcement. Practically important because for many sites the effective AI-access policy is set at the CDN rather than in robots.txt, and the two can silently... - [web.dev](https://web.dev/): Google's resource for web platform engineering, included here as the counterweight to the access debate: whatever the commercial arrangements become, the technical obligation stays the same — content must be reachable, fast and rendered where a machine can see it. Optional, and the best general re... - [Profound](https://www.tryprofound.com/): The best-known platform in the category. Its published capabilities include prompt-volume data, monitoring of how models describe a brand, citation and sentiment tracking, agent analytics on which bots fetch which pages, benchmarking against competitors, and content tooling. It also runs a research... - [Scrunch AI](https://scrunch.com/): An enterprise-positioned competitor focused on how brands appear across AI platforms and on optimising the experience AI agents have of a site. Read after Profound for the comparison. The category is consolidating, so check the current ownership and status of any vendor before committing — this ma... - [Otterly.AI](https://otterly.ai/): A smaller, more accessible monitoring tool tracking brand mentions and links across AI search platforms. Included to show the range of the market: the same core technique — repeated prompting and recording what comes back — is sold at wildly different price points, and the differences are largel... - [AthenaHQ](https://athenahq.ai/): Another entrant in the AI-visibility monitoring space, positioned around tracking and improving how brands surface across multiple models. Optional, and worth a look mainly for how similar the category's propositions have become — a useful signal in itself about how undifferentiated the underlying... - [Peec AI](https://peec.ai/): A European entrant in the same category, frequently compared against the others on price and model coverage. Included for completeness of the landscape. If you are running a procurement exercise, the questions that actually separate these products are which models they sample, how often, in which ma... - [Search Engine Roundtable](https://www.seroundtable.com/): Barry Schwartz's long-running daily record of what search representatives actually said and when. Unglamorous and invaluable: when a vendor claims Google confirmed something, this is where you check. The archive stretches back two decades, which makes it the best available defence against confidentl... - [Yoast: the complete guide to SEO](https://yoast.com/complete-guide-seo-copywriting/): A long free guide covering keyword research, search intent, and the actual craft of writing and editing for search -- the practitioner half that Google's documentation deliberately leaves out. Read it after Google's own pages rather than before, so you can tell which parts are Google's position and... - [Google's spam policies](https://developers.google.com/search/docs/essentials/spam-policies): The enforcement side of quality, and far more concrete than the guidance on helpful content. It names the specific practices that get a site demoted or removed: link spam, scaled content abuse, cloaking, doorway pages, expired domain abuse and site reputation abuse. Read it before approving any agen... - [Ahrefs: technical AEO](https://ahrefs.com/aeo/technical-aeo): A practitioner guide to the technical side of answer-engine optimisation, and one of the more grounded pieces of vendor content in the field — it distinguishes what is demonstrated from what is inferred, which most does not. Read as a counterpart to the academic paper: this is what the ideas look... - [Ahrefs: what is llms.txt](https://ahrefs.com/blog/what-is-llms-txt/): The explainer to read before the data, covering what the file is, what it proposes to do, and what adopting it involves. Honest about the uncertainty. Read it, then read the two studies — the sequence of understanding the proposal and then seeing the evidence against it is the most instructive ten... - [SE Ranking: how to optimise for ChatGPT](https://seranking.com/blog/how-to-optimize-for-chatgpt/): A practitioner guide grounded in the finding that community platforms are disproportionately represented in AI citations — which leads to the genuinely useful and slightly uncomfortable conclusion that earning a reputation on forums where people actually discuss your category may do more than anot... - [Semrush blog](https://www.semrush.com/blog/): A large library of research posts and guides, including periodic original data studies on AI search behaviour. Use it as a source of hypotheses and of industry sentiment. Apply the same test as to every vendor publication: is this a sampled measurement they are showing their working for, or an asser... - [Generative AI in Search](https://blog.google/products-and-platforms/products/search/generative-ai-google-search-may-2024/): Google's own announcement of generative answers in Search, and the right primary source for what the company said it was building and why. Read it rather than the coverage of it: the framing Google chose — letting Google do the searching for you — states the strategic intent more plainly than an... - [AI Mode in Search](https://blog.google/products-and-platforms/products/search/ai-mode-search/): The full conversational search surface, distinct from AI Overviews appearing above traditional results. Read after the generative-AI announcement. The distinction matters operationally: Overviews sit on a results page that still has links below, while AI Mode is a different surface with different be... - [Search engine imagery](https://commons.wikimedia.org/wiki/Category:Search_engines): Freely reusable screenshots and diagrams of search interfaces over time. Optional, and genuinely useful for one purpose: putting a results page from ten years ago next to one from today makes the argument about the shrinking link layer faster and more convincingly than any statistic. - [Google Search interface imagery](https://commons.wikimedia.org/wiki/Category:Google_Search): Screenshots of Google's interface across its history, free to reuse. Optional. Useful when building a deck: the visual record of how much of the page has been progressively occupied by answers rather than links is the single most persuasive slide in most presentations on this subject. - [Retrieval in practice: the server-rendering trap](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics): The most common and most expensive technical failure in this field, documented in Google's own crawling guidance: a single-page application that renders content client-side serves an empty shell to anything that does not execute JavaScript. Googlebot does render; many AI crawlers do not. A site can... ## Attribution Curated by SKEMR. The underlying content belongs to the linked publishers. Please cite the source URLs rather than SKEMR for factual claims.