Technical setup: robots.txt, llms.txt and schema

Everything in this module rests on one sentence: content a crawler cannot read never reaches the answer. Technical setup is unglamorous, but skip it and the rest of the work is wasted.

1. AI crawler access (robots.txt)

AI crawlers use user-agent names distinct from classic search bots. On many sites they are blocked without anyone realising — usually by a broad rule added with the intent of "blocking bots".

• GPTBot — OpenAI's crawler (training and search).
• OAI-SearchBot — used for ChatGPT search results.
• PerplexityBot — Perplexity's crawler.
• ClaudeBot — Anthropic's crawler.
• Google-Extended — the signal governing Gemini's use of your content.

The decision here is strategic, not technical: do you want your content used in AI answers? If you want visibility, you have to allow access — blocking crawlers and then asking why you never appear is incoherent. If your content is valuable enough that you would rather it not be quoted, blocking is a legitimate choice; just make it knowing you are giving up the visibility goal.

2. llms.txt: a roadmap for crawlers

llms.txt is a plain-text file at your site root that tells AI crawlers "here's a summary of my site, and here are the pages that matter". It is not a mandated standard — it is an emerging convention. On its own it is not magic; its job is to make it easier for a crawler to find your best pages.

A good llms.txt contains information, not marketing: a flat description of what the brand does, definitions of key concepts, links to important pages, and contact details. That is what makes it quotable.

3. Structured data (schema.org)

Schema markup restates what is on the page in a form a machine can read without interpretation. Three types matter most for AI visibility:

• Organization — the brand's name, logo, description and official profiles. Helps a model match you to the right entity.
• FAQPage — question-and-answer structure. The most directly useful type, because it is exactly the shape answer engines are looking for.
• Article / BlogPosting — headline, publish and update dates, author. This is where freshness is read.

4. Rendering and accessibility

• If content loads via JavaScript after the fact, many crawlers never see it. Critical text must be in the first HTML response.
• Never bury key information in an image; alt text is compensation, not a solution.
• Page speed directly affects how deeply a crawler explores your site.
• Keep canonical tags consistent — the same content living at several URLs splits the signal.