In my last post, I looked at how the web is changing for agents and what agent-ready means: a site that lets an agent read its content, query it for specific answers, and act on it, while site owners can tell which agents are visiting and decide what they may do. This post goes through the approaches site owners can use to get there, how far along each one is, and what we know about how agents use them.
A handful of proposals have made it off the drawing board and onto a lot of real sites, although most are still not published standards:
- llms.txt and markdown versions of pages help agents read
- Structured data and Model Context Protocol (MCP) servers help them query
- Discovery files tell them what’s there
- Content Signals covers part of the trust question
- WebMCP, which will get its own post, is aimed at taking action
llms.txt: the agent “index”
llms.txt was originally proposed by Jeremy Howard at llmstxt.org, and it’s a markdown file at the root of a site, like example.com/llms.txt, that gives language models a short, curated map of the site. It opens with the site’s name and a one-line summary, then lists the most useful pages under headings, each with a link and a sentence about what’s there. Instead of having to crawl every page, an agent can read it in one go and know what the site is about.
Despite the similar naming convention, it doesn’t work like robots.txt. robots.txt tells crawlers what they may fetch, while llms.txt doesn’t control anything – it’s just an index which acts as a sort of FYI to agents.
Although it hasn’t been adopted by any standards body, and AI companies don’t say whether their crawlers actually read it, its usage has spread. Even OpenAI, Anthropic, and Google each publish one for their developer docs, and documentation platforms like Mintlify and GitBook generate theirs automatically. Lighthouse’s Agentic Browsing category checks for one as well.
The proposal reached version 2 this year, which adds one requirement: the site should point to the file with a Link header or a tag, so agents know exactly where it sits (more on both in the discovery section below).
Markdown versions of pages: smaller and cheaper
Sites can offer markdown versions of their pages for agents to consume instead of the frontend HTML. The same page in markdown is a fraction of the size of its HTML, because the navigation, scripts, styling, and ads are gone and only the core content is left. In an example shared by Cloudflare, one page dropped from 16,180 tokens as HTML to 3,150 as markdown. The advantage of a smaller page is that it’s cheaper for the agent to parse, and leaves more room in its context window for the actual question.
Sites can offer markdown in two ways:
- Separate .md web address: this usually sits at the page URL with .md added to the end. This approach is widely used, including in Anthropic’s and Stripe’s documentation. An agent has to know that address exists, so sites usually point to it with a tag, a Link header, and/or a listing in llms.txt.
- Content negotiation: the agent requests the ordinary page URL with an Accept: text/markdown header, and the server answers in markdown instead of HTML. Coding agents like Claude Code and OpenCode already send that header.
In practice, the two often work from the same content. A site, or the plugin that handles this for it, can generate the markdown version of a page once, when the page is published or updated, and store it. Both the .md address and a negotiated request then pull from that one stored copy, and the only difference is how the agent asked for it. Working from one source is more efficient than converting the page on every request, and it means an agent gets the same, up-to-date content whichever way it asks.
Offer both the .md page and content negotiation
It’s a good idea to offer both the .md address and content negotiation, so an agent gets markdown regardless of its preference. Based on some analysis, agents seem to fetch .md addresses directly more often than they negotiate, possibly because sites point to those addresses with tags, Link headers, and llms.txt files, while negotiation catches the agents that ask for markdown at the normal URL. When both routes pull from the same stored copy, offering the second one doesn’t cost anything and isn’t any more complicated.
Negotiation does need careful caching, though. The HTML and markdown versions share one URL, so the server has to send a Vary: Accept header, and any page cache or CDN (content delivery network) in front of the site has to respect it and store the two versions separately. If it doesn’t, an ordinary visitor can be served the cached markdown instead of the HTML page, and the browser may treat it as a file to download. Cloudflare adds Vary: Accept to its converted responses for this reason. Check with your host whether its page cache keeps the two versions apart.
A different approach: Cloudflare’s Markdown for Agents
Cloudflare takes a different route with Markdown for Agents, which it launched in February 2026. It also uses content negotiation, but it doesn’t ask for a markdown version stored on your site – it fetches the page’s normal HTML from your server and converts it at its edge. The conversion strips the navigation, scripts, and styling, puts the page’s title and description at the top, and adds any JSON-LD structured data in a block at the end. Each converted response also carries an x-markdown-tokens header with an estimate of how many tokens the page will use, so an agent can decide how much of its context window to spend on it.
That makes it the simplest option for a site already using Cloudflare, since it works on any platform and there’s no second copy of the content to keep up to date. However, the markdown is only as good as the HTML it draws from (for example, if the page is hard to parse or the content only appears through JavaScript), and you can’t edit or control what Cloudflare produces.
Discovery: telling agents what’s there
Discovery is how an agent finds out what a site offers for machines without guessing at URLs. Most of it uses parts of the web that already exist.
Two sources for agent discovery have already come up in this post: the tag and the Link header, which sites use to point agents to llms.txt and to markdown versions of pages. The link tag is the older of the two. It sits in a page’s
, and WordPress has used to point to its RSS feeds since the early 2000s. The Link header carries the same information in the response headers, so an agent can see everything a site makes available without downloading or parsing the page, and one header can point to llms.txt, the sitemap, the feed, and an API catalog at once.
The newer type of discovery source is a set of agent-focused files under /.well-known/, a folder the web set aside in 2010 for site-wide files at fixed addresses. It already holds older files, like security.txt, and the agent files are the latest additions:
- /.well-known/api-catalog lists a site’s APIs. It’s defined in RFC 9727, published in 2025, and it’s the only one of the new agent-focused proposals in this post that has become a published standard, reviewed and agreed upon by the IETF (Internet Engineering Task Force).
- /.well-known/agent-skills/index.json lists Agent Skills, which are short markdown files that teach an agent how to use the site. The skill format came from Anthropic and is documented at agentskills.io. The discovery convention for this is a draft led by Cloudflare.
- MCP server cards describe a site’s MCP server, including where to connect and which protocol versions it supports, and a site lists them in an AI catalog at /.well-known/ai-catalog.json. A2A (Agent2Agent) agent cards do the same for a site’s own AI agent, at /.well-known/agent-card.json. Both are recent, and few sites or agents use them yet.
These files help an agent that is built to look for them, but most mainstream agents aren’t yet. Agent-ready checkers and rankers do score sites on these files, so you’ll see them in agent-readiness reports either way.
Content Signals: agent permissions
robots.txt has always said who may fetch what, and now Content Signals adds a line saying what agents and bots may or may not do with the information they fetch:
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no- search covers search indexing
- ai-input covers using a page as live input to an AI answer
- ai-train covers training models
The above example says yes to search and AI answers, and no to training. Cloudflare introduced this format in 2025 and adds it to the robots.txt files it manages for its customers. The same signals can also be sent as a Content-Signal header on each page.
Cloudflare’s Markdown for Agents (mentioned above) uses this header, and if your server doesn’t send one, the markdown Cloudflare serves for your pages defaults to ai-train=yes, search=yes, ai-input=yes. Its docs don’t mention checking your robots.txt first, so if you don’t want your content used for training, your server needs to send its own header as well to be on the safe side.
The IETF’s AI Preferences working group is writing a standard vocabulary for the same three ideas. Agents don’t have to respect these signals, since they’re voluntary for now, and most crawlers don’t read them yet, but they’re still the clearest way to state your position, so they’re a “might-as-well” type of implementation.
Structured data: old, simple, and still useful
Schema.org structured data has been around since 2011, when Google, Bing, and Yahoo launched it together so while it’s not among the new approaches to agent-readiness, it continues to matter. Schema.org is a shared vocabulary of types, like Product, Event, Article, Organization, and FAQPage, each with its own properties. A site uses it to label the facts on a page, usually in a block of JSON-LD, which is the format Google recommends. The block sits in a
For an agent, this means the facts are labeled and easy to understand, and it doesn’t have to try to work out from the layout which number on the page is the price, or whether the date near the top is when an article was published or when it was last updated. Properties like sameAs also help make it clear which company or person a page is about by linking them to their Wikipedia or LinkedIn page. It’s the same markup Google uses for rich results, like the star ratings and prices in search listings, so many sites already have some, and for stores, events, and recipes it’s the most widely deployed machine-readable layer on the web.
Structured data does have a drawback, though: many of the tools agents use to fetch pages turn the HTML into plain text or markdown and drop every
Structured data shows how far basic on-page hygiene can go for agents. It’s SEO work that sites have been doing for 15 years, and agents can use it as is. A lot of WordPress SEO plugins generate most of it automatically from content that’s already on the site, and Elementor’s widgets include schema settings of their own, so many site owners have it already.
MCP servers
MCP, the Model Context Protocol, was introduced by Anthropic in late 2024, and most major AI assistants and coding tools now support it. Where llms.txt and markdown help an agent read a site, an MCP server lets it ask the site direct questions and get answers back. The server publishes a list of tools, each with a name, a description, and a schema for its inputs, and an agent connected to it can call those tools without clicking through the site. A server can also offer resources, which are documents or records the agent can read, and prompts, which are ready-made instructions, though tools are what most servers lead with. Most site-level servers now run remotely over HTTP, at an address on the site, so nothing has to be installed on the user’s computer.
Companies mostly run one so people can work with them from the AI assistant they already use. For example, Microsoft runs a free, public MCP server for its Learn documentation, so a developer can search and read Microsoft’s docs from inside their coding tool. Stripe’s server at mcp.stripe.com lets a signed-in user work with the customers, payments, and invoices in their own account. According to Shopify, every Shopify store has a storefront MCP server at /api/mcp, which an agent can use to search the store’s catalog, answer questions about its policies, and manage a shopping cart.
These tools give agents live and precise answers like the current price or inventory level, and a request like “find in-stock mugs under $30” can be a single call with typed inputs instead of the agent having to read through several pages.
To use any of these, someone has to connect the server by adding its address to their AI assistant, and in some cases the user also has to sign in to an account. Discovery is starting to help: the official MCP Registry, launched in preview in September 2025, lists public servers, and the MCP server cards from the discovery section let agents find a site’s server through its AI catalog. A person still makes the connection, though, so while MCP suits site owners, teams, and customers who use a service regularly, it isn’t practical for an ordinary visitor. WebMCP, a proposal from engineers at Google and Microsoft, is aimed at filling that gap: it lets a web page offer tools directly to an AI agent working in the visitor’s browser, with no server to run and nothing for the visitor to connect. We’ll look at WebMCP more closely in the next post.
Early-stage agent-ready proposals
This space is changing fast, and new ideas keep being proposed. Here are some early-stage ones that are also worth knowing about:
- Web Bot Auth lets an agent sign its requests so a site can check who it really is, instead of relying on the user-agent string, which can be spoofed. It’s still in IETF drafts, and addresses an important area related to agent trust and security that needs attention.
- NLWeb, from Microsoft, gives a site an /ask endpoint that answers questions in natural language. Adoption seems low for this.
- DNS-based discovery proposes a way to find a site’s agent services through its DNS records, before loading anything. It’s still an early IETF draft.
- Commerce has its own protocols for agents that buy things, from OpenAI, Google, and the payment companies. They’re moving fast and will matter a lot for online stores.
What you can do now
Most of what helps agents today helps people too, and it’s all worth doing regardless of where these specs land:
- Make the page itself easy to read. Use real headings in order, links that are links, buttons that are buttons, and content that’s in the HTML when the page loads rather than loaded later by JavaScript. Agents read the HTML page more than anything else, so this has the biggest effect.
- Fix your forms for accessibility: labels, autocomplete values, error messages tied to their fields with aria-describedby, and a status message that announces success. Accessibility work helps agents use forms too, and Elementor’s Ally Checker is one way to find what needs fixing.
- Add structured data. Products, events, organizations, articles, and FAQs all have schema.org types, and JSON-LD is the format both search engines and agents read.
- Check what you’re blocking. Look at your robots.txt, and also at the bot settings in your host and CDN, which can block AI agents without the site owner knowing. Decide which agents you want, then use Content Signals to say what they may do with your content.
- Offer markdown versions of your pages, both as .md addresses and through content negotiation on the normal URL. There are many solutions for this, including Elementor’s Generate website markdown feature. Whichever you use, make sure your page cache keeps the markdown and HTML versions separate and equally available.
- Add llms.txt, and point to it with a Link header or tag. While it doesn’t change much on its own, you might as well add it since it takes minutes to set up.
- Advertise what you publish. If you have feeds, a sitemap, markdown pages, or an API, list them in Link headers and tags so an agent can easily find them.
- If you run WordPress, you’ve got a head start for all the reasons detailed in my post on WordPress being born ready: feeds, sitemaps, and a REST API all built in. The Abilities API in core, with the MCP Adapter, lets plugins describe what an agent can do on the site and expose it over MCP.
- Measure what agents do. Your logs show which agents visit and what they request. If you keep an eye on them, you can get more insight into what agents actually do on your site, and where you should put your focus.
For a full checklist, The Website Specification lists what a good site should have, agent readiness included, with each item tagged as required, recommended, optional, or avoid, and a link to the standard behind it.
Where this leaves us
A lot of what agents actually use today is related to the regular web page: HTML that’s easy to read, structured data that identifies the facts, and markdown versions of the same pages. The newer files, like llms.txt and the ones under /.well-known/, take little effort to add, and can help agents that are built to look for them. Trust is the area that still needs work, since Content Signals depends on crawlers choosing to respect it and Web Bot Auth is still in drafts. MCP handles querying, and even acting, once someone connects it, which makes it a better fit for a site’s regular users than for a first-time visitor.
My next post will pick up from here and look at WebMCP, which aims to handle the many use cases where an agent should be able to submit a form, book a time, or buy something for an ordinary visitor on a site.
