Generative engine optimization

Roboto's own site is the proof of concept. The content is MDX on disk. The markdown is the source, not a format derived from a CMS export. When an LLM crawler asks for markdown at a given URL, it gets a clean document: structured headings, inline links, no CMS wrapper noise.

This matters because content negotiation per page sits on top of it without friction. The same URL serves HTML to browsers and markdown to bots. LLMs ingest the markdown natively. There's no transformation step that could introduce encoding errors, stripped links, or missing context.

We wrote about the full approach in AEO is just SEO with content negotiation. The substrate decision is upstream of every other citation optimization. Get this right first.

Roboto's background agent monitors citation share across platforms on a weekly cadence. When citation share drops for a target prompt cluster, the agent flags it, identifies what changed in the competitive set, and drafts a remediation PR. The human reviews and ships. The loop closes in days, not quarters.

Quarterly advisory cycles break at the feedback step. You get a report in month three that tells you what happened in month one. By the time a content edit makes it through approval and publish, the competitive landscape has shifted again. The diagnostic is stale before the fix ships.

We documented the architecture and the reasoning behind building it in why we built our own background agent. The short version: the only way to consistently move citation share is to make the loop faster than your competitors' loops.

Per Ramp's first-party research, markdown was the only format that reliably surfaced in LLM responses. Bots ask for markdown. Browsers get HTML. The same URL serves both. This is content negotiation, and instrumenting it per page is the single change with the widest citation coverage.

The implementation varies by stack. On a Next.js MDX-on-disk site, it's a route handler that detects the Accept header or User-Agent and serves the raw markdown. On a CMS-driven site, it's a transform step. The goal is the same: a clean markdown document at the canonical URL, no JS dependency, no render blocking.

We cover the specific approach and the monitoring setup in AI content discovery. If you're running a Sanity-backed site, the companion post AEO and SEO best practices for Sanity covers the CMS-specific implementation.

The feedback loop only closes if the data arrives in time to ship a change before the next measurement cycle. Monthly tracking gives you a four-week lag between a citation shift and a potential fix. By the time the fix ships, you've already lost another month. Weekly tracking compresses that to days.

What you're watching: which prompts cite you, which platforms, citation share relative to tracked competitors, and how those numbers moved week over week. Prompt clusters where you dropped are the work queue. Prompt clusters where you gained tell you what's working.

The tooling for this is assembling quickly. Profound's citation tracking, Ahrefs Brand Radar AI Responses, and BrightEdge Generative Parser all publish weekly data. The gap most teams have is wiring the data into a workflow that produces an actionable brief rather than a dashboard nobody checks.

The loop starts with data. We wire the client's stack into the relevant sources: Ahrefs Brand Radar AI Responses, Google Search Console, PostHog source-channel attribution, Serper for prompt-level citation testing, and Profound where the client has access. Each source feeds a different layer of the diagnostic: which prompts cite you, which don't, which competitors gained, which pages are being crawled.

Background agents run on a weekly cadence. They detect citation loss on tracked prompt clusters, citation gain on new clusters, competitor citation movements, and new prompt opportunities the client isn't currently appearing in. The agent doesn't just flag these; it drafts the remediation. A citation drop on a target cluster produces a draft PR: content edit, schema patch, llms.txt update, or content negotiation rule change, depending on what the diagnostic points to.

That PR goes to a human for review. The client approves and ships, or we ship under pre-approved rules for low-risk changes. Either way, the change is live within days of the signal. The agent then re-monitors the affected prompt cluster and measures whether citation share recovered. If it didn't, the next iteration adjusts the approach.

This is Roboto's own infrastructure, running live on this site. We documented the build in why we built our own background agent. We deploy it for clients with their data sources, their content repo, and their approval flow.

Content negotiation per page is where we start. Ramp's first-party finding was direct: markdown was the only format that reliably surfaced in LLM responses. We instrument every key page with a route handler that detects bot traffic and serves clean markdown at the canonical URL. Browsers get HTML. Bots get markdown. No separate URL, no client-side rendering dependency.

Bot classification is where most implementations break. Single-signal classification on User-Agent alone misses the bulk of AI crawler traffic. Correct classification uses UA combined with IP range, ASN, and bot score. The Cloudflare gotcha compounds this: ChatGPT, Perplexity, and Claude bots are categorised as "AI Assistants" in Cloudflare's taxonomy, not "AI Search". Firewall and analytics rules targeting only the "AI Search" category miss all three major platforms.

Schema is a layer we implement correctly. ServiceSchema, FAQPage, BreadcrumbList, and Article where appropriate. We use it because it helps LLMs parse page structure and intent, and we don't oversell it as a citation driver.

llms.txt is another layer. We generate and maintain it. For sites with large content corpora, the prioritisation logic matters: which pages are highest-value for LLM consumption, in what order. A flat file listing 400 pages with no hierarchy is less useful than a structured file that guides the crawler to the 20 pages that matter most.

Crawlability is non-negotiable. Every list item renders into the DOM. Navigation is in static HTML. Status codes are correct. Sitemaps are clean and submitted. These are table stakes; we verify them as part of every audit before we touch the higher-leverage layers.

Volume compounds citation share, but only if the content clears a quality bar that LLMs trust. The editorial infrastructure is what makes volume safe to ship. We build four gates into every publishing pipeline.

First, an AI-tell detector that runs before merge. It flags em-dashes, negative parallelism patterns, banned phrase families, and AI vocabulary (the "leverage", "seamlessly", "delve" cluster). Content that fails the gate goes back for a humanizer pass before it's eligible for review.

Second, a fact-checker agent that verifies specific claims against primary sources. Blog posts with statistics or attributed quotes get each claim checked against the cited source. If the citation doesn't support the claim, the agent flags it for writer review. This is what separates content a trusted source would publish from content that erodes trust over repeated encounters.

Third, a humanizer pass on AI-assisted drafts. The pass strips AI vocabulary patterns and sentence structures that read as generated. The goal is copy that sounds like a practitioner wrote it, because a practitioner should be writing it, with AI handling research and first-draft structure.

The cms-auto-updater pipeline is the worked example. It runs on a weekly cadence, pulls evidence from primary sources, drafts content updates, runs all three gates, and opens a PR for review. No unverified copy ships. The result is 5 to 10 times the content output per quarter without the citation-eroding slop that comes from raw AI publish pipelines.

Citation tracking requires more than one tool because different platforms surface different data. For citation share by prompt cluster, Profound is the most specific tool available. BrightEdge Generative Parser covers broader industry benchmarks and quarterly trends. Ahrefs Brand Radar AI Responses shows which prompts trigger brand mentions across AI platforms. Semrush's AI toolkit adds coverage, particularly for clients already in the Semrush ecosystem.

Crawl observability is separate. Cloudflare Radar gives aggregate bot traffic share. Server log analysis, following Merj's playbook, gives you ground truth on which crawlers are actually hitting your site versus what the UA headers claim. These two together tell you where you have crawl gaps that no amount of content optimization will fix without first fixing access.

Social citation sources deserve dedicated attention given the Profound data. Reddit at 24% of Perplexity's citations is not a marginal signal; it's a primary one. For clients with B2B or technical audiences, Reddit threads, YouTube transcripts, Medium posts, and LinkedIn articles that discuss their space are potential citation surfaces worth instrumenting. The strategy varies by vertical: what works for a developer tool is different from what works for an enterprise SaaS product.

Search-side telemetry ties the loop together. Google Search Console shows which queries are driving impressions and clicks from traditional search, and how that share moves as AI Overviews expand. Ahrefs Brand Radar AI Responses shows the AI-side equivalent. PostHog source-channel attribution shows which traffic is arriving via AI referrals versus organic. Together, these three sources tell you whether a citation gain in AI search is translating to measurable traffic, and whether the traffic is converting.