Skip to main content
AI engines can only cite pages they can read. These two automations handle the technical side: the files and markup agents look for, and content that only shows up after JavaScript runs.

Fix agent readiness

Fixes llms.txt, AI crawler rules and structured data, based on your Agent Readiness scan. On each run, it works through these steps:
  1. If the last scan is more than 20 hours old, Autopilot rescans your site and waits up to four minutes. If the rescan doesn’t finish in time, it uses the last scan.
  2. It sorts the issues into what the repository can fix and what needs product work. Checks like an OpenAPI spec, an MCP server, OAuth or rate limit headers need changes to your product itself, so Autopilot leaves them alone. Bonus checks are skipped, and so are issues you dismissed in the last 30 days or merged in the last 14.
  3. An agent reads your repository and makes the fix: llms.txt written from your brand description and site facts, AI crawler rules in robots.txt, sitemap references, JSON-LD and metadata. It prefers additive changes, and it edits your framework’s own source (like a Next.js app/robots.ts) instead of adding a static file that conflicts with it.
  4. It opens a pull request titled after the fix. The body ends with the scan URL, the readiness score and the framework it detected.
It never changes package.json, lockfiles, CI or environment files, and never adds dependencies. When it doesn’t open a pull request, the run log says why: After you merge a fix, its issues are left alone for 14 days while it deploys and the next scan picks it up. Closing a fix keeps Autopilot away from those issues for 30 days.

Make pages readable for AI

Fixes content AI crawlers can’t see without JavaScript. Many AI crawlers don’t run JavaScript, so a page that renders in the browser can be nearly empty for them. Each run goes like this:
  1. It checks up to eight pages: your homepage, the pages AI cites most in the last 30 days, then pages from your sitemap.
  2. It fetches each page as ChatGPT-User without JavaScript and compares it with what a browser shows. A page fails when the browser shows at least 150 words and the raw HTML has less than half of them.
  3. It moves the missing content into the server’s HTML or the build output, for example by removing ssr: false, client:only or a guard that only renders after mount.
  4. It adds no copy (at most 20 words where markup needs them). The pull request explains how it checked each page.
When crawlers get a 401, 403, 429 or a bot challenge while a browser gets the page, a firewall or CDN rule is blocking them. Autopilot can’t change that from your repository, so it tells you in the brief: “AI crawlers get 403 from … while a browser gets the page: a bot wall or firewall rule blocks them. Allow ChatGPT-User, PerplexityBot and Claude-User in your CDN or firewall.” The run log explains every run that didn’t open a pull request:

Agent readiness

What the scan checks and how the score works.

AI crawlers

See which AI crawlers visit your site.
Last modified on November 18, 2026