Fix agent readiness
Fixesllms.txt, AI crawler rules and structured data, based on your Agent Readiness scan.
On each run, it works through these steps:
- If the last scan is more than 20 hours old, Autopilot rescans your site and waits up to four minutes. If the rescan doesn’t finish in time, it uses the last scan.
- It sorts the issues into what the repository can fix and what needs product work. Checks like an OpenAPI spec, an MCP server, OAuth or rate limit headers need changes to your product itself, so Autopilot leaves them alone. Bonus checks are skipped, and so are issues you dismissed in the last 30 days or merged in the last 14.
- An agent reads your repository and makes the fix:
llms.txtwritten from your brand description and site facts, AI crawler rules inrobots.txt, sitemap references, JSON-LD and metadata. It prefers additive changes, and it edits your framework’s own source (like a Next.jsapp/robots.ts) instead of adding a static file that conflicts with it. - It opens a pull request titled after the fix. The body ends with the scan URL, the readiness score and the framework it detected.
package.json, lockfiles, CI or environment files, and never adds dependencies.
When it doesn’t open a pull request, the run log says why:
After you merge a fix, its issues are left alone for 14 days while it deploys and the next scan picks it up. Closing a fix keeps Autopilot away from those issues for 30 days.
Make pages readable for AI
Fixes content AI crawlers can’t see without JavaScript. Many AI crawlers don’t run JavaScript, so a page that renders in the browser can be nearly empty for them.
Each run goes like this:
- It checks up to eight pages: your homepage, the pages AI cites most in the last 30 days, then pages from your sitemap.
- It fetches each page as
ChatGPT-Userwithout JavaScript and compares it with what a browser shows. A page fails when the browser shows at least 150 words and the raw HTML has less than half of them. - It moves the missing content into the server’s HTML or the build output, for example by removing
ssr: false,client:onlyor a guard that only renders after mount. - It adds no copy (at most 20 words where markup needs them). The pull request explains how it checked each page.
401, 403, 429 or a bot challenge while a browser gets the page, a firewall or CDN rule is blocking them. Autopilot can’t change that from your repository, so it tells you in the brief: “AI crawlers get 403 from … while a browser gets the page: a bot wall or firewall rule blocks them. Allow ChatGPT-User, PerplexityBot and Claude-User in your CDN or firewall.”
The run log explains every run that didn’t open a pull request:
Related
Agent readiness
What the scan checks and how the score works.
AI crawlers
See which AI crawlers visit your site.