1---2name: agent-readiness3description: >-4 Checks whether AI crawlers and AI agents can read and use a B2B SaaS site: robots.txt rules per crawler (search,5 user-triggered and training kept apart), what the firewall returns to each, whether key facts such as pricing are6 in the HTML without JavaScript, indexing and snippet eligibility, structured data that matches the page, and the7 forms agents act on. Says what Google says matters and what it calls unnecessary, and treats llms.txt as optional.8 Returns PASS, PASS WITH NOTES, FAIL or BLOCKED. Use it before any AI visibility work and after every site release.9 Not for measuring visibility (use ai-answer-panel) or spam risk (use ai-search-risk).10license: MIT11compatibility: >-12 Works in Claude, ChatGPT, Codex, Cursor and any agent that reads SKILL.md. Fetches robots.txt and pages if the agent13 has network access, otherwise pasted files and curl results. Python 3 is optional, for the robots, HTML and fetch14 checks.15metadata:16 author: SHAKEUP17 version: "1.0.0"18 stage: check19 plugin: seo-ai-search20 homepage: https://www.shakeuphq.com/skills/agent-readiness21---22 23# Agent readiness24 25Finds what stops AI crawlers and agents from reading a site before anyone spends a quarter on content and outreach that a blocked page can't use.26 27## Purpose28 29- One job: test a site's priority pages for crawler access, rendering, indexing, structured data and agent usability, and return findings with severities.30- Stage: check. It ends in PASS or FAIL with severities.31- It is accurate about the platforms: it says what Google says matters and what Google calls unnecessary, and never sells llms.txt or special AI markup as a visibility lever.32- It never changes robots.txt, the firewall or the site. It writes the fix and who owns it.33 34## Use when / not35 36| Situation | Use |37| --- | --- |38| Before AI visibility work, or when AI answers never cite the site | this skill, first |39| After a site release, a CDN or firewall change, or a redesign | this skill, on the same priority pages |40| "Do we need llms.txt?" | this skill (the answer is in step 7) |41| Measuring how often AI answers name you | `ai-answer-panel` |42| A tactic might be spam | `ai-search-risk` |43 44## Inputs45 46| Input | Required | Ways to provide, in order | If missing |47| --- | --- | --- | --- |48| The site and its priority pages, about 10 | yes | the reader's list; else homepage, pricing, 2 or 3 product pages, integrations, a comparison page, security, docs, the best case study | `BLOCKED` without the site; propose the 10 from the sitemap |49| robots.txt | yes | fetch `/robots.txt`; the file the reader pastes | a warning: crawler rules not tested |50| Raw HTML of each priority page, without JavaScript | no | fetch it if the agent can; `curl` output or "view source" the reader pastes | a warning: rendering not tested |51| Status per crawler user agent, from outside the company network | no | `scripts/agent_check.py fetch`; `curl -A "<agent>"` results the reader pastes; a crawler-checker tool's result | a warning: firewall not tested |52| Server logs, 30 days, filtered by AI user agents | no | the reader's log tool or CDN analytics | say "no log evidence" |53| Search Console: indexing, manual actions, the generative AI features setting | no | a connected Search Console; screenshots | a warning: indexing and the AI setting not checked |54 55## Profile read56 57From `company-profile.md`: `## Company` (website, pricing, so the check knows what should be visible), `## Product` (pages to treat as priority), `## Accounts and data` (Search Console, analytics), and every `## Learnings` entry for `agent-readiness` with status `noted`, `fact` or `accepted`, including any written policy on training crawlers.58 59No profile: ask for the website and the 10 pages a buyer's question would lead to, in one message. Say that `company-profile` saves this next time.60 61## Steps62 631. **Apply learnings.** Read them first. A recorded policy ("we block training crawlers on purpose") turns those blocks into no finding.64 652. **Fix the page list.** About 10 priority pages, each with its job (pricing, product, comparison). A release check uses the same list as the last run.66 673. **Test crawler rules.** With Python:68 69 ```70 python3 scripts/agent_check.py robots --file robots.txt --path /pricing --path /product/<name>71 ```72 73 Without Python, read each `User-agent` group: a crawler follows its own group, else `*`; the longest matching rule wins and `Allow` wins a tie. Then classify every block with the crawler table in `references/sources.md`: search and user crawlers blocked is critical; a training crawler blocked is a choice, a suggestion to document it; Google-Extended blocked doesn't touch Google Search.74 754. **Test the firewall.** With network access:76 77 ```78 python3 scripts/agent_check.py fetch --url https://<site>/pricing79 ```80 81 Or ask the reader to run, from outside the company network, `curl -A "OAI-SearchBot" -o /dev/null -s -w "%{http_code}" https://<site>/pricing` for each crawler. Anything other than 200 with real content (a 403, a 429, a challenge page) is critical. A firewall that verifies crawlers by IP may block a spoofed request and let the real crawler through: confirm in server logs before anyone changes a rule.82 835. **Test the HTML without JavaScript.** For each priority page:84 85 ```86 python3 scripts/agent_check.py html --file pricing.html87 ```88 89 Without Python, view the source and search it for the facts that matter: the prices, the plan names, the integration list, the one-line description of the product. A fact that appears only after JavaScript runs, or only in an image, a PDF or a closed tab, is missing for crawlers that don't render. Check meta robots for noindex and nosnippet, and the canonical.90 916. **Check what agents act on.** On the demo or contact form, pricing and signup: real buttons and links, a label tied to every field, no overlay over the controls, the main action in the same place across pages.92 937. **Place the optional items honestly.**94 95 | Item | Severity | Why |96 | --- | --- | --- |97 | llms.txt missing | none; mention only | Google says Search ignores it; it may help agents reading developer docs, so it's half a day for an API product and optional for the rest |98 | No Organization or Product structured data | suggestion | not required for Google's AI features; it helps rich results and ties the site to its profiles |99 | Structured data that contradicts the page (a price only in the markup) | warning | contradictions cost trust, and Google's structured data policies can remove rich results for it |100 | Pricing not public ("contact sales") | warning | answers to budget questions can't use what isn't published; a business call, so a warning, not a failure |101 1028. **Write each finding** with the page, the severity, the evidence (the rule, the status, the missing fact), the fix and the owner (engineering, security or infrastructure, marketing).103 104## Check and verdict105 106| Finding | Severity |107| --- | --- |108| Googlebot, Bingbot, or a search or user crawler blocked by robots.txt on a priority page (with no written policy) | critical |109| A non-200, a challenge page or a rate limit to a search or user crawler | critical |110| noindex or nosnippet on a priority page | critical |111| A key fact (price, product description, integrations) missing from the HTML without JavaScript | critical |112| The site excluded from generative AI features in Search Console | critical |113| A test not run (crawler rules, firewall, rendering, indexing) | warning |114| Structured data contradicting the page; a canonical pointing elsewhere; unlabeled form fields | warning |115| Pricing not public | warning |116| A training crawler blocked with no written policy; Google-Extended blocked; no structured data | suggestion |117 118| Verdict | Condition (first match wins) |119| --- | --- |120| `BLOCKED` | No site after asking |121| `FAIL` | 1 or more critical findings |122| `PASS WITH NOTES` | Warnings or suggestions only, including any test not run |123| `PASS` | Every test run, no findings |124 125No named stops. A `PASS` needs every test run; the check never calls a site ready on robots.txt alone.126 127## Learn128 129Append a learning to the profile when:130 131- the reader states a crawler policy (a `fact`: which bots are blocked on purpose and who decided), so later runs don't flag it;132- a fix ships and is retested (a `fact` with the date and the release);133- a release breaks something this check catches (a `fact`, and a `proposal` to run the check in the release process).134 135Example: `- 2026-10-24 | agent-readiness | fact | GPTBot and ClaudeBot blocked on purpose; search and user crawlers allowed | Why: legal decision, 2026-09, owner Head of Security | Status: noted`136 137## Output138 139```140Verdict: <PASS | PASS WITH NOTES | FAIL | BLOCKED> · Coverage: <complete | partial (tests not run)> · Checks: <script | by hand>141Counts: <n> critical, <n> warnings, <n> suggestions · Pages: <n>142 143| Page | Robots (search / user / training) | Firewall | Facts in HTML | Indexable, snippet | Structured data | Finding | Severity | Fix | Owner |144 145Crawler policy: <what is blocked on purpose, or "none written">146llms.txt: <present | absent>, optional (Google Search ignores it)147Not tested: <list, with what would test each>148Learnings applied: <list, or none>149```150 151Then the handoff block.152 153## Hand off154 155```handoff156skill: agent-readiness157version: 1.0.0158verdict: FAIL159coverage: partial (no server logs)160for: <site>, <date or release>161outputs:162 critical: 2163 blocked_crawlers: [OAI-SearchBot, PerplexityBot]164 js_only_facts: [/pricing]165 retest: after the robots.txt change and the pricing release166next: [ai-answer-panel]167open: [confirm the 403 in server logs]168```169 170## Failure modes171 172| Problem | Do this |173| --- | --- |174| robots.txt returns a 404 | Google treats a 4xx (except 429) as no robots.txt, so nothing is blocked by it; say so, and check the firewall anyway |175| robots.txt returns a 5xx | Google stops crawling the site for the first 12 hours and then falls back to its cached copy; critical for the whole site. Source: Google's robots.txt documentation, https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt |176| A spoofed user agent gets a 403 but logs show the real crawler getting 200 | No finding; note the firewall verifies crawlers |177| The reader wants an llms.txt file written | Write it if they want it for agents or docs, and say it won't change Google visibility |178| The site is a single-page app with server rendering on some routes only | Test each priority page; findings are per page |179| A CDN serves a cached shell to bots | Test with a cache-busting query string and compare |180 181## Influenced by182 183- Google's guide to optimizing for generative AI features, for what Google says a page needs and what it calls unnecessary (including llms.txt and special markup): https://developers.google.com/search/docs/fundamentals/ai-optimization-guide184- OpenAI's crawler documentation, which separates the search crawler, user fetches and the training crawler: https://developers.openai.com/api/docs/bots185- web.dev's guide to agent-friendly sites, for the form and control checks: https://web.dev/articles/ai-agent-site-ux186- SHAKEUP's AI visibility playbook: the 10-URL audit, the security layer tested before robots.txt, and training and search crawlers as separate decisions.