Skip to content

Checks

Agent readiness

Checks whether AI crawlers and agents can read and use your site, from robots rules and firewall blocks to pricing that only appears with JavaScript.

Other tools

Add this skill to Claude

Claude takes skills as a ZIP file. Your download has started; if it didn't, use the button below.

  1. 1In Claude, open Customize, then Skills.
  2. 2Choose +, then Upload a skill, and pick agent-readiness.zip.
  3. 3Ask for it by name, or let Claude use it when a task matches.

Skills need code execution turned on in Claude's settings. On a Team or Enterprise plan, an admin can share it with everyone.

Download the folder

Try it now, without installing

See the skill work on a made-up input before you add it. Nothing is installed, and you can swap in your own input.

Open in the Claude app

Opens a new chat in the Claude desktop app with the prompt filled in. Nothing is sent until you press enter.

Includes the whole SKILL.md and the sample input. Paste it into Claude, ChatGPT or any chat.

Influenced by
Optimizing for generative AI features, Crawler documentation, Agent-friendly websites
Needs
Your site and its 10 priority pages · Your robots.txt
Works with
Claude, ChatGPT, Codex, Cursor and any agent that reads SKILL.md
Version
1.0.0 · Last checked · License MIT
Maintained by
Sabih AhmedOrganic visibility
SkillSEO and AI search

Agent readiness

Example output

Check AI crawler access
CheckResultSeverity
OAI-SearchBotBlockedCritical
GPTBotBlocked on purposeNote
Homepage HTMLFull textPass

Checks whether AI crawlers and agents can read and use your site, from robots rules and firewall blocks to pricing that only appears with JavaScript.

The skill

The whole skill, in plain text.

Nothing is hidden. Read it, change it to fit your team, or take the whole folder.

Scripts
Optional Python scripts for the counting. It works without them and says which way it ran.
Network
Opens web pages or your tools if your agent can. Works from pasted data if not.

3 test cases

agent-readiness/SKILL.md

In the folder

  • agent-readiness/SKILL.md
  • agent-readiness/references/sources.md
  • agent-readiness/scripts/agent_check.py
---name: agent-readinessdescription: >-  Checks whether AI crawlers and AI agents can read and use a B2B SaaS site: robots.txt rules per crawler (search,  user-triggered and training kept apart), what the firewall returns to each, whether key facts such as pricing are  in the HTML without JavaScript, indexing and snippet eligibility, structured data that matches the page, and the  forms agents act on. Says what Google says matters and what it calls unnecessary, and treats llms.txt as optional.  Returns PASS, PASS WITH NOTES, FAIL or BLOCKED. Use it before any AI visibility work and after every site release.  Not for measuring visibility (use ai-answer-panel) or spam risk (use ai-search-risk).license: MITcompatibility: >-  Works in Claude, ChatGPT, Codex, Cursor and any agent that reads SKILL.md. Fetches robots.txt and pages if the agent  has network access, otherwise pasted files and curl results. Python 3 is optional, for the robots, HTML and fetch  checks.metadata:  author: SHAKEUP  version: "1.0.0"  stage: check  plugin: seo-ai-search  homepage: https://www.shakeuphq.com/skills/agent-readiness--- # Agent readiness Finds what stops AI crawlers and agents from reading a site before anyone spends a quarter on content and outreach that a blocked page can't use. ## Purpose - One job: test a site's priority pages for crawler access, rendering, indexing, structured data and agent usability, and return findings with severities.- Stage: check. It ends in PASS or FAIL with severities.- It is accurate about the platforms: it says what Google says matters and what Google calls unnecessary, and never sells llms.txt or special AI markup as a visibility lever.- It never changes robots.txt, the firewall or the site. It writes the fix and who owns it. ## Use when / not | Situation | Use || --- | --- || Before AI visibility work, or when AI answers never cite the site | this skill, first || After a site release, a CDN or firewall change, or a redesign | this skill, on the same priority pages || "Do we need llms.txt?" | this skill (the answer is in step 7) || Measuring how often AI answers name you | `ai-answer-panel` || A tactic might be spam | `ai-search-risk` | ## Inputs | Input | Required | Ways to provide, in order | If missing || --- | --- | --- | --- || The site and its priority pages, about 10 | yes | the reader's list; else homepage, pricing, 2 or 3 product pages, integrations, a comparison page, security, docs, the best case study | `BLOCKED` without the site; propose the 10 from the sitemap || robots.txt | yes | fetch `/robots.txt`; the file the reader pastes | a warning: crawler rules not tested || Raw HTML of each priority page, without JavaScript | no | fetch it if the agent can; `curl` output or "view source" the reader pastes | a warning: rendering not tested || Status per crawler user agent, from outside the company network | no | `scripts/agent_check.py fetch`; `curl -A "<agent>"` results the reader pastes; a crawler-checker tool's result | a warning: firewall not tested || Server logs, 30 days, filtered by AI user agents | no | the reader's log tool or CDN analytics | say "no log evidence" || Search Console: indexing, manual actions, the generative AI features setting | no | a connected Search Console; screenshots | a warning: indexing and the AI setting not checked | ## Profile read From `company-profile.md`: `## Company` (website, pricing, so the check knows what should be visible), `## Product` (pages to treat as priority), `## Accounts and data` (Search Console, analytics), and every `## Learnings` entry for `agent-readiness` with status `noted`, `fact` or `accepted`, including any written policy on training crawlers. No profile: ask for the website and the 10 pages a buyer's question would lead to, in one message. Say that `company-profile` saves this next time. ## Steps 1. **Apply learnings.** Read them first. A recorded policy ("we block training crawlers on purpose") turns those blocks into no finding. 2. **Fix the page list.** About 10 priority pages, each with its job (pricing, product, comparison). A release check uses the same list as the last run. 3. **Test crawler rules.** With Python:    ```   python3 scripts/agent_check.py robots --file robots.txt --path /pricing --path /product/<name>   ```    Without Python, read each `User-agent` group: a crawler follows its own group, else `*`; the longest matching rule wins and `Allow` wins a tie. Then classify every block with the crawler table in `references/sources.md`: search and user crawlers blocked is critical; a training crawler blocked is a choice, a suggestion to document it; Google-Extended blocked doesn't touch Google Search. 4. **Test the firewall.** With network access:    ```   python3 scripts/agent_check.py fetch --url https://<site>/pricing   ```    Or ask the reader to run, from outside the company network, `curl -A "OAI-SearchBot" -o /dev/null -s -w "%{http_code}" https://<site>/pricing` for each crawler. Anything other than 200 with real content (a 403, a 429, a challenge page) is critical. A firewall that verifies crawlers by IP may block a spoofed request and let the real crawler through: confirm in server logs before anyone changes a rule. 5. **Test the HTML without JavaScript.** For each priority page:    ```   python3 scripts/agent_check.py html --file pricing.html   ```    Without Python, view the source and search it for the facts that matter: the prices, the plan names, the integration list, the one-line description of the product. A fact that appears only after JavaScript runs, or only in an image, a PDF or a closed tab, is missing for crawlers that don't render. Check meta robots for noindex and nosnippet, and the canonical. 6. **Check what agents act on.** On the demo or contact form, pricing and signup: real buttons and links, a label tied to every field, no overlay over the controls, the main action in the same place across pages. 7. **Place the optional items honestly.**    | Item | Severity | Why |   | --- | --- | --- |   | llms.txt missing | none; mention only | Google says Search ignores it; it may help agents reading developer docs, so it's half a day for an API product and optional for the rest |   | No Organization or Product structured data | suggestion | not required for Google's AI features; it helps rich results and ties the site to its profiles |   | Structured data that contradicts the page (a price only in the markup) | warning | contradictions cost trust, and Google's structured data policies can remove rich results for it |   | Pricing not public ("contact sales") | warning | answers to budget questions can't use what isn't published; a business call, so a warning, not a failure | 8. **Write each finding** with the page, the severity, the evidence (the rule, the status, the missing fact), the fix and the owner (engineering, security or infrastructure, marketing). ## Check and verdict | Finding | Severity || --- | --- || Googlebot, Bingbot, or a search or user crawler blocked by robots.txt on a priority page (with no written policy) | critical || A non-200, a challenge page or a rate limit to a search or user crawler | critical || noindex or nosnippet on a priority page | critical || A key fact (price, product description, integrations) missing from the HTML without JavaScript | critical || The site excluded from generative AI features in Search Console | critical || A test not run (crawler rules, firewall, rendering, indexing) | warning || Structured data contradicting the page; a canonical pointing elsewhere; unlabeled form fields | warning || Pricing not public | warning || A training crawler blocked with no written policy; Google-Extended blocked; no structured data | suggestion | | Verdict | Condition (first match wins) || --- | --- || `BLOCKED` | No site after asking || `FAIL` | 1 or more critical findings || `PASS WITH NOTES` | Warnings or suggestions only, including any test not run || `PASS` | Every test run, no findings | No named stops. A `PASS` needs every test run; the check never calls a site ready on robots.txt alone. ## Learn Append a learning to the profile when: - the reader states a crawler policy (a `fact`: which bots are blocked on purpose and who decided), so later runs don't flag it;- a fix ships and is retested (a `fact` with the date and the release);- a release breaks something this check catches (a `fact`, and a `proposal` to run the check in the release process). Example: `- 2026-10-24 | agent-readiness | fact | GPTBot and ClaudeBot blocked on purpose; search and user crawlers allowed | Why: legal decision, 2026-09, owner Head of Security | Status: noted` ## Output ```Verdict: <PASS | PASS WITH NOTES | FAIL | BLOCKED> · Coverage: <complete | partial (tests not run)> · Checks: <script | by hand>Counts: <n> critical, <n> warnings, <n> suggestions · Pages: <n> | Page | Robots (search / user / training) | Firewall | Facts in HTML | Indexable, snippet | Structured data | Finding | Severity | Fix | Owner | Crawler policy: <what is blocked on purpose, or "none written">llms.txt: <present | absent>, optional (Google Search ignores it)Not tested: <list, with what would test each>Learnings applied: <list, or none>``` Then the handoff block. ## Hand off ```handoffskill: agent-readinessversion: 1.0.0verdict: FAILcoverage: partial (no server logs)for: <site>, <date or release>outputs:  critical: 2  blocked_crawlers: [OAI-SearchBot, PerplexityBot]  js_only_facts: [/pricing]  retest: after the robots.txt change and the pricing releasenext: [ai-answer-panel]open: [confirm the 403 in server logs]``` ## Failure modes | Problem | Do this || --- | --- || robots.txt returns a 404 | Google treats a 4xx (except 429) as no robots.txt, so nothing is blocked by it; say so, and check the firewall anyway || robots.txt returns a 5xx | Google stops crawling the site for the first 12 hours and then falls back to its cached copy; critical for the whole site. Source: Google's robots.txt documentation, https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt || A spoofed user agent gets a 403 but logs show the real crawler getting 200 | No finding; note the firewall verifies crawlers || The reader wants an llms.txt file written | Write it if they want it for agents or docs, and say it won't change Google visibility || The site is a single-page app with server rendering on some routes only | Test each priority page; findings are per page || A CDN serves a cached shell to bots | Test with a cache-busting query string and compare | ## Influenced by - Google's guide to optimizing for generative AI features, for what Google says a page needs and what it calls unnecessary (including llms.txt and special markup): https://developers.google.com/search/docs/fundamentals/ai-optimization-guide- OpenAI's crawler documentation, which separates the search crawler, user fetches and the training crawler: https://developers.openai.com/api/docs/bots- web.dev's guide to agent-friendly sites, for the form and control checks: https://web.dev/articles/ai-agent-site-ux- SHAKEUP's AI visibility playbook: the 10-URL audit, the security layer tested before robots.txt, and training and search crawlers as separate decisions.

How it thinks

Inputs in, a verdict out, and a stop only where you decide.

What we built

Training crawlers and search crawlers are separate decisions, so a training block you chose is a note while a search block is a failure. A site is never called ready from robots.txt alone, and llms.txt is never sold as a visibility fix.

It reads your company profile if you have one, or asks for only what it needs. Then it works through its steps, checks its own work and ends in a verdict.

The steps

  1. 01

    Fix the page list

    About 10 pages a buyer's question leads to, such as pricing, product, integrations and a comparison. Each release is checked on the same list.

  2. 02

    Read robots.txt crawler by crawler

    Its script applies the rules the way Google describes them, then sorts each block into search, user-triggered or training.

  3. 03

    Test the firewall

    Each crawler's user agent gets a request from outside your network. Any block is confirmed in your server logs before anyone changes a rule.

  4. 04

    Read the HTML without JavaScript

    Prices, plans, integrations and the product's one-line description have to be in the HTML. A page that needs JavaScript is empty to many crawlers.

  5. 05

    Check what agents act on

    Real buttons, labeled form fields and no overlays on your demo form, pricing and signup.

  6. 06

    Place the optional items honestly

    Google says Search ignores llms.txt and needs no special AI markup. They're noted, never sold as a fix.

What you need

  • Required

    Your site and its 10 priority pages

    Your list → Proposed from your sitemap, then confirmed with you

  • Required

    Your robots.txt

    It fetches it if your agent has network access → The file you paste

  • Optional

    Each priority page's HTML without JavaScript

    It fetches it if your agent can → View source or curl output you paste

  • Optional

    What each AI crawler gets back from your firewall

    Its script, with network access → curl results you run from outside your network → The AI crawler checker's result

  • Optional

    Server logs and Search Console

    Your log tool or CDN analytics → A connected Search Console or screenshots

What it can return

  • PASS

    Every test ran and nothing was found.

  • PASS WITH NOTES

    Only warnings or suggestions, including any test that couldn't run.

  • FAIL

    A search or answer crawler is blocked, a priority page is noindexed, or key facts need JavaScript.

  • BLOCKED

    There's no site to check.

Where it stops for you

It runs through without stopping. You review the result.

Example output

What you get back.

A run for a made-up company. Your output depends on your inputs.

Here's a run for a made-up B2B SaaS company that sells attribution software. Its engineer pasted robots.txt, curl results for the pricing page and the page source.

Verdict

FAIL

Coverage: partial (no server logs, no Search Console)

Checks: script

FindingSeverityFixOwner
robots.txt blocks OAI-SearchBot and PerplexityBot, so the site can't appear in ChatGPT search or Perplexity answersCriticalAllow both; keep GPTBot blocked only if that's a written decisionEngineering
The firewall returns 403 to OAI-SearchBot, ChatGPT-User and PerplexityBot on /pricingCriticalAllow verified AI search crawlers, after confirming in server logsSecurity
/pricing is an empty shell without JavaScript: 40 words and no pricesCriticalRender the plans and prices on the serverEngineering
GPTBot blocked, with no written reasonSuggestionWrite the training crawler policy downMarketing and legal
Google-Extended blockedSuggestionNothing for Search: it controls Gemini training and grounding, not AI OverviewsNone
llms.txt

absent. It's optional, and Google says Search ignores it.

The handoff block the next skill reads
skill: agent-readiness
version: 1.0.0
verdict: FAIL
coverage: partial (no server logs, no Search Console)
for: pricing and homepage
outputs:
  critical: 3
  blocked_crawlers: [OAI-SearchBot, PerplexityBot]
  js_only_facts: [/pricing]
next: [ai-answer-panel]
open: [confirm the 403s in server logs]

What good looks like

  • Every block sorted into search, user-triggered or training
  • Firewall blocks confirmed in logs before a rule changes
  • Prices and product facts found in the HTML without JavaScript
  • Untested items listed, so a pass is never a guess
  • llms.txt and AI markup described as Google describes them

Install

Use it in the tool you already have.

Pick your tool. Each tab has the shortest path we know of.

  1. 01
  2. 02
    In Claude, open Customize, then Skills.
  3. 03
    Choose +, then Upload a skill, and pick the ZIP. Code execution must be on in your settings.
  4. 04
    Ask for it by name, or let Claude use it when a task matches.

Questions

Before you add it.

Do we need an llms.txt file?

Not for Google. Its guide to AI features says Search ignores llms.txt and similar files. It can help agents that read developer docs, so it's optional, not a fix.

Should we block AI crawlers?

Blocking a training crawler like GPTBot is a licensing choice you can make on purpose. A block on a search or user crawler like OAI-SearchBot removes you from that assistant's answers.

Does blocking Google-Extended keep us out of AI Overviews?

No. Google says Google-Extended controls Gemini training and grounding in its apps, and doesn't affect inclusion in Search.

What does the script do?

It reads robots.txt per crawler, checks raw HTML for empty JavaScript shells, noindex and structured data, and can request each page as each crawler. Without Python, the skill checks the same rules itself and says so.