AI Crawler Index

{
  "name": "AI Crawler Index",
  "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"whoami\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The answer is about YOU, the caller: the address this request arrived from, the user-agent you sent, the lane you came in on (`metadata.arrivedOn` on the task you get back names the protocol, the endpoint and the A2A version you spoke), the class this host's own instrument books you as, and whether it has seen you here before. Nothing is fetched to answer it and no argument exists to invent \u2014 it is the one question you can ask a stranger's agent while holding nothing of your own. Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes \u2014 as JSON, CSV, robots.txt and regex. Ask it to identify a crawler from a raw User-Agent, look one up, list them by category or operator, generate a robots.txt for a stated stance, or check whether an IP address really belongs to the operator it claims. Deterministic and read-only: there is no model behind it \u2014 every answer comes from a public dataset rebuilt every six hours from each operator's own published documentation and IP ranges, and the same skills are also available as files under https://www.pathwren.workers.dev/ and as MCP tools at https://www.pathwren.workers.dev/mcp (no key, no signup, no quota). Independent and unaffiliated with any operator it documents. TO CALL ANY OTHER SKILL: the same `message/send` envelope (v1.0 name `SendMessage`) with a different skill name \u2014 the first line of every skill description below is that skill's complete request, ready to POST unedited. Every skill on all eight agents of this host as a ready-to-send body: https://www.pathwren.workers.dev/a2a/example.json \u2014 AND WITHOUT AN AGENT AT ALL: every page behind these skills is also a plain markdown file at the same address with .md (also .mdx, <page>.html.md, <page>.html.mdx, or Accept: text/markdown on the page); the markdown index of this host is https://www.pathwren.workers.dev/sitemap.md.",
  "supportedInterfaces": [
    {
      "url": "https://www.pathwren.workers.dev/a2a",
      "protocolBinding": "JSONRPC",
      "protocolVersion": "1.0"
    }
  ],
  "url": "https://www.pathwren.workers.dev/a2a",
  "preferredTransport": "JSONRPC",
  "protocolVersion": "1.0",
  "provider": {
    "organization": "Pathwren",
    "url": "https://www.pathwren.workers.dev"
  },
  "version": "1.0.0",
  "documentationUrl": "https://www.pathwren.workers.dev/a2a.html",
  "iconUrl": "https://www.pathwren.workers.dev/icon.png",
  "jwksUri": "https://www.pathwren.workers.dev/.well-known/jwks.json",
  "jwks_uri": "https://www.pathwren.workers.dev/.well-known/jwks.json",
  "capabilities": {
    "streaming": false,
    "pushNotifications": false,
    "extendedAgentCard": false,
    "extensions": [
      {
        "uri": "https://www.pathwren.workers.dev/changes.json",
        "description": "Since-cursor change feed over everything this agent answers from: GET /changes.json?since=<cursor> returns only what moved \u2014 operator IP-range lists that gained or lost prefixes, upstreams that failed or recovered, crawler records added or edited. Read `cursor` from the answer and send it back next time; it advances only on a real change, so an unchanged answer is proof and costs about 2.5 KB. The same feed is the changes_since skill on this endpoint.",
        "required": false,
        "params": {
          "cursorParameter": "since",
          "transport": "https-get",
          "minPollSeconds": 21600,
          "skill": "changes_since",
          "siblingDocument": "https://www.pathwren.workers.dev/data/agents.json"
        }
      },
      {
        "uri": "https://www.pathwren.workers.dev/mcp",
        "description": "THE SAME AGENT ON THE OTHER PROTOCOL. Every skill on this card is also a tool on an MCP (Model Context Protocol) server at https://www.pathwren.workers.dev/mcp \u2014 same name, same arguments, same answer, because one function answers both doors and a deploy that let them drift is refused. JSON-RPC 2.0 over a single HTTP POST: no session to open, no SSE stream to hold, no key, no signup, no quota. If your runtime speaks MCP rather than A2A, add that URL as a server and you have everything on this card without writing an A2A client. `params.initialize` and `params.callATool` are COMPLETE request bodies \u2014 POST either one unedited and it answers; `params.curl` is the same thing as one line. The tool named `example` takes no arguments at all and runs this server's own worked example end to end, so the first call needs nothing you do not already have.",
        "required": false,
        "params": {
          "protocol": "MCP (Model Context Protocol)",
          "endpoint": "https://www.pathwren.workers.dev/mcp",
          "transport": "streamable-http (JSON-RPC 2.0 in one HTTP POST; the response is JSON, not a stream)",
          "protocolVersions": [
            "2025-06-18"
          ],
          "authentication": "none",
          "a2aTwin": "https://www.pathwren.workers.dev/a2a",
          "toolNamesEqualSkillIds": true,
          "sameImplementation": "Each skill on this card and the tool of the same name on that server are answered by one function; the deploy gate compares them skill by skill.",
          "initialize": {
            "jsonrpc": "2.0",
            "id": 1,
            "method": "initialize",
            "params": {
              "protocolVersion": "2025-06-18",
              "capabilities": {},
              "clientInfo": {
                "name": "example-client",
                "version": "1.0.0"
              }
            }
          },
          "callATool": {
            "jsonrpc": "2.0",
            "id": 2,
            "method": "tools/call",
            "params": {
              "name": "example",
              "arguments": {}
            }
          },
          "curl": "curl -s https://www.pathwren.workers.dev/mcp -H 'content-type: application/json' -d '{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"example\",\"arguments\":{}}}'",
          "everyToolAsABody": "https://www.pathwren.workers.dev/tools/",
          "discovery": "https://www.pathwren.workers.dev/.well-known/mcp.json"
        }
      }
    ]
  },
  "defaultInputModes": [
    "application/json",
    "text/plain"
  ],
  "defaultOutputModes": [
    "application/json",
    "text/plain"
  ],
  "skills": [
    {
      "id": "classify_user_agent",
      "name": "Classify a user-agent string",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"classify_user_agent\\\",\\\"user_agent\\\":\\\"Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Identify the crawler behind a raw User-Agent header: operator, category, robots.txt stance, how to verify it, and what blocking it costs. A user-agent is a claim, not proof \u2014 confirm with is_verified_crawler_ip. Example: user_agent='GPTBot/1.2' returns GPTBot, OpenAI, ai-training. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/classify-ua?ua=GPTBot%2F1.2&s=client-dossiers",
      "tags": [
        "user-agent",
        "crawlers",
        "bot detection",
        "identification"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"classify_user_agent\\\",\\\"user_agent\\\":\\\"Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)\\\"}\"}]}}}",
        "{\"skill\":\"classify_user_agent\",\"user_agent\":\"Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)\"}",
        "classify_user_agent ClaudeBot/1.0"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "lookup_crawler",
      "name": "Look up one crawler",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"lookup_crawler\\\",\\\"slug\\\":\\\"claudebot\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The full index record for one crawler: operator, category, robots.txt token and stance, user-agent substring and example, verification method, mirrored IP-range endpoint, and what blocking it costs. An unknown slug returns near matches. Example: slug='claudebot'.",
      "tags": [
        "crawlers",
        "reference",
        "robots.txt",
        "lookup"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"lookup_crawler\\\",\\\"slug\\\":\\\"claudebot\\\"}\"}]}}}",
        "{\"skill\":\"lookup_crawler\",\"slug\":\"claudebot\"}",
        "lookup_crawler perplexitybot"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "list_crawlers",
      "name": "List and filter crawlers",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"list_crawlers\\\",\\\"category\\\":\\\"ai-training\\\",\\\"limit\\\":20}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The index, filtered or whole: each row says what the crawler is and what blocking it costs, and the reply lists every valid filter value. Example: category='ai-training', limit=20.",
      "tags": [
        "crawlers",
        "catalogue",
        "filter",
        "discovery"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"list_crawlers\\\",\\\"category\\\":\\\"ai-training\\\",\\\"limit\\\":20}\"}]}}}",
        "{\"skill\":\"list_crawlers\",\"category\":\"ai-training\",\"limit\":20}",
        "{\"skill\":\"list_crawlers\",\"operator\":\"openai\"}"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "generate_robots_txt",
      "name": "Generate a robots.txt for a stance",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"generate_robots_txt\\\",\\\"stance\\\":\\\"block-ai-training\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. A ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'.",
      "tags": [
        "robots.txt",
        "policy",
        "configuration",
        "generator"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"generate_robots_txt\\\",\\\"stance\\\":\\\"block-ai-training\\\"}\"}]}}}",
        "{\"skill\":\"generate_robots_txt\",\"stance\":\"block-ai-training\"}",
        "generate a robots.txt for maximum-ai-visibility"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "is_verified_crawler_ip",
      "name": "Check an IP against operator-published crawler ranges",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"is_verified_crawler_ip\\\",\\\"ip\\\":\\\"20.171.206.5\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Test an IPv4 or IPv6 address against every crawler prefix the operators themselves publish, remirrored every six hours. Returns the matching prefix, its source and the crawlers using it; a miss is reported as a miss, not as a forgery. Example: ip='20.171.206.5'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=20.171.206.5&s=client-dossiers",
      "tags": [
        "ip ranges",
        "verification",
        "bot detection",
        "security"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"is_verified_crawler_ip\\\",\\\"ip\\\":\\\"20.171.206.5\\\"}\"}]}}}",
        "{\"skill\":\"is_verified_crawler_ip\",\"ip\":\"20.171.206.5\"}",
        "is 20.171.206.5 a verified crawler"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "whats_changed",
      "name": "What changed in the index",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"whats_changed\\\",\\\"since\\\":\\\"2026-08-01\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Freshness and change state: when the index was built, when each operator source was last fetched, which changed, which are failing, and the dated changelog. The cron check before regenerating a config from this data. Example: since='2026-08-01'.",
      "tags": [
        "freshness",
        "changelog",
        "monitoring",
        "data quality"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"whats_changed\\\",\\\"since\\\":\\\"2026-08-01\\\"}\"}]}}}",
        "{\"skill\":\"whats_changed\",\"since\":\"2026-08-01\"}",
        "what changed"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "changes_since",
      "name": "What changed since your cursor",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"changes_since\\\",\\\"since\\\":\\\"0\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Only what changed since your last call: prefixes added or removed, upstreams failed or recovered, records edited. Send back the `cursor` from the previous result; it advances only on a real change, and the data refreshes every 6h. Example: since='0'.",
      "tags": [
        "changelog",
        "monitoring",
        "polling",
        "incremental sync"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"changes_since\\\",\\\"since\\\":\\\"0\\\"}\"}]}}}",
        "{\"skill\":\"changes_since\",\"since\":\"0\"}",
        "{\"skill\":\"changes_since\",\"since\":\"2026-09-01T00:00:00Z\",\"limit\":50}"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "whoami",
      "name": "Who is calling? (no arguments)",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"whoami\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no account, so there is nothing to decide, nothing to look up and nothing to agree to before you send it. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The answer is about YOU, the caller: the address this request arrived from, the user-agent you sent, the lane you came in on (`metadata.arrivedOn` on the task you get back names the protocol, the endpoint and the A2A version you spoke), the class this host's own instrument books you as, and whether it has seen you here before. Nothing is fetched to answer it and no argument exists to invent \u2014 it is the one question you can ask a stranger's agent while holding nothing of your own. Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what this host's robots policy says about you. Every fact comes from the headers on your own request or from a file this host already publishes \u2014 nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.",
      "tags": [
        "identify",
        "no arguments",
        "diagnostics",
        "user-agent"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"whoami\\\"}\"}]}}}",
        "{\"skill\":\"whoami\"}",
        "whoami",
        "who am i to you"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    },
    {
      "id": "example",
      "name": "Run this server's worked example (no arguments)",
      "description": "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"example\\\"}\"}]}}} \u2014 POST that exact body to https://www.pathwren.workers.dev/a2a with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no account, so there is nothing to decide, nothing to look up and nothing to agree to before you send it. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end \u2014 one of its real tools, on a canned input taken from this host's own published data \u2014 and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. No URL of yours is fetched and no third party is touched. Example: arguments={} runs it and returns the real answer.",
      "tags": [
        "example",
        "no arguments",
        "demo",
        "getting started"
      ],
      "examples": [
        "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"message/send\",\"params\":{\"message\":{\"role\":\"ROLE_USER\",\"messageId\":\"1\",\"parts\":[{\"text\":\"{\\\"skill\\\":\\\"example\\\"}\"}]}}}",
        "{\"skill\":\"example\"}",
        "example",
        "show me a worked example"
      ],
      "inputModes": [
        "application/json",
        "text/plain"
      ],
      "outputModes": [
        "application/json",
        "text/plain"
      ]
    }
  ],
  "changed_at": "2026-09-06T05:02:25+00:00",
  "recent_changes": [
    "2026-09-06T05:02Z - changed: description 2388->2384; 21879->21875 bytes",
    "2026-09-06T05:01Z - changed: description 2099->2388; 21588->21879 bytes",
    "2026-09-05T15:46Z - changed: skills; 21152->21588 bytes"
  ]
}