AI Crawler Index
pathwren/ai-crawler-indexName any bot user-agent, verify a crawler IP, or generate robots.txt. Free, no key, no signup. Direct endpoint, no gateway: https://www.pathwren.workers.dev/c/smithery-direct/mcp (streamable HTTP, no key, no signup). Tools: classify_user_agent, lookup_crawler, list_crawlers, generate_robots_txt, is_verified_crawler_ip, whats_changed, example. Call the `example` tool with no arguments for a worked answer against real data before you write any code. What it does and how, in one page: https://www.pathwren.workers.dev/c/smithery-direct/mcp.html
Watch this dependency
Get before/after evidence when observed tools, schemas, authentication access, availability or the endpoint changes.
Inspect evidence and set up a watch Share public evidence report
Score: 100/100
Handshake verified by our own probe.
why this score
Tools · 9
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, n…
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you are done: free, read-only, no API key, …
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Identify the crawler behind a raw User-Agent header: operator, category, robots.tx…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The full index record for one crawler: operator, category, robots.txt token and st…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The index, filtered or whole: each row says what the crawler is and what blocking …
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — A ready-to-paste robots.txt block for one stance, with the reasoning and the crawl…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Test an IPv4 or IPv6 address against every crawler prefix the operators themselves…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Freshness and change state: when the index was built, when each operator source wa…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Only what changed since your last call: prefixes added or removed, upstreams faile…
Resources · 5
Every indexed crawler: operator, category, robots.txt token, user-agent, verification method and cost of blocking.
https://www.pathwren.workers.dev/data/agents.json
One case-insensitive alternation per set (all, AI, training) for a log classifier or a WAF rule.
https://www.pathwren.workers.dev/data/ua-regex.json
Union and per-source crawler IP prefixes, mirrored from the operators every six hours.
https://www.pathwren.workers.dev/ip-ranges/all.json
Per-source fetch state, counts, hashes and change flags for the 15 upstream IP-range endpoints.
https://www.pathwren.workers.dev/status.json
The since-cursor feed for everything this host publishes: operator IP-range lists that gained or lost prefixes, upstreams that failed or recovered, crawler records added or edited. Read `cursor` from …
https://www.pathwren.workers.dev/changes.json?since=0
Prompts · 1
Takes no arguments. Safe to call. Deterministic. Touches no third party. The first call worth making against ai-crawler-index: run whoami (no arguments) to see what this host can tell you about your o…
How to use
Add to your Claude Desktop / Cursor / Cline MCP config:
{
"mcpServers": {
"ai_crawler_index": {
"url": "https://www.pathwren.workers.dev/c/smithery-direct/mcp.html",
"transport": "http"
}
}
}