Back to search
100
MCP live MCP 2025-06-18 streamable-http

Crawl Census

io.github.taylorsmithgg/crawl-census

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

Watch this dependency Share evidence report
Uptime
100.0%
5 direct probes · 30d
Response
368ms
last probe
Tools
7
callable
Resources
0
readable
Prompts
0
available
Primary URL
https://crawlcensus.com/mcp
Score: 100/100
Handshake verified by our own probe.
pass Answers the MCP handshake 50/50
The handshake and the list calls succeeded on the primary URL.
pass Lists tools, resources or prompts 30/30
7 tools, 0 resources, 0 prompts returned.
pass 3 or more catalogue items 10/10
7 items in total; 3 or more earns these points.
pass Describes itself 10/10
83 characters; more than 30 earns these points.

Tools · 7

scan_site

Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of…

site_report

Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

census_stats

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

crawl_preflight

Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each do…

agent_profile

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.…

census_facts

Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line,…

submit_domains

Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth subm…

How to use

Claude Desktop or claude.ai: add a custom connector (Customize → Connectors; on Team and Enterprise plans an owner adds it under Organization settings → Connectors) with this server URL: https://crawlcensus.com/mcp

Cursor or Cline: add to your MCP config:

{
  "mcpServers": {
    "crawl_census": {
      "url": "https://crawlcensus.com/mcp",
      "transport": "streamable-http"
    }
  }
}