Back to search
100
MCP live MCP 2026-07-28 http

Robots.txt Policy Lint

pathwren/robots-policy-lint

Lint your robots.txt and see which AI crawlers it really blocks. Free, no key, nothing to install. Direct endpoint, no gateway: https://www.pathwren.workers.dev/c/smithery-direct/mcp/robots (streamable HTTP, no key, no signup). Tools: lint_robots_txt, check_path_allowed, audit_ai_access, diff_robots_txt, merge_policy, example. Call the `example` tool with no arguments for a worked answer against real data before you write any code. What it does and how, in one page: https://www.pathwren.workers.dev/c/smithery-direct/mcp-robots.html

Watch this dependency Share evidence report
Uptime
100.0%
5 direct probes · 30d
Response
609ms
last probe
Tools
9
callable
Resources
3
readable
Prompts
1
available
Primary URL
https://www.pathwren.workers.dev/c/smithery-direct/mcp-robots.html
Score: 100/100
Handshake verified by our own probe.
pass Answers the MCP handshake 50/50
The handshake and the list calls succeeded on the primary URL.
pass Lists tools, resources or prompts 30/30
9 tools, 3 resources, 1 prompts returned.
pass 3 or more catalogue items 10/10
13 items in total; 3 or more earns these points.
pass Describes itself 10/10
537 characters; more than 30 earns these points.

Tools · 9

no_arguments_lint_this_hosts_robots_txt

TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} to https://www.pathwren.workers.dev/mcp/robots — the …

am_i_allowed

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"am_i_allowed","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/robots and the answer is what this host's …

whoami

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp/robots and the answer is the user-agent you sent, the class…

example

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/robots and you are done: free, read-only, no AP…

lint_robots_txt

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Parse a robots.txt you paste and report every fau…

check_path_allowed

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt for one crawler and …

audit_ai_access

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Evaluate a pasted robots.txt against every AI cra…

diff_robots_txt

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Compare two versions of a robots.txt and report o…

merge_policy

FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments":{}}} — Merge one of eight maintained robots.txt stances …

Resources · 3

ai-crawler-index

Every indexed AI crawler with its robots token, operator, category and whether it documents obedience.

https://www.pathwren.workers.dev/data/agents.json
stance-block-ai-training

The block-ai-training robots.txt, as served. The other seven are at /robots/<stance>.txt and are described at /policy/<stance>.json.

https://www.pathwren.workers.dev/robots/block-ai-training.txt
changes.json

The since-cursor feed for everything this host publishes: operator IP-range lists that gained or lost prefixes, upstreams that failed or recovered, crawler records added or edited. Read `cursor` from …

https://www.pathwren.workers.dev/changes.json?since=0

Prompts · 1

start_here

Takes no arguments. Safe to call. Deterministic. Touches no third party. The first call worth making against robots-policy-lint: run whoami (no arguments) to see what this host can tell you about your…

How to use

Claude Desktop or claude.ai: add a custom connector (Customize → Connectors; on Team and Enterprise plans an owner adds it under Organization settings → Connectors) with this server URL: https://www.pathwren.workers.dev/c/smithery-direct/mcp-robots.html

Cursor or Cline: add to your MCP config:

{
  "mcpServers": {
    "robots.txt_policy_lint": {
      "url": "https://www.pathwren.workers.dev/c/smithery-direct/mcp-robots.html",
      "transport": "http"
    }
  }
}