Crawler Log Triage
pathwren/crawler-log-triagePaste access-log lines: every bot named, fakes flagged, robots.txt and WAF rules. Free, no key. Direct endpoint, no gateway: https://www.pathwren.workers.dev/c/smithery-direct/mcp/triage (streamable HTTP, no key, no signup). Tools: triage_log, find_impersonators, summarize_by_operator, robots_from_log, waf_ruleset_from_log, example. Call the `example` tool with no arguments for a worked answer against real data before you write any code. What it does and how, in one page: https://www.pathwren.workers.dev/c/smithery-direct/mcp-triage.html
Score: 100/100
Handshake verified by our own probe.
Tools · 9
TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} to https://www.pathwren.workers.dev/mcp/triage…
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"triage_my_request","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and the answer is your own req…
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp/triage and the answer is the user-agent you sent, the class…
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and you are done: free, read-only, no AP…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Paste log lines — combined, JSON, `uniq -c…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Only the lines claiming a crawler whose op…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Aggregate only: who crawled you, how many …
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — A robots.txt naming only the crawlers in y…
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — nginx, Caddy, Cloudflare, HAProxy or Apach…
Resources · 4
150 crawlers: operator, category, robots token, UA substring, verification method, cost of blocking. CC0.
https://www.pathwren.workers.dev/data/agents.json
1997 IPv4 and 1062 IPv6 prefixes from 15 operator sources, each with its source URL and fetch time.
https://www.pathwren.workers.dev/ip-ranges/all.json
Escaped case-insensitive alternations for grepping a log yourself, whole and per category.
https://www.pathwren.workers.dev/data/ua-regex.json
The since-cursor feed for everything this host publishes: operator IP-range lists that gained or lost prefixes, upstreams that failed or recovered, crawler records added or edited. Read `cursor` from …
https://www.pathwren.workers.dev/changes.json?since=0
Prompts · 1
Takes no arguments. Safe to call. Deterministic. Touches no third party. The first call worth making against crawler-log-triage: run whoami (no arguments) to see what this host can tell you about your…
How to use
Claude Desktop or claude.ai: add a custom connector (Customize → Connectors; on Team and Enterprise plans an owner adds it under Organization settings → Connectors) with this server URL: https://www.pathwren.workers.dev/c/smithery-direct/mcp-triage.html
Cursor or Cline: add to your MCP config:
{
"mcpServers": {
"crawler_log_triage": {
"url": "https://www.pathwren.workers.dev/c/smithery-direct/mcp-triage.html",
"transport": "http"
}
}
}