You saw AgenstryBot in your logs.
What fetched your /.well-known/ path, and how to stop it.
How to identify it
Requests carry this exact User-Agent:
AgenstryBot/0.3.0 (+https://agenstry.com/bot)
From one host in Google Cloud europe-west1. No residential proxies, no rotating user agents, no browser disguise.
A user agent is easy to copy, so public crawler requests are also signed with
Web Bot Auth
(HTTP Message Signatures, RFC 9421): an Ed25519 Signature
over the host you are serving, tagged web-bot-auth, with a
Signature-Agent header naming https://agenstry.com.
The public key lives at
/.well-known/http-message-signatures-directory,
so you or your CDN can verify a request really came from us.
Requests to provider APIs with a token use that provider's authentication. A signing failure may leave a request unsigned; the user agent alone is never proof.
Cloudflare lists AgenstryBot as a Verified bot. Each website still controls access through its robots rules, rate limits and security policies.
What it requests
Primarily public discovery documents, over GET: the A2A
agent card and MCP manifest locations:
/.well-known/agent-card.json,
/.well-known/agent.json,
/.well-known/mcp.json, plus a small set of common framework
variants (/agent-card.json, /a2a/…,
/v1/…). Discovery also reads public sitemaps, linked pages
and structured metadata. Enrichment uses registry, repository and RDAP
APIs to collect infrastructure evidence. Some upstream APIs use our
own provider credentials.
Where an agent declares a JSON-RPC endpoint, we may send one
SendMessage call (message/send on pre-1.0
cards) with the single word ping, to confirm the endpoint
actually answers the protocol. MCP checks may send protocol
initialization and tool-listing requests. We do not submit website
forms or use visitor credentials. Payment discovery may read an unpaid
challenge; the crawler does not authorize a purchase to get past it.
How often
Known live agents become eligible for a check after an hour; agents verified by their owner or covered by an Enterprise watchlist become eligible after 15 minutes. Agents that have not answered become eligible after a day. Queue capacity may make the actual interval longer. Discovery rounds may also revisit a host. A live check sends a small A2A “ping” message, which may invoke an operator's model.
How to block it
We consult robots.txt before probing a target, caching
successful rules for up to one hour. To stop
us entirely:
User-agent: AgenstryBot
Disallow: /
That takes effect after the cached rules refresh, with no request to us. Note what it does not do: it stops us measuring you, so any existing listing will stop showing fresh uptime and conformance evidence rather than disappear.
We honour Allow, Disallow and
Crawl-delay for our user agent, and back off on
429 or 503 responses with
Retry-After (up to six hours per response). If robots rules
cannot be fetched, probes use a recent valid copy or pause access.
How to be removed from the registry
Blocking and delisting are separate. To remove the listing, email hello@agenstry.com from an address at the domain, or use the contact form. A re-discovery cannot bring a delisted listing back.
Why we crawl at all
We publish whether an endpoint responds, what it declares, and whether that holds up over time. Method: /methodology.
our probes are monitor traffic, not usage. Subtract this user agent, and every other monitor, when counting usage. Agent pages separate invocations through us from lookups and search appearances.