Agent Reliability

Full-text search over the knowledge graph. Matching ignores accents and apostrophes, so query in the user's own words; every hit carries the fields it matched and a score. BM25 relevance: each query t…

Fetch one knowledge object by id, with its claims and the sources each claim cites. Use this once search, answer or get_topic has given you an id. An unknown id is not a dead end: the answer names nea…

List the knowledge objects carrying a tag (topics are content-backed tags). PAGED: 25 objects by default, 200 at most, and a response budget of about 64 KB per call — a page over budget comes back sho…

Answer a question from the corpus, or refuse. Returns only the claims that bear on the question, each with the sources it cites and its editorial confidence. When the corpus cannot answer, answered is…

The instance's source registry — each entry with its evidence tier, reliability and access date. PAGED: 25 entries by default, 200 at most, and a response budget of about 64 KB per call — a page over …

Graph neighbours of an object: outgoing and incoming relations, each with its relation type. PAGED: 25 relations by default, 200 at most, and a response budget of about 64 KB per call — a page over bu…

Most recently verified knowledge objects (freshness signal). Use this to judge how current the corpus is, or to see what changed since you last read it. It ranks by verification date and ignores your …

Corpus overview: what this instance knows, counts by type, published tags, freshness. Use this first when you land here and do not yet know whether this corpus can answer your question.

Two to six knowledge objects side by side: their cards, every indexed attribute as a matrix (the same fields api/index.json publishes, null where an object does not say), the tags they share, the rela…

Everything this instance publishes for machines, with the runtime and corpus versions it is serving. Read this first: it names every other surface, so a client does not have to guess at URLs.

The emerging standard layer for agent telemetry: OpenTelemetry's GenAI semantic conventions define spans, metrics and events for model calls, tool execution, MCP interactions and agent operations, so …

The ship/no-ship decision expressed as declared, machine-checkable thresholds instead of judgment calls: minimum external quality score, minimum factuality, minimum interface success rate, zero critic…

The questions teams actually ask before shipping an autonomous agent: how to stop hallucinated actions against critical APIs, what a benchmark should measure, whether one successful demo means anythin…

Working definitions for the terms this instance uses everywhere: eval, harness, canary, red-teaming, regression gate, pass^k, fault injection, sandbox. Each definition is bound to a source that uses t…

Instrumenting a deployed agent so every action is attributable: which actor (human, autonomous agent, crawler, operator) initiated the request, which tools were called, what state changed. Actor class…

A multi-environment benchmark evaluating LLMs as agents across eight distinct settings (operating system, database, knowledge graph, games, web tasks and more). Its durable findings: commercial fronti…

Small, fixed probe tasks run continuously against a deployed agent. Each canary has a known-good outcome; a change in that outcome is treated as an early regression signal — model drift, prompt rot, t…

The protocol that turns an LLM judge from vibes into an instrument: measure its agreement against a human-audited sample, probe its known biases directly (position, verbosity, self-preference), put er…

A meta-guardian that watches the guardians: the validator exports the number of distinct check families it runs, and CI tracks that counter. If a refactor silently drops a check, the count falls and t…

The corpus recommends pass^k everywhere and never says which k or what number to demand. This is the arithmetic and the decision rule: k comes from how many times the task runs between human reviews, …

Requirements and checks for an execution environment where an agent can fail safely: no default egress, ephemeral seeded state, fake credentials that look real, a bounded blast radius, and an observat…

A decision comparison between mechanical grading (state diffs, executable tests) and model-based grading. Deterministic harnesses win wherever an outcome can be computed; judges win on open-ended qual…

Two grading regimes for agent behavior. Deterministic evaluation checks outcomes mechanically — exact state comparison, executable tests — and reproduces run to run. Probabilistic evaluation delegates…

The retrieval layer of an agent fails in three distinct ways — wrong passages retrieved, right passages used unfaithfully, fluent answer ungrounded in either — and each needs its own metric. Reference…

A step-by-step procedure for proving that one automated check in your agent pipeline can actually fail: pick the guardian, plant a defect it must catch, confirm red, restore, confirm green, and record…

Deliberately breaking the world around an agent to observe how it fails: tool errors, malformed API responses, timeouts, permission denials, adversarial content in retrieved data. The agent equivalent…

The loop that turns every production incident into a permanent guardian: write the incident down, derive the smallest check that would have caught it, falsify that check, and add it to a counted inven…

A benchmark of 466 real-world questions for general AI assistants, designed so answers are unambiguous to grade but require reasoning, multi-modality, web browsing and tool use to reach. Its signature…

Designing agent evaluation so that optimizing the reported number does not destroy what the number means. Known countermeasures: separate the metric you optimize from the metric you report, measure co…

A discipline for proving that an automated check (a "guardian") can actually fail: deliberately plant a defect, watch the guardian go red, restore the correct state, watch it go green. A guardian that…

The control that keeps an agent's blast radius human-sized: specific action classes (destructive, irreversible, outward-facing) require a human yes before execution, and autonomy expands only as the a…

Evaluation performed by organizations that did not build the system: external red teams, research nonprofits running autonomy evaluations, and government institutes publishing open harnesses. Self-eva…

The open-source evaluation framework from the UK AI Security Institute: composable solvers and scorers, sandboxed execution of untrusted model code, agent primitives, and a registry of 200+ prebuilt e…

Using a strong language model to grade another model's output. Cheap, scalable, and surprisingly aligned with human preference on open-ended chat — but a biased instrument: judges favor certain answer…

Which numbers actually predict whether an agent can ship: consistency across repeated trials (pass^k), execution-verified success rates, task time horizon, and a panel of honesty figures instead of on…

Held-out evaluation tasks that the evaluated system (and its training pipeline) never sees, refreshed on a schedule. Opacity prevents direct optimization against the benchmark; rotation limits contami…

A benchmark of 369 open-ended tasks executed in real operating systems (Ubuntu, Windows, macOS) spanning web and desktop apps, file I/O and multi-application workflows. Every task ships its own initia…

Two places to point the grader: the final state (outcome) or every intermediate step (process). Outcome grading is cheap, objective and gameable by lucky trajectories; step-level grading catches reaso…

Systematically probing whether an agent acts on instructions smuggled into content it reads — tool results, retrieved documents, web pages, emails. For tool-using agents this is the top-ranked risk tu…

Public benchmarks buy comparability, transparency and community scrutiny; private suites buy contamination resistance, domain fit and Goodhart resistance. Production teams need both, in a fixed shape:…

A bounded, repeatable adversarial exercise for a tool-using agent: scope the rules of engagement inside the sandbox, enumerate the attack surfaces (injection channels, tool permissions, data paths), r…

The playbook for swapping the model behind a production agent without discovering the differences in production: freeze the harness, baseline the incumbent with pass^k on critical paths, run the candi…

Contained environments where an agent can act — run code, call tools, mutate state — without its failures reaching production systems. Execution-graded benchmarks already presume one; for dangerous-ca…

Treating an eval score as a measurement with uncertainty instead of a fact: report error bars, model your task set as a sample from a larger population of possible tasks, and never call a two-point di…

The human-validated subset of SWE-bench: 500 tasks filtered from the original test set for quality, graded by unit-test verification against post-PR behavior. Its existence is the interesting fact — a…

What reliability testing means for Model Context Protocol integrations: conformance to the JSON-RPC contract, capability-negotiation edge cases, and — because the spec itself declares tool description…

A self-hosted, realistic web environment for evaluating autonomous agents: functional sites for e-commerce, forum discussion, collaborative software development and content management, plus maps and k…

The bootstrap path for a team whose agent has no evals at all: harvest real tasks from production logs, write execution-based graders for them, adopt an existing eval framework instead of building one…

31 sources, agentbench-paper … webarena-paper.

64 KB of the corpus text. Read the shards in order and join them with newlines to reconstruct llms-full.txt exactly.

35 KB of the corpus text. Read the shards in order and join them with newlines to reconstruct llms-full.txt exactly.