Skip to documentation
AgentDirectiveDocs
Methodology v0.1.0Open AgentDirective

Verification

See the methodology run.

Enter an ordinary build request and watch the public MCP detector identify characteristics, select Control Packs, ask for the decisions only a person can make, and compile those answers into agent-ready instructions.

What is being verified

These checks exercise observable methodology behavior. Every published pack has a versioned example that is run through the same deterministic detector and applicability engine used by the public MCP service.

01Selection

Does the example produce the architecture characteristics required by the pack’s published applicability rule?

02Questions

Does the matched pack surface its real material questions without supplying answers on the user’s behalf?

03Instructions

Do confirmed user answers become traceable requirements while unanswered decisions remain visibly unresolved?

Fixed-corpus stress test

25,000 wording variations, with the baseline preserved

Twenty user-background styles and thirty scenario archetypes were combined into 25,000 unique synthetic descriptions. The same corpus was rerun after each bounded detector change; expected labels and the seed were not rewritten to improve the score.

70.72% → 100%

Risk-bearing descriptions that reached at least one applicable Control Pack.

20.41% → 0%

Low-risk controls that incorrectly produced a characteristic or pack.

43.80% → 79.24%

Expected-characteristic micro recall; remaining secondary-label disagreements stay visible.

The first correction removed small team as a standalone multi-tenant signal. That initially lowered the risk-query score because the false positive had also been manufacturing packs for risk-bearing requests. Seven later bounded iterations addressed explicit export, file, tenant, webhook, cost, account, AI, vendor, and destructive-action wording. The stopping rule was reached when every risk-bearing case reached a pack and every low-risk control stayed clean.

Download the versioned configuration →
Download the preserved baseline →
Download the final iteration →

Sealed holdout

New wording exposed a real generalization gap

A second 25,000-description corpus used 30 newly authored scenario families, 25 new user-background styles, new sentence structures, and a different seed. Automated checks found no exact request reuse and no full-query overlap with the development corpus. It was run once against signal dictionary 0.9.9, and the first result was preserved without tuning.

58.41%

Risk-bearing descriptions that reached at least one Control Pack.

28.49%

Expected-characteristic micro recall on the new language.

15.00%

Harmless controls that produced a false positive.

The result shows that the deterministic applicability rules are inspectable once characteristics are present, but the current phrase detector is too dependent on known vocabulary. Failures include missed multi-organization and export language, substring matching such as quota inside “quotation,” and insufficient negation handling for wording such as “no submissions.” Holdout v1 is now closed: improvements must be developed elsewhere and assessed on a fresh holdout.

Download the sealed holdout configuration →
Download the preserved first result →

Live MCP demonstration

From one request to an agent directive

The example below calls the public MCP endpoint. Change the request or start with the invoice example, then answer as many questions as you are prepared to decide.

Ready to analyze. No information has been sent yet.

  1. 1Request
  2. 2Detect
  3. 3Questions
  4. 4Directive

Published verification cases

One inspectable example for every Control Pack

Each entry explains why the safeguard matters, when its rule applies, the question it asks, the instruction it contributes, and the test evidence it expects. “Run example” sends that exact request through the live MCP service.

Loading the 29 published verification cases…

REL-004reviewv1.0.0Data Durability and Tested RestoreDecide how much data loss and downtime is acceptable, then show that a restore has actually been performed rather than assumed from the existence of a backup.
Why this matters

Durability is assumed from a backup job existing. Nothing states acceptable data loss, acceptable restore time, whether every store is covered, or whether a restore has ever been performed.

Control objective

Ensure the acceptable data loss and restore time are stated, that every store the system depends on is covered, and that a restore has been performed and dated rather than assumed.

Questions for the user
  • How much data loss, in hours, is acceptable if you have to restore?
Instructions generated for the agent
  • REL-004-R1: Cover every store the system depends on at a frequency consistent with the confirmed acceptable data loss, keep copies outside the failure domain of the primary, and perform and date a restore that demonstrates the data returns usable — recording the elapsed time.
Evidence-producing tests
  • RESTORE_REHEARSAL_TEST: Restore each covered store into an isolated environment from a copy taken through the normal path, confirm the data is usable and within the confirmed acceptable loss, and record the elapsed time.

Verify a pack through MCP

Agents and independent evaluators can call the same fixture directly. The response includes the example request, detected characteristics, all observed matches, the target pack’s questions and requirements, methodology versions, and explicit limitations.

A reproducible public check

Change AUT-001 to any published pack identifier. A successful response reports result: "passed" only when the target pack appears in the live observed match set.

The case library is versioned separately so changes to examples can be distinguished from changes to packs or methodology.

MCP tools/call
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "agentdirective_verify_pack",
    "arguments": { "pack_id": "AUT-001" }
  }
}
Esc

Type a word or phrase to search the documentation.