Skip to documentation
AgentDirectiveDocs
Methodology v0.1.0Open AgentDirective

Verification

See the methodology run.

Enter an ordinary build request and watch the public MCP detector identify characteristics, select Control Packs, ask for the decisions only a person can make, and compile those answers into agent-ready instructions.

What is being verified

These checks exercise observable methodology behavior. Every published pack has a versioned example that is run through the same deterministic detector and applicability engine used by the public MCP service.

01Selection

Does the example produce the architecture characteristics required by the pack’s published applicability rule?

02Questions

Does the matched pack surface its real material questions without supplying answers on the user’s behalf?

03Instructions

Do confirmed user answers become traceable requirements while unanswered decisions remain visibly unresolved?

Fixed-corpus stress test

25,000 wording variations, with the baseline preserved

Twenty user-background styles and thirty scenario archetypes were combined into 25,000 unique synthetic descriptions. The same corpus was rerun after each bounded detector change; expected labels and the seed were not rewritten to improve the score.

70.72% → 100%

Risk-bearing descriptions that reached at least one applicable Control Pack.

20.41% → 0%

Low-risk controls that incorrectly produced a characteristic or pack.

43.80% → 79.24%

Expected-characteristic micro recall; remaining secondary-label disagreements stay visible.

The first correction removed small team as a standalone multi-tenant signal. That initially lowered the risk-query score because the false positive had also been manufacturing packs for risk-bearing requests. Seven later bounded iterations addressed explicit export, file, tenant, webhook, cost, account, AI, vendor, and destructive-action wording. The stopping rule was reached when every risk-bearing case reached a pack and every low-risk control stayed clean.

Download the versioned configuration →
Download the preserved baseline →
Download the final iteration →

Sealed holdout

New wording exposed a real generalization gap

A second 25,000-description corpus used 30 newly authored scenario families, 25 new user-background styles, new sentence structures, and a different seed. Automated checks found no exact request reuse and no full-query overlap with the development corpus. It was run once against signal dictionary 0.9.9, and the first result was preserved without tuning.

58.41%

Risk-bearing descriptions that reached at least one Control Pack.

28.49%

Expected-characteristic micro recall on the new language.

15.00%

Harmless controls that produced a false positive.

The result shows that the deterministic applicability rules are inspectable once characteristics are present, but the current phrase detector is too dependent on known vocabulary. Failures include missed multi-organization and export language, substring matching such as quota inside “quotation,” and insufficient negation handling for wording such as “no submissions.” Holdout v1 is now closed: improvements must be developed elsewhere and assessed on a fresh holdout.

Download the sealed holdout configuration →
Download the preserved first result →

Live MCP demonstration

From one request to an agent directive

The example below calls the public MCP endpoint. Change the request or start with the invoice example, then answer as many questions as you are prepared to decide.

Ready to analyze. No information has been sent yet.

  1. 1Request
  2. 2Detect
  3. 3Questions
  4. 4Directive

Published verification cases

One inspectable example for every Control Pack

Each entry explains why the safeguard matters, when its rule applies, the question it asks, the instruction it contributes, and the test evidence it expects. “Run example” sends that exact request through the live MCP service.

Loading the 29 published verification cases…

AUT-004reviewv1.0.0Automated Work Completeness and AccuracyDecide what a complete and correct automated run looks like before relying on one, so a run that skipped, duplicated, or mishandled work is distinguishable from a run that did the job.
Why this matters

An automated job, workflow, or agent finishes without raising an error, and its result is accepted as complete and correct. Nothing states what completeness means for a run, nothing compares the result against a source of truth, and skipped, duplicated, or incorrect items look identical to correctly handled ones.

Control objective

Ensure every automated run states what it was expected to process, reports what it actually processed and what it did not, checks its output against a defined expectation or source of truth before the result is relied on, and surfaces discrepancies to a person instead of absorbing them.

Questions for the user
  • What has to be true for one run to count as complete and correct?
  • What should happen when a run finishes but some items could not be processed?
Instructions generated for the agent
  • AUT-004-R1: For each automated run, record the intended work set, the items processed, the items skipped or failed with a reason, and the outcome of the confirmed completeness and accuracy check. Make a re-run safe against duplicates, and do not report a run as successful when its own check did not pass.
  • AUT-004-R2: Implement the confirmed behaviour for a run that could not process every item, surface the exceptions to a person with enough detail to act on them, and never discard an unprocessed item without recording it.
Evidence-producing tests
  • RUN_COMPLETENESS_TEST: Run against a known work set with a deliberately missing, duplicated, and malformed item, and confirm the completeness and accuracy check fails rather than reporting success.
  • PARTIAL_RUN_TEST: Interrupt a run mid-way and re-run it, and confirm the confirmed partial-completion behaviour, no duplicate effects, and an exception record for every unprocessed item.

Verify a pack through MCP

Agents and independent evaluators can call the same fixture directly. The response includes the example request, detected characteristics, all observed matches, the target pack’s questions and requirements, methodology versions, and explicit limitations.

A reproducible public check

Change AUT-001 to any published pack identifier. A successful response reports result: "passed" only when the target pack appears in the live observed match set.

The case library is versioned separately so changes to examples can be distinguished from changes to packs or methodology.

MCP tools/call
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "agentdirective_verify_pack",
    "arguments": { "pack_id": "AUT-001" }
  }
}

Type a word or phrase to search the documentation.