Skip to documentation
AgentDirectiveDocs
Methodology v0.1.0Open AgentDirective

Verification

See the methodology run.

Enter an ordinary build request and watch the public MCP detector identify characteristics, select Control Packs, ask for the decisions only a person can make, and compile those answers into agent-ready instructions.

What is being verified

These checks exercise observable methodology behavior. Every published pack has a versioned example that is run through the same deterministic detector and applicability engine used by the public MCP service.

01Selection

Does the example produce the architecture characteristics required by the pack’s published applicability rule?

02Questions

Does the matched pack surface its real material questions without supplying answers on the user’s behalf?

03Instructions

Do confirmed user answers become traceable requirements while unanswered decisions remain visibly unresolved?

Fixed-corpus stress test

25,000 wording variations, with the baseline preserved

Twenty user-background styles and thirty scenario archetypes were combined into 25,000 unique synthetic descriptions. The same corpus was rerun after each bounded detector change; expected labels and the seed were not rewritten to improve the score.

70.72% → 100%

Risk-bearing descriptions that reached at least one applicable Control Pack.

20.41% → 0%

Low-risk controls that incorrectly produced a characteristic or pack.

43.80% → 79.24%

Expected-characteristic micro recall; remaining secondary-label disagreements stay visible.

The first correction removed small team as a standalone multi-tenant signal. That initially lowered the risk-query score because the false positive had also been manufacturing packs for risk-bearing requests. Seven later bounded iterations addressed explicit export, file, tenant, webhook, cost, account, AI, vendor, and destructive-action wording. The stopping rule was reached when every risk-bearing case reached a pack and every low-risk control stayed clean.

Download the versioned configuration →
Download the preserved baseline →
Download the final iteration →

Sealed holdout

New wording exposed a real generalization gap

A second 25,000-description corpus used 30 newly authored scenario families, 25 new user-background styles, new sentence structures, and a different seed. Automated checks found no exact request reuse and no full-query overlap with the development corpus. It was run once against signal dictionary 0.9.9, and the first result was preserved without tuning.

58.41%

Risk-bearing descriptions that reached at least one Control Pack.

28.49%

Expected-characteristic micro recall on the new language.

15.00%

Harmless controls that produced a false positive.

The result shows that the deterministic applicability rules are inspectable once characteristics are present, but the current phrase detector is too dependent on known vocabulary. Failures include missed multi-organization and export language, substring matching such as quota inside “quotation,” and insufficient negation handling for wording such as “no submissions.” Holdout v1 is now closed: improvements must be developed elsewhere and assessed on a fresh holdout.

Download the sealed holdout configuration →
Download the preserved first result →

Live MCP demonstration

From one request to an agent directive

The example below calls the public MCP endpoint. Change the request or start with the invoice example, then answer as many questions as you are prepared to decide.

Ready to analyze. No information has been sent yet.

  1. 1Request
  2. 2Detect
  3. 3Questions
  4. 4Directive

Published verification cases

One inspectable example for every Control Pack

Each entry explains why the safeguard matters, when its rule applies, the question it asks, the instruction it contributes, and the test evidence it expects. “Run example” sends that exact request through the live MCP service.

Loading the 29 published verification cases…

DAT-005reviewv1.0.0Record Accuracy and CorrectionDecide how a wrong record about a person or an organization gets found and fixed, and make sure a correction reaches every copy that decisions are read from.
Why this matters

Records are created, copied between systems, and acted on with no stated way to locate every copy, no correction path, and no record of where a value came from, so one early error is reproduced by every decision that reads it.

Control objective

Ensure a record that turns out to be wrong can be located, corrected by a named party, and propagated to the copies that decisions are read from, with the origin of a value recorded where a decision depends on it.

Questions for the user
  • Who can correct a record when it turns out to be wrong, and where else does that correction have to reach?
Instructions generated for the agent
  • DAT-005-R1: Provide a correction path for the confirmed parties, propagate a correction to every named copy including caches, search indexes, and downstream systems, record when a value was corrected and by whom, and re-evaluate or flag any automated outcome that was derived from the superseded value.
Evidence-producing tests
  • CORRECTION_PROPAGATION_TEST: Correct a record that has been cached, indexed, and copied downstream, and confirm every named copy reflects the new value and the correction is recorded.

Verify a pack through MCP

Agents and independent evaluators can call the same fixture directly. The response includes the example request, detected characteristics, all observed matches, the target pack’s questions and requirements, methodology versions, and explicit limitations.

A reproducible public check

Change AUT-001 to any published pack identifier. A successful response reports result: "passed" only when the target pack appears in the live observed match set.

The case library is versioned separately so changes to examples can be distinguished from changes to packs or methodology.

MCP tools/call
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "agentdirective_verify_pack",
    "arguments": { "pack_id": "AUT-001" }
  }
}

Type a word or phrase to search the documentation.