Skip to documentation
AgentDirectiveDocs
Methodology v0.1.0Open AgentDirective

Verification

See the methodology run.

Enter an ordinary build request and watch the public MCP detector identify characteristics, select Control Packs, ask for the decisions only a person can make, and compile those answers into agent-ready instructions.

What is being verified

These checks exercise observable methodology behavior. Every published pack has a versioned example that is run through the same deterministic detector and applicability engine used by the public MCP service.

01Selection

Does the example produce the architecture characteristics required by the pack’s published applicability rule?

02Questions

Does the matched pack surface its real material questions without supplying answers on the user’s behalf?

03Instructions

Do confirmed user answers become traceable requirements while unanswered decisions remain visibly unresolved?

Fixed-corpus stress test

25,000 wording variations, with the baseline preserved

Twenty user-background styles and thirty scenario archetypes were combined into 25,000 unique synthetic descriptions. The same corpus was rerun after each bounded detector change; expected labels and the seed were not rewritten to improve the score.

70.72% → 100%

Risk-bearing descriptions that reached at least one applicable Control Pack.

20.41% → 0%

Low-risk controls that incorrectly produced a characteristic or pack.

43.80% → 79.24%

Expected-characteristic micro recall; remaining secondary-label disagreements stay visible.

The first correction removed small team as a standalone multi-tenant signal. That initially lowered the risk-query score because the false positive had also been manufacturing packs for risk-bearing requests. Seven later bounded iterations addressed explicit export, file, tenant, webhook, cost, account, AI, vendor, and destructive-action wording. The stopping rule was reached when every risk-bearing case reached a pack and every low-risk control stayed clean.

Download the versioned configuration →
Download the preserved baseline →
Download the final iteration →

Sealed holdout

New wording exposed a real generalization gap

A second 25,000-description corpus used 30 newly authored scenario families, 25 new user-background styles, new sentence structures, and a different seed. Automated checks found no exact request reuse and no full-query overlap with the development corpus. It was run once against signal dictionary 0.9.9, and the first result was preserved without tuning.

58.41%

Risk-bearing descriptions that reached at least one Control Pack.

28.49%

Expected-characteristic micro recall on the new language.

15.00%

Harmless controls that produced a false positive.

The result shows that the deterministic applicability rules are inspectable once characteristics are present, but the current phrase detector is too dependent on known vocabulary. Failures include missed multi-organization and export language, substring matching such as quota inside “quotation,” and insufficient negation handling for wording such as “no submissions.” Holdout v1 is now closed: improvements must be developed elsewhere and assessed on a fresh holdout.

Download the sealed holdout configuration →
Download the preserved first result →

Live MCP demonstration

From one request to an agent directive

The example below calls the public MCP endpoint. Change the request or start with the invoice example, then answer as many questions as you are prepared to decide.

Ready to analyze. No information has been sent yet.

  1. 1Request
  2. 2Detect
  3. 3Questions
  4. 4Directive

Published verification cases

One inspectable example for every Control Pack

Each entry explains why the safeguard matters, when its rule applies, the question it asks, the instruction it contributes, and the test evidence it expects. “Run example” sends that exact request through the live MCP service.

Loading the 29 published verification cases…

IDN-004reviewv1.0.0Access Rights and Authorization BoundaryState who may perform which operations, enforce it where the action happens rather than only in the interface, remove access deliberately rather than by habit, and record every change to who holds what.
Why this matters

Roles are implied by the interface rather than stated and checked at the point of action. Access is granted on request and never removed, temporary access has no expiry, privileged operations need no second person, and changes to who holds what access are not recorded — so nobody can say what the effective permission set currently is.

Control objective

Ensure the operations that carry authority have a stated set of who may perform them, that the check happens where the action is executed, that grants can be scoped and expired, and that every change to access is recorded.

Questions for the user
  • Which roles may perform which privileged operations, and which are they explicitly not allowed to perform?
  • How does access end when it is no longer needed?
Instructions generated for the agent
  • IDN-004-R1: Enforce the confirmed role-to-operation mapping at the point the operation executes, not only in the user interface or client. Deny by default for operations not granted, and do not rely on an unguessable route, a hidden control, or client-side state as the authorization boundary.
  • IDN-004-R2: Implement the confirmed way access ends, record every grant, change, and removal with who made it and when, and make the current holders of each privileged operation answerable from that record.
Evidence-producing tests
  • AUTHORIZATION_BOUNDARY_TEST: Attempt each privileged operation as a role that is not granted it, including by calling the underlying route directly rather than through the interface, and confirm denial and a recorded decision.
  • ACCESS_LIFECYCLE_TEST: Grant temporary access, trigger the confirmed ending condition, and confirm the access no longer works and that the grant, change, and removal are each recorded.

Verify a pack through MCP

Agents and independent evaluators can call the same fixture directly. The response includes the example request, detected characteristics, all observed matches, the target pack’s questions and requirements, methodology versions, and explicit limitations.

A reproducible public check

Change AUT-001 to any published pack identifier. A successful response reports result: "passed" only when the target pack appears in the live observed match set.

The case library is versioned separately so changes to examples can be distinguished from changes to packs or methodology.

MCP tools/call
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "agentdirective_verify_pack",
    "arguments": { "pack_id": "AUT-001" }
  }
}
Esc

Type a word or phrase to search the documentation.