Skip to documentation
AgentDirectiveDocs
Methodology v0.1.0Back to main site

Verification

See it run.

Enter a normal build request and watch the public service find traits, select safeguards, ask the questions only you can answer, and create agent-ready instructions.

What this checks

These checks exercise visible behavior. Every published pack has a versioned example that runs through the same detector and matching rules used by the public service.

01Selection

Does the example produce the architecture characteristics required by the pack’s published applicability rule?

02Questions

Does the matched pack surface its real material questions without supplying answers on the user’s behalf?

03Instructions

Do confirmed user answers become traceable requirements while unanswered decisions remain visibly unresolved?

Fixed-corpus stress test

25,000 wording variations, with the baseline preserved

Twenty user-background styles and thirty scenario archetypes were combined into 25,000 unique synthetic descriptions. The same corpus was rerun after each bounded detector change; expected labels and the seed were not rewritten to improve the score.

70.72% → 100%

Risk-bearing descriptions that reached at least one applicable Control Pack.

20.41% → 0%

Low-risk controls that incorrectly produced a characteristic or pack.

43.80% → 79.24%

Expected-characteristic micro recall; remaining secondary-label disagreements stay visible.

The first correction removed small team as a standalone multi-tenant signal. That initially lowered the risk-query score because the false positive had also been manufacturing packs for risk-bearing requests. Seven later bounded iterations addressed explicit export, file, tenant, webhook, cost, account, AI, vendor, and destructive-action wording. The stopping rule was reached when every risk-bearing case reached a pack and every low-risk control stayed clean.

Download the versioned configuration →
Download the preserved baseline →
Download the final iteration →

Sealed holdout

New wording exposed a real generalization gap

A second 25,000-description corpus used 30 newly authored scenario families, 25 new user-background styles, new sentence structures, and a different seed. Automated checks found no exact request reuse and no full-query overlap with the development corpus. It was run once against signal dictionary 0.9.9, and the first result was preserved without tuning.

58.41%

Risk-bearing descriptions that reached at least one Control Pack.

28.49%

Expected-characteristic micro recall on the new language.

15.00%

Harmless controls that produced a false positive.

The result shows that the deterministic applicability rules are inspectable once characteristics are present, but the current phrase detector is too dependent on known vocabulary. Failures include missed multi-organization and export language, substring matching such as quota inside “quotation,” and insufficient negation handling for wording such as “no submissions.” Holdout v1 is now closed: improvements must be developed elsewhere and assessed on a fresh holdout.

Download the sealed holdout configuration →
Download the preserved first result →

Live MCP demonstration

From one request to an agent directive

The example below calls the public MCP endpoint. Change the request or start with the invoice example, then answer as many questions as you are prepared to decide.

Ready to analyze. No information has been sent yet.

  1. 1Request
  2. 2Detect
  3. 3Questions
  4. 4Directive

Published verification cases

One inspectable example for every Control Pack

Each entry explains why the safeguard matters, when its rule applies, the question it asks, the instruction it contributes, and the test evidence it expects. “Run example” sends that exact request through the live MCP service.

Loading the 34 published verification cases…

CHG-003reviewv1.0.0Runtime Configuration and Feature SwitchesName the switches that change what production does, say who may change each one and what it currently controls, and keep a record of every change and the way back.
Why this matters

A flag, environment value or remote setting changes what the system does, bypassing the approval, rollout, health check and rollback that a deployment would have carried, and often with no record of who changed it.

Control objective

Identify the switches whose change is consequential, state who may change each and what it currently controls, record every change with the person and the previous value, and keep a stated way back.

Questions for the user
  • Which settings change what the system does in production without a deployment?
  • Who may change a consequential switch, and what happens at the moment it changes?
Instructions generated for the agent
  • CHG-003-R1: Record the switches whose change alters production behaviour, and for each one state what it currently controls, so a switch cannot be flipped on an understanding of it that is out of date.
  • CHG-003-R2: Enforce the confirmed authority for changing a consequential switch, and record each change with the acting person, the previous value, the new value and the time, without recording secrets.
Evidence-producing tests
  • SWITCH_INVENTORY_TEST: Compare the switches the running system reads against the recorded list and fail on one that is not described.
  • SWITCH_AUTHORITY_TEST: Attempt a switch change as an actor outside the confirmed authority and verify it is refused and recorded.
  • SWITCH_REVERSAL_TEST: Change a consequential switch, verify the change is recorded with its previous value, then restore that value and verify the system returns to the prior behaviour.

Verify a pack through MCP

Agents and independent evaluators can call the same fixture directly. The response includes the example request, detected characteristics, all observed matches, the target pack’s questions and requirements, methodology versions, and explicit limitations.

A reproducible public check

Change AUT-001 to any published pack identifier. A successful response reports result: "passed" only when the target pack appears in the live observed match set.

The case library is versioned separately so changes to examples can be distinguished from changes to packs or methodology.

MCP tools/call
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "agentdirective_verify_pack",
    "arguments": { "pack_id": "AUT-001" }
  }
}

Type a word or phrase to search the documentation.