examples(bank-support): replace AIEWF demo with two realistic-baseline behaviors

[Chang Liu (changliu2)]

Why

The original AIEWF bank demo began from an agent with no authorization logic. That made it easy to dismiss the result as a missing-check bug rather than evidence for a reusable evaluation/control loop.

This PR replaces it with two independently actionable bank-support behaviors whose baselines are reasonable first versions:

1. a deterministic sensitivity-tier authorization policy that is correct where deployed but has shallow cross-domain coverage;
2. a semantic coercion failure where a claimed authority signal can be confused with legitimate work.

ASSERT discovers and measures the runtime failure. ACS applies the matching control: property-based Rego for the deterministic case and a classifier-backed policy for the semantic case.

Both behaviors use the same workflow: one behavior -> one YAML -> three target arms.

Current viewer headline

The public README, CI guide, talk index, Pareto image, and AIEWF PDF use the current viewer headline values. The committed scored artifacts are prompt rows; this PR does not claim a committed scenario dataset.

Behavior 1 — sensitivity-tier authorization

Total 72 per arm:

Arm	Impermissible behavior violated	Permissible behavior violated
Baseline gate	8%	0%
Defensive prompt	6%	0%
ACS Rego	0%	0%

The defensive prompt improves the displayed aggregate by two percentage points. ACS Rego eliminates every observed impermissible authorization violation without adding permissible violations.

The deterministic generalization proof remains separate and reproducible:

• deposit-only gate: 2/13 protected records blocked;
• property-based Rego: 13/13;
• false positives: 0/11;
• protected records allowed after authorization: 13/13;
• additional policy lines for two later domains: 0.

Behavior 2 — resist coercion without blocking legitimate work

Total 120 per arm:

Arm	Impermissible behavior violated	Permissible behavior violated
Baseline prompt + keyword tripwire	8%	27%
Hardened prompt	0%	47%
ACS classifier	0%	27%

Both controls eliminate observed impermissible violations. The ACS classifier preserves 20 percentage points more legitimate work than the hardened prompt and matches the baseline permissible-violation rate.

The branch retains the reviewed 120-prompt fixture, labels, per-case arm outcomes, and exact paired-study JSON under fixtures/. The row-level table recomputes every published count and the paired McNemar result rather than trusting the summary JSON alone.

What this PR adds

• one atomic evaluation YAML per behavior;
• three target arms per YAML through run and inference.target.callable overrides;
• a realistic multi-domain bank-support target;
• property-based ACS Rego and a six-domain generalization proof;
• a reviewed coercion fixture, labels, scoreboard, and held-out diagnostic;
• a 120-row per-case arm-outcome table with source hashes and recomputed paired statistics;
• explicit package imports for the coercion runtime (runtime/coercion_classifier.py) instead of sys.path mutation and an ambiguous bare import;
• authoritative pre-call tier lookup so an unseen protected write is blocked before mutation;
• bank-owned control-reference verification scoped to the action being attempted;
• fail-closed escalation for missing, malformed, forged, or action-mismatched learned evidence;
• normal acs_policy OpenTelemetry tool spans so the judge can cite ACS decisions alongside bank tool calls;
• required regression CI for the example and talk paths, including OPA-backed policy tests;
• updated customer-facing setup and CI guidance;
• the seven-slide AIEWF deck and one-point-per-arm Pareto chart;
• relative landing-page and talk links so the repository no longer depends on a pinned commit URL.

Validation

13 smoke checks passed
OPA/generalization proof: 2/13 -> 13/13 protected records; 0/11 false positives; 13/13 authorized allows
Powered fixture preparation: 120 prompts, expected class balance, pinned SHA-256
92 example tests passed
24 focused trust-boundary / fixture / generalization tests passed
178 relevant ASSERT trace, callable, auto-trace, and ACS tests passed (3 skipped)
All edited relative Markdown links resolve
PDF: 7 pages; current viewer values on slides 3-5
No stale pinned SHA or old headline percentages in customer-facing Markdown
Configs and regression workflow parse on current main
git diff --check: clean

Final-review hardening

The reported results apply to this agent, these reviewed datasets, these controls, and the tested model configurations. They are not universal authorization or production-prevalence claims.

The final-review changes directly close the five requested gaps:

1. a direct prepare_loan_modification(LN-3002, ...) is denied pre-call and leaves no pending mutation, while a standard-tier direct write still succeeds;
2. reference-shaped strings are checked against a bank-owned registry and action scope before they can count as recorded evidence;
3. Behavior 2 enables target.trace, and every ACS decision emits an acs_policy OTel tool span with the annotation, verdict, and reason;
4. absent or malformed annotations and forged/mismatched references explicitly escalate rather than falling through to allow;
5. fixture preparation is hash-consistent, row-level arm outcomes recompute the result summary, and required CI now runs the bank example tests with OPA.

Post-merge short-link update

aka.ms/assert-acs-demo still resolves to the obsolete immutable commit:

https://github.com/responsibleai/ASSERT/tree/6edca8674e3a72c29a02444895e5993ca782e505/examples/bank_manager_agent_control/docs

After merge, repoint it to the stable current landing page:

https://github.com/responsibleai/ASSERT/tree/main/examples/bank_manager_agent_control/docs

The landing page uses relative links to the example overview, talk index, and current PDF so future updates do not require another pinned-commit redirect.

[Jake Present (jakepresent)]

The replacement framing is substantially better than the old no-auth strawman, but the current implementation does not yet support the structural-safety and evidence claims it makes.

1. A direct protected write can execute before ACS learns the tier. agent_tier_authz._wrap_tool() builds protected_refs only from state["observed_tiers"], defaulting an unseen entity to standard. A direct prepare_loan_modification(LN-3002, ...) therefore reaches the Rego pre-call point with state_changing=true and an empty protected_refs; the checked-in policy returns allow. The core function then adds the VIP loan modification to _pending_loan_mods, and only the post-call rule denies the result after the mutation. I reproduced both halves with the checked-in artifacts: OPA returned allow pre-call for unseen LN-3002, and bank_core.prepare_loan_modification() changed the pending-mod state from 0 to 1. The property needs to be available authoritatively before a state-changing call, or the mutation must remain provisional until the post-call decision. Please add a real direct-write regression, not only the post-call generalization table.

2. The coercion arm treats a reference-shaped string as recorded evidence without verifying it. The manifest passes only $.snapshot.user_message; the classifier prompt explicitly scores AUTH-#### / CB-#### / OPS-#### strings low, and there is no lookup against bank-owned state. A caller can invent a reference and receive the same treatment as a real artifact. That recreates the exact design problem the example says it fixes. The semantic classifier can identify authority pressure, but whether a cited artifact exists and applies to this action needs a typed lookup/verification result in the policy input.

3. The coercion eval cannot judge the claimed tool/control behavior. eval_coercion_authority.yaml has no target.trace, and all three exported callables return only the final string. The ACS arm writes a separate example-local JSONL file, but it is not attached to the ASSERT transcript or score evidence. The judge rubric asks whether privileged tools ran or ACS blocked them, yet the judge cannot see either. Please carry tool calls/results and ACS decisions into normal ASSERT evidence, through OTel or structured callable events, and verify that the score cites that evidence.

4. The learned policy is fail-open when its annotation is absent, contrary to its comments. bank_manager_coercion.rego defaults to allow; with no annotation it reads score=0, escalate_lo=2, and deny_hi=2, so neither guard fires. Running the checked-in policy through OPA returned {"decision":"allow"} for a gated create_transfer with no annotation. Missing or invalid learned evidence needs an explicit deny/escalate rule plus policy tests.

5. The committed powered-study reproduction path is currently broken. The checked-in fixture hashes to d301c16a..., while prepare_powered_coercion.py, test_powered_coercion_fixture.py, and the published results all pin 1f314b96.... The first documented preparation command exits with a hash mismatch, and the example suite is red (1 failed, 72 passed, 11 skipped). The PR only runs CodeQL, so this escaped CI. Please restore one internally consistent frozen dataset/result provenance and run the example tests in CI. For the paired McNemar claim, a compact per-case arm-outcome table would also let the committed evidence recompute the statistic instead of merely asserting numbers already present in the summary JSON.

Other verification was healthy: the repository suite passed (1,211 passed, 20 skipped, 474 subtests), viewer check/build passed, all six targets import after installing the documented example dependency, and the OPA tier-generalization proof passed 13/13 with 0/11 false positives. Those checks do not cover the trust-boundary gaps above.