feat(story contracts): declare and derive ## Entry Point / ## Seams - #2395
feat(story contracts): declare and derive ## Entry Point / ## Seams#2395agarwal-ishaan wants to merge 8 commits into
Conversation
The contract template offered no ## Entry Point, so essentially every contract landed on the traceability path: `pdd test --from-story` routes on that heading's presence, and only 1 of 34 contracts declared it. The behavioural generator was effectively unreachable. - the contract template now declares ## Entry Point and ## Seams, with the consequence spelled out (omit it and the generated test only pins text), and ## Oracle now says its bullets must be Python expressions over `result` when an Entry Point is declared - contract generation derives the section deterministically from the story's linked prompt -- the prompt->code mapping supplies the module and the prompt's <pdd-interface> supplies the callable. No model is involved, so the result is checkable rather than guessed. Derivation is emitted ONLY when a single linked prompt declares a single callable. A partial or guessed Entry Point is worse than none: the generator routes on heading presence, so a declared-but-incomplete block turns `pdd test --from-story` into a hard error (pdd#1889 C-F7), and a wrong callable yields a confidently wrong behavioural test. When ambiguous the section is omitted and the story keeps the traceability path it has today. Existing contracts are deliberately NOT backfilled: adding a section changes the contract text, which changes story_bundle_hash(), which stales the recorded PDD_STORY_HASH in every linked test. That migration wants its own change. Refs promptdriven#2391 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gltanaka
left a comment
There was a problem hiding this comment.
Tier A, full-diff review: CHANGES REQUESTED. Issue #2391 is real and still needs a fix: the behavioral route introduced by #1867 is effectively unreachable from normal contract generation. The deterministic-binding direction here is preferable to guessing, but this head does not yet produce valid or safely executable behavioral contracts, is conflicting with current main, and has a failing required Unit Tests check.
I reproduced the prose-Oracle hard failure, required-argument invocation failure, and contract-sync downgrade through the actual generation handoff. I also audited the protected prompt transition, architecture/prompt artifacts, related #2393/#2397 work, and current CI. See the six inline findings.
Artifact gaps remain after those blockers: architecture.json does not declare the new public derive_contract_entry_point surface or describe the behavior (either make it private or update <pdd-interface> and sync architecture), and the generated-contract reference in pdd/docs/prompting_guide.md still omits ## Entry Point / ## Seams and still demonstrates prose Oracle bullets. The .pdd profile/rotation files are present, but the same-PR authority transition violates docs/ci.md and the ownership digest is stale.
Validation: tests/test_story_contract_entry_point.py passes (11 tests); the focused story suites reached 82 passes with one unrelated local subprocess/import contamination; both protected rollout-policy failures reproduce locally; GitHub's full unit job reports 16,336 passed / 2 failed; JSON parsing and git diff --check pass. Contributor evidence is not sufficient end-to-end because it tests helpers and hand-authored valid contracts, not generated contract → pdd test --from-story → pytest. Before re-review, add a realistic safe generation-to-pytest test that is green on correct behavior and red after mutation, plus a regeneration test preserving the executable contract. PR #2397 directly overlaps this implementation and is also changes-requested; consolidate on one approach rather than merging both.
| # instead -- no model involved, and omitted entirely when ambiguous. | ||
| entry_point = derive_contract_entry_point(linked_prompt_paths, prompts_root) | ||
| if entry_point: | ||
| body = _insert_entry_point(body, entry_point) |
There was a problem hiding this comment.
P1: This injects ## Entry Point into output from generate_story_contract_LLM.prompt, but that meta-prompt still mandates prose Oracle/Negative bullets (- <detail>, - <forbidden outcome ...>). Heading-presence routing then sends the contract to _assertion_from_bullet, which rejects the prose. I reproduced ValueError: ... requires Oracle/Negative Cases bullets to be Python assertion expressions; got: 'returned value shape'. Validate the complete executable spec before insertion and omit Entry Point unless it is valid, or land the safely constrained executable Oracle format through the required prompt-governance sequence. Add generated-contract → --from-story coverage.
| "## Entry Point\n\n" | ||
| f"- module: {module}\n" | ||
| f"- callable: {callables[0]}\n" | ||
| "- args: []\n" |
There was a problem hiding this comment.
P1: Every derived invocation discards required arguments. The new test fixture declares checkout_total(a, b) and then explicitly accepts args: []; passing that derived contract through generated pytest reproduces TypeError: checkout_total() missing 2 required positional arguments. Do not infer fixture values. Auto-derive only callables with zero required arguments, or require an explicitly authored invocation contract.
| These details matter for pass/fail: | ||
| These details matter for pass/fail. | ||
|
|
||
| When `## Entry Point` is declared, write these as Python expressions over |
There was a problem hiding this comment.
P1 security: This newly advertises executable Python expressions, while downstream validation only uses ast.parse(..., mode="eval") and splices the expression verbatim into pytest. Calls such as __import__(...).system(...) are accepted. Once this PR injects Entry Point, issue-derived LLM text can cross into executable test code. Use a structured predicate format or a strict AST allowlist limited to result, literals, subscriptions, boolean/comparison operators, and explicitly safe helpers; reject calls/imports/unsafe attributes before writing the contract.
| inventory = _scan_prompt_inventory(prompts_root, extra_paths=extra_prompt_paths) | ||
| # Materialize once: the caller may pass a generator, and it is read twice | ||
| # (inventory scan, then Entry Point derivation). | ||
| linked_prompt_paths = list(extra_prompt_paths) |
There was a problem hiding this comment.
P1: sync_user_story_contract() calls this path with extra_prompt_paths=[] even though it parses primary_refs. I reproduced a successful forced sync of a story linked to one valid prompt/source/callable and the regenerated contract had no Entry Point, silently downgrading the next --from-story generation to text-pin mode. Resolve the story metadata refs against prompts_root, pass those paths here during sync, and add a preservation regression test.
| "6e589170b67c9547fad99dca53d32a085ecb3e9074a564419f97fc7316546888", | ||
| "6e589170b67c9547fad99dca53d32a085ecb3e9074a564419f97fc7316546888", | ||
| ) | ||
| _STORY_ENTRY_POINT_PROFILE_STATE = ( |
There was a problem hiding this comment.
P1 governance/security: This candidate-specific state makes authority installed and consumed by this same PR appear valid. The branch simultaneously changes the managed prompt, adds/retire-reissues rotation authority, updates the final profile, and adds this verifier acceptance. docs/ci.md:31-57 explicitly forbids installing and consuming a row in one PR. Remove this self-authorization; first merge a Phase A containing only the exact dormant authority with prompt/profile bytes unchanged, then consume those protected bytes in a rebased Phase B.
| # lose their repaired ownership, and they resurface as unowned tracked paths. | ||
| _SYNC_ROLLOUT_REPAIR_OWNERSHIP_BYTES = ( | ||
| "8f5762a5dd7be6cc14c85138810b8bad8183f4403c74584489a0d81798ba2a07", | ||
| "b3b5ed24958d6115c0a47358f30bfbab001cf13a6ef692ced94a307fc753ef9f", |
There was a problem hiding this comment.
P1: This pin does not match the checked-in .pdd/sync-ownership.json. Both local validation and required CI compute a18bfe5b655527387ad151a3042a4a0ac8f60250645e18eaf740c9d853e6965d, not b3b5ed...; the bridge falls through and the eight repaired metadata paths lose ordinary ownership. Rebase on current main, reconcile the four protected-state conflicts, pin the exact resulting bytes through the authorized sequence, and rerun the full policy suite.
…ntry Point - Reject inserting `## Entry Point` unless every Oracle/Negative Cases bullet already parses as a safe assertion expression, since the meta-prompt still asks for prose there and the behavioural generator hard-errors on non-expression bullets (P1: pdd/user_story_tests.py). - Harden `_assertion_from_bullet` with a strict AST allowlist (result, literals, containers, subscripts/non-dunder attributes, boolean/comparison/arithmetic ops, and a small safe-call allowlist) so LLM/issue-derived Oracle text can no longer splice arbitrary code into generated pytest source (P1 security: pdd/story_test_generator.py). - Only derive an Entry Point for callables whose declared signature has zero required parameters; a callable with required args and a blind `args: []` produced a generated test that TypeErrors before reaching the Oracle (P1: pdd/user_story_tests.py). - `sync_user_story_contract` now resolves the story's pdd-story-prompts refs against prompts_root and passes them through, instead of always calling contract generation with `extra_prompt_paths=[]`, which silently downgraded every forced sync to the traceability path (P1). - Revert this PR's self-authorizing rotation of `user_story_tests_python.prompt` (prompt text, profile/rotation JSON, and `_STORY_ENTRY_POINT_PROFILE_STATE`): it installed and consumed a managed-prompt transition in the same PR, which docs/ci.md forbids. The prompt text wasn't load-bearing for any test. - Re-pin `_SYNC_ROLLOUT_REPAIR_OWNERSHIP_BYTES` in pdd/sync_core/manifest.py to the sha256 `.pdd/sync-ownership.json` actually has at this head; the previous pin was stale and fell through to `base_rules`, silently dropping repaired ownership for eight tracked metadata paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion revert Reverting this PR's self-authorizing rotation of user_story_tests_python.prompt changed the whole-file sha256 of .pdd/verification-profiles.json, which test_sync_rollout_repair_executes_the_actual_protected_transition checks against a fixed set of reviewed (base, head) pairs. Follow the existing convention (_CODE_GENERATOR_LANGUAGE_GATE_PROFILE_BYTES et al.): a named constant in verification.py, referenced from the test, rather than an inline literal. No new prompt-transition authorization is granted -- the reverted prompt/profile row is back to the CONTRACT-SHA256 main already recognizes, confirmed by test_current_profile_reconciliation_matches_current_prompt_and_profile_rows passing unchanged. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ntry Point
- Harden the Oracle/Negative-Cases assertion allowlist: bare names are
now restricted to `result` plus explicit safe builtins (the generated
test module has `importlib`/`module` in scope, so a bare name check
alone let `importlib.import_module("os").system(...)` and
`module.os.system(...)` through), every attribute/subscript chain
must be rooted at `result`, and a method call on `result` must be in
a small read-only allowlist (blocks `result.clear()` and similar
mutators) (P1 security: pdd/story_test_generator.py).
- Derive the Entry Point module via PDD's actual prompt->code mapping
(`_prompt_to_code_path` / `_resolve_src_dir`, honoring `PDD_SRC_DIR`)
instead of a reinvented `<prompts_root>/../<name>` convention that
doesn't match how PDD actually lays out generated code; require the
mapped source file to exist. Updated the greeter test fixture to use
the real default `src/` layout and added PDD_SRC_DIR-override
coverage (P1 correctness/compatibility: pdd/user_story_tests.py).
- Validate that Entry Point args/kwargs actually bind to the declared
callable's signature via `inspect.Signature.bind`, not just that
they're Python literals -- a `greet(name)` with `args: []` previously
passed generation and TypeError'd before reaching the Oracle (P1
correctness: pdd/user_story_tests.py).
- Remove the self-authorizing `story_contract_state` rotation
acceptance (`_STORY_CONTRACT_ROTATION_POLICY_BYTES` /
`_STORY_CONTRACT_PROFILE_BYTES`) from verification.py: this PR
installs and consumes a managed-prompt transition for both
generate_story_contract_LLM.prompt and user_story_tests_python.prompt
in the same PR, which docs/ci.md forbids. Unlike promptdriven#2395, these prompt
edits are load-bearing (the meta-prompt needs the new
`PRIMARY_PROMPT_INTERFACES` block to pass interface data to the LLM
at all), so they can't simply be reverted -- per the reviewer's
explicit instruction, a real Phase A rotation must be merged to main
first, then this PR rebased to consume it as Phase B.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs #2391
Problem
pdd test --from-storycan generate a behavioural test — one that imports the callable, invokes it, and asserts the contract's## Oracle/## Negative Casesas Python expressions overresult. It routes to that generator on the presence of a## Entry Pointheading (pdd/story_test_generation.py:288).The contract template does not offer that section, so almost nothing declares it: 1 of 34 contracts has
## Entry Point, and 0 have## Seams. The behavioural path is effectively unreachable, and every other story falls through to a text-pinning test that never executes the code it is about.Change
Template.
user_stories/contracts/template.contract.mdnow declares## Entry Pointand## Seams, states the consequence of omitting them, and warns that a partial block is rejected rather than silently downgraded.## Oraclenow says its bullets must be Python expressions overresultwhen an Entry Point is present.Deterministic derivation. Contract generation derives the section from the story's linked prompt rather than asking a model for it: the prompt→code mapping supplies the module, and the prompt's own
<pdd-interface>supplies the callable. No LLM involved, so the result is checkable rather than guessed. #2391 suggested<pdd-interface>as the natural source; doing it in Python is what keeps it ungated (see below).Emitted only when unambiguous — one linked prompt declaring exactly one callable. This is deliberate and conservative:
pdd test --from-storyinto a hard error (pdd#1889 C-F7)Measured on this repo, it currently fires on 0 of 34 stories: 20 link more than one prompt, 10 link a prompt with no
<pdd-interface>(CLI-shaped prompts), 3 declare several callables. That is the intended behaviour of the restriction, not a defect — the cases it declines are the ones where it would produce the error below.Two findings that change what #2391 asks for
1. The issue's step 2 is gated work. It proposes teaching
_llm_generate_story_contractto populate the sections. The section vocabulary is not read from the template — it is hardcoded in the meta-prompt's<output_format>(pdd/prompts/generate_story_contract_LLM.prompt:64-100);template.contract.mdis a separate human-facing document. So that step means editing a managed prompt, which invalidates itsCONTRACT-SHA256and needs the two-phase transitiondocs/ci.mdrequires. Deriving in Python sidesteps that entirely.2.
## Entry Pointalone is not enough, and alone it breaks things. The behavioural generator requires## Oraclebullets to be Python expressions —_assertion_from_bullet(pdd/story_test_generator.py:84)ast.parses each one and raises otherwise. Every existing contract has prose there, because the meta-prompt asks for prose (## Oracle/These details matter for pass/fail:/- <detail>).Repro — a contract with
## Entry Pointand a prose Oracle:So adding an Entry Point to a contract whose Oracle is prose converts a working (if weak) generation into a hard error — the same failure the no-partial-blocks rule avoids, arriving through a different door. Entry Point and Oracle-as-expressions have to change together, and the Oracle format lives in the gated meta-prompt.
This is why the derivation is restricted to the case that cannot break: the missing sections are a symptom, and the binding constraint is that contracts are authored in prose while the behavioural generator needs expressions.
Deliberately not done: backfilling existing contracts
#2391 suggests backfilling opportunistically. Adding a section changes the contract text, which changes
story_bundle_hash(), which invalidates thePDD_STORY_HASHrecorded in every linked test — staling them all at once. Combined with the prose-Oracle problem above, a backfill today would stale the suite and break generation for the stories it touched. That is a migration deserving its own change.Tests
9 new tests in
tests/test_story_contract_entry_point.pycovering derivation (single callable), each refusal path (several prompts, several callables, no interface, no prompts root), and insertion (before## Oracle, no-op when already declared, appended when absent). 213 tests pass across the story, generation and coverage suites.The new test file gets a
.pdd/sync-ownership.jsonrule, and the ownership digest is re-pinned inpdd/sync_core/manifest.pywith the previously authorized head kept alongside the new pair, per the multi-head convention already used there.Relationship to the other story work
Independent of #2390 (story verification engine) and #2392 (silent traceability fallback) — this branch is cut from
mainand touches different files. #2392 makes the fallback visible; this makes the behavioural path reachable; neither fully closes the gap until the Oracle format changes.