Skip to content

feat(story contracts): declare and derive ## Entry Point / ## Seams - #2395

Open
agarwal-ishaan wants to merge 8 commits into
promptdriven:mainfrom
agarwal-ishaan:fix/issue-2391-contract-entry-point
Open

feat(story contracts): declare and derive ## Entry Point / ## Seams#2395
agarwal-ishaan wants to merge 8 commits into
promptdriven:mainfrom
agarwal-ishaan:fix/issue-2391-contract-entry-point

Conversation

@agarwal-ishaan

@agarwal-ishaan agarwal-ishaan commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Refs #2391

Problem

pdd test --from-story can generate a behavioural test — one that imports the callable, invokes it, and asserts the contract's ## Oracle / ## Negative Cases as Python expressions over result. It routes to that generator on the presence of a ## Entry Point heading (pdd/story_test_generation.py:288).

The contract template does not offer that section, so almost nothing declares it: 1 of 34 contracts has ## Entry Point, and 0 have ## Seams. The behavioural path is effectively unreachable, and every other story falls through to a text-pinning test that never executes the code it is about.

Change

Template. user_stories/contracts/template.contract.md now declares ## Entry Point and ## Seams, states the consequence of omitting them, and warns that a partial block is rejected rather than silently downgraded. ## Oracle now says its bullets must be Python expressions over result when an Entry Point is present.

Deterministic derivation. Contract generation derives the section from the story's linked prompt rather than asking a model for it: the prompt→code mapping supplies the module, and the prompt's own <pdd-interface> supplies the callable. No LLM involved, so the result is checkable rather than guessed. #2391 suggested <pdd-interface> as the natural source; doing it in Python is what keeps it ungated (see below).

Emitted only when unambiguous — one linked prompt declaring exactly one callable. This is deliberate and conservative:

  • a partial block is worse than none: the generator routes on heading presence, so a declared-but-incomplete Entry Point turns pdd test --from-story into a hard error (pdd#1889 C-F7)
  • a guessed callable produces a confidently wrong behavioural test, which is worse than a documentary one because it looks like real cover

Measured on this repo, it currently fires on 0 of 34 stories: 20 link more than one prompt, 10 link a prompt with no <pdd-interface> (CLI-shaped prompts), 3 declare several callables. That is the intended behaviour of the restriction, not a defect — the cases it declines are the ones where it would produce the error below.

Two findings that change what #2391 asks for

1. The issue's step 2 is gated work. It proposes teaching _llm_generate_story_contract to populate the sections. The section vocabulary is not read from the template — it is hardcoded in the meta-prompt's <output_format> (pdd/prompts/generate_story_contract_LLM.prompt:64-100); template.contract.md is a separate human-facing document. So that step means editing a managed prompt, which invalidates its CONTRACT-SHA256 and needs the two-phase transition docs/ci.md requires. Deriving in Python sidesteps that entirely.

2. ## Entry Point alone is not enough, and alone it breaks things. The behavioural generator requires ## Oracle bullets to be Python expressions — _assertion_from_bullet (pdd/story_test_generator.py:84) ast.parses each one and raises otherwise. Every existing contract has prose there, because the meta-prompt asks for prose (## Oracle / These details matter for pass/fail: / - <detail>).

Repro — a contract with ## Entry Point and a prose Oracle:

ValueError: Story test generation requires Oracle/Negative Cases bullets to be
Python assertion expressions; got: 'selected workflow: agentic bug vs manual bug repair'

So adding an Entry Point to a contract whose Oracle is prose converts a working (if weak) generation into a hard error — the same failure the no-partial-blocks rule avoids, arriving through a different door. Entry Point and Oracle-as-expressions have to change together, and the Oracle format lives in the gated meta-prompt.

This is why the derivation is restricted to the case that cannot break: the missing sections are a symptom, and the binding constraint is that contracts are authored in prose while the behavioural generator needs expressions.

Deliberately not done: backfilling existing contracts

#2391 suggests backfilling opportunistically. Adding a section changes the contract text, which changes story_bundle_hash(), which invalidates the PDD_STORY_HASH recorded in every linked test — staling them all at once. Combined with the prose-Oracle problem above, a backfill today would stale the suite and break generation for the stories it touched. That is a migration deserving its own change.

Tests

9 new tests in tests/test_story_contract_entry_point.py covering derivation (single callable), each refusal path (several prompts, several callables, no interface, no prompts root), and insertion (before ## Oracle, no-op when already declared, appended when absent). 213 tests pass across the story, generation and coverage suites.

The new test file gets a .pdd/sync-ownership.json rule, and the ownership digest is re-pinned in pdd/sync_core/manifest.py with the previously authorized head kept alongside the new pair, per the multi-head convention already used there.

Relationship to the other story work

Independent of #2390 (story verification engine) and #2392 (silent traceability fallback) — this branch is cut from main and touches different files. #2392 makes the fallback visible; this makes the behavioural path reachable; neither fully closes the gap until the Oracle format changes.

agarwal-ishaan and others added 4 commits August 10, 2026 19:36
The contract template offered no ## Entry Point, so essentially every contract
landed on the traceability path: `pdd test --from-story` routes on that
heading's presence, and only 1 of 34 contracts declared it. The behavioural
generator was effectively unreachable.

- the contract template now declares ## Entry Point and ## Seams, with the
  consequence spelled out (omit it and the generated test only pins text), and
  ## Oracle now says its bullets must be Python expressions over `result` when
  an Entry Point is declared
- contract generation derives the section deterministically from the story's
  linked prompt -- the prompt->code mapping supplies the module and the
  prompt's <pdd-interface> supplies the callable. No model is involved, so the
  result is checkable rather than guessed.

Derivation is emitted ONLY when a single linked prompt declares a single
callable. A partial or guessed Entry Point is worse than none: the generator
routes on heading presence, so a declared-but-incomplete block turns
`pdd test --from-story` into a hard error (pdd#1889 C-F7), and a wrong callable
yields a confidently wrong behavioural test. When ambiguous the section is
omitted and the story keeps the traceability path it has today.

Existing contracts are deliberately NOT backfilled: adding a section changes
the contract text, which changes story_bundle_hash(), which stales the recorded
PDD_STORY_HASH in every linked test. That migration wants its own change.

Refs promptdriven#2391

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@agarwal-ishaan
agarwal-ishaan marked this pull request as ready for review August 13, 2026 14:02

@gltanaka gltanaka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier A, full-diff review: CHANGES REQUESTED. Issue #2391 is real and still needs a fix: the behavioral route introduced by #1867 is effectively unreachable from normal contract generation. The deterministic-binding direction here is preferable to guessing, but this head does not yet produce valid or safely executable behavioral contracts, is conflicting with current main, and has a failing required Unit Tests check.

I reproduced the prose-Oracle hard failure, required-argument invocation failure, and contract-sync downgrade through the actual generation handoff. I also audited the protected prompt transition, architecture/prompt artifacts, related #2393/#2397 work, and current CI. See the six inline findings.

Artifact gaps remain after those blockers: architecture.json does not declare the new public derive_contract_entry_point surface or describe the behavior (either make it private or update <pdd-interface> and sync architecture), and the generated-contract reference in pdd/docs/prompting_guide.md still omits ## Entry Point / ## Seams and still demonstrates prose Oracle bullets. The .pdd profile/rotation files are present, but the same-PR authority transition violates docs/ci.md and the ownership digest is stale.

Validation: tests/test_story_contract_entry_point.py passes (11 tests); the focused story suites reached 82 passes with one unrelated local subprocess/import contamination; both protected rollout-policy failures reproduce locally; GitHub's full unit job reports 16,336 passed / 2 failed; JSON parsing and git diff --check pass. Contributor evidence is not sufficient end-to-end because it tests helpers and hand-authored valid contracts, not generated contract → pdd test --from-story → pytest. Before re-review, add a realistic safe generation-to-pytest test that is green on correct behavior and red after mutation, plus a regeneration test preserving the executable contract. PR #2397 directly overlaps this implementation and is also changes-requested; consolidate on one approach rather than merging both.

Comment thread pdd/user_story_tests.py
# instead -- no model involved, and omitted entirely when ambiguous.
entry_point = derive_contract_entry_point(linked_prompt_paths, prompts_root)
if entry_point:
body = _insert_entry_point(body, entry_point)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This injects ## Entry Point into output from generate_story_contract_LLM.prompt, but that meta-prompt still mandates prose Oracle/Negative bullets (- <detail>, - <forbidden outcome ...>). Heading-presence routing then sends the contract to _assertion_from_bullet, which rejects the prose. I reproduced ValueError: ... requires Oracle/Negative Cases bullets to be Python assertion expressions; got: 'returned value shape'. Validate the complete executable spec before insertion and omit Entry Point unless it is valid, or land the safely constrained executable Oracle format through the required prompt-governance sequence. Add generated-contract → --from-story coverage.

Comment thread pdd/user_story_tests.py
"## Entry Point\n\n"
f"- module: {module}\n"
f"- callable: {callables[0]}\n"
"- args: []\n"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Every derived invocation discards required arguments. The new test fixture declares checkout_total(a, b) and then explicitly accepts args: []; passing that derived contract through generated pytest reproduces TypeError: checkout_total() missing 2 required positional arguments. Do not infer fixture values. Auto-derive only callables with zero required arguments, or require an explicitly authored invocation contract.

These details matter for pass/fail:
These details matter for pass/fail.

When `## Entry Point` is declared, write these as Python expressions over

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 security: This newly advertises executable Python expressions, while downstream validation only uses ast.parse(..., mode="eval") and splices the expression verbatim into pytest. Calls such as __import__(...).system(...) are accepted. Once this PR injects Entry Point, issue-derived LLM text can cross into executable test code. Use a structured predicate format or a strict AST allowlist limited to result, literals, subscriptions, boolean/comparison operators, and explicitly safe helpers; reject calls/imports/unsafe attributes before writing the contract.

Comment thread pdd/user_story_tests.py
inventory = _scan_prompt_inventory(prompts_root, extra_paths=extra_prompt_paths)
# Materialize once: the caller may pass a generator, and it is read twice
# (inventory scan, then Entry Point derivation).
linked_prompt_paths = list(extra_prompt_paths)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: sync_user_story_contract() calls this path with extra_prompt_paths=[] even though it parses primary_refs. I reproduced a successful forced sync of a story linked to one valid prompt/source/callable and the regenerated contract had no Entry Point, silently downgrading the next --from-story generation to text-pin mode. Resolve the story metadata refs against prompts_root, pass those paths here during sync, and add a preservation regression test.

Comment thread pdd/sync_core/verification.py Outdated
"6e589170b67c9547fad99dca53d32a085ecb3e9074a564419f97fc7316546888",
"6e589170b67c9547fad99dca53d32a085ecb3e9074a564419f97fc7316546888",
)
_STORY_ENTRY_POINT_PROFILE_STATE = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 governance/security: This candidate-specific state makes authority installed and consumed by this same PR appear valid. The branch simultaneously changes the managed prompt, adds/retire-reissues rotation authority, updates the final profile, and adds this verifier acceptance. docs/ci.md:31-57 explicitly forbids installing and consuming a row in one PR. Remove this self-authorization; first merge a Phase A containing only the exact dormant authority with prompt/profile bytes unchanged, then consume those protected bytes in a rebased Phase B.

Comment thread pdd/sync_core/manifest.py Outdated
# lose their repaired ownership, and they resurface as unowned tracked paths.
_SYNC_ROLLOUT_REPAIR_OWNERSHIP_BYTES = (
"8f5762a5dd7be6cc14c85138810b8bad8183f4403c74584489a0d81798ba2a07",
"b3b5ed24958d6115c0a47358f30bfbab001cf13a6ef692ced94a307fc753ef9f",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This pin does not match the checked-in .pdd/sync-ownership.json. Both local validation and required CI compute a18bfe5b655527387ad151a3042a4a0ac8f60250645e18eaf740c9d853e6965d, not b3b5ed...; the bridge falls through and the eight repaired metadata paths lose ordinary ownership. Rebase on current main, reconcile the four protected-state conflicts, pin the exact resulting bytes through the authorized sequence, and rerun the full policy suite.

agarwal-ishaan and others added 2 commits August 14, 2026 21:47
…ntry Point

- Reject inserting `## Entry Point` unless every Oracle/Negative Cases
  bullet already parses as a safe assertion expression, since the
  meta-prompt still asks for prose there and the behavioural generator
  hard-errors on non-expression bullets (P1: pdd/user_story_tests.py).
- Harden `_assertion_from_bullet` with a strict AST allowlist (result,
  literals, containers, subscripts/non-dunder attributes,
  boolean/comparison/arithmetic ops, and a small safe-call allowlist) so
  LLM/issue-derived Oracle text can no longer splice arbitrary code into
  generated pytest source (P1 security: pdd/story_test_generator.py).
- Only derive an Entry Point for callables whose declared signature has
  zero required parameters; a callable with required args and a blind
  `args: []` produced a generated test that TypeErrors before reaching
  the Oracle (P1: pdd/user_story_tests.py).
- `sync_user_story_contract` now resolves the story's pdd-story-prompts
  refs against prompts_root and passes them through, instead of always
  calling contract generation with `extra_prompt_paths=[]`, which
  silently downgraded every forced sync to the traceability path (P1).
- Revert this PR's self-authorizing rotation of
  `user_story_tests_python.prompt` (prompt text, profile/rotation JSON,
  and `_STORY_ENTRY_POINT_PROFILE_STATE`): it installed and consumed a
  managed-prompt transition in the same PR, which docs/ci.md forbids.
  The prompt text wasn't load-bearing for any test.
- Re-pin `_SYNC_ROLLOUT_REPAIR_OWNERSHIP_BYTES` in
  pdd/sync_core/manifest.py to the sha256 `.pdd/sync-ownership.json`
  actually has at this head; the previous pin was stale and fell
  through to `base_rules`, silently dropping repaired ownership for
  eight tracked metadata paths.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tion revert

Reverting this PR's self-authorizing rotation of
user_story_tests_python.prompt changed the whole-file sha256 of
.pdd/verification-profiles.json, which
test_sync_rollout_repair_executes_the_actual_protected_transition checks
against a fixed set of reviewed (base, head) pairs. Follow the existing
convention (_CODE_GENERATOR_LANGUAGE_GATE_PROFILE_BYTES et al.): a named
constant in verification.py, referenced from the test, rather than an
inline literal. No new prompt-transition authorization is granted -- the
reverted prompt/profile row is back to the CONTRACT-SHA256 main already
recognizes, confirmed by
test_current_profile_reconciliation_matches_current_prompt_and_profile_rows
passing unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
agarwal-ishaan added a commit to agarwal-ishaan/pdd that referenced this pull request Aug 15, 2026
…ntry Point

- Harden the Oracle/Negative-Cases assertion allowlist: bare names are
  now restricted to `result` plus explicit safe builtins (the generated
  test module has `importlib`/`module` in scope, so a bare name check
  alone let `importlib.import_module("os").system(...)` and
  `module.os.system(...)` through), every attribute/subscript chain
  must be rooted at `result`, and a method call on `result` must be in
  a small read-only allowlist (blocks `result.clear()` and similar
  mutators) (P1 security: pdd/story_test_generator.py).
- Derive the Entry Point module via PDD's actual prompt->code mapping
  (`_prompt_to_code_path` / `_resolve_src_dir`, honoring `PDD_SRC_DIR`)
  instead of a reinvented `<prompts_root>/../<name>` convention that
  doesn't match how PDD actually lays out generated code; require the
  mapped source file to exist. Updated the greeter test fixture to use
  the real default `src/` layout and added PDD_SRC_DIR-override
  coverage (P1 correctness/compatibility: pdd/user_story_tests.py).
- Validate that Entry Point args/kwargs actually bind to the declared
  callable's signature via `inspect.Signature.bind`, not just that
  they're Python literals -- a `greet(name)` with `args: []` previously
  passed generation and TypeError'd before reaching the Oracle (P1
  correctness: pdd/user_story_tests.py).
- Remove the self-authorizing `story_contract_state` rotation
  acceptance (`_STORY_CONTRACT_ROTATION_POLICY_BYTES` /
  `_STORY_CONTRACT_PROFILE_BYTES`) from verification.py: this PR
  installs and consumes a managed-prompt transition for both
  generate_story_contract_LLM.prompt and user_story_tests_python.prompt
  in the same PR, which docs/ci.md forbids. Unlike promptdriven#2395, these prompt
  edits are load-bearing (the meta-prompt needs the new
  `PRIMARY_PROMPT_INTERFACES` block to pass interface data to the LLM
  at all), so they can't simply be reverted -- per the reviewer's
  explicit instruction, a real Phase A rotation must be merged to main
  first, then this PR rebased to consume it as Phase B.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants