Extra credit: two S3-band builds probing structural editing reliability - #115
Open
shivansh193 wants to merge 3 commits into
Open
Extra credit: two S3-band builds probing structural editing reliability#115shivansh193 wants to merge 3 commits into
shivansh193 wants to merge 3 commits into
Conversation
Reconciles a base agreement against Supplementary Conditions, then redlines the reconciled document against a risk playbook retrieved via cross_session_search. Reconciliation verified working end to end against the live API. The redline step surfaced a real, reproducible SuperDocs bug: cross_session_search can silently re-open a stale snapshot of a document already open and edited in the same session, so a job reports "completed" with an "approved" diff that never actually applies to the session's real document. Full diagnosis in PROGRESS.md.
Repairs a document's Section numbering, body cross-references, and stale Table of Contents against a fully known ground truth. Two identical real runs against the live API produced very different outcomes: Run 1 (6/8 checks) cleanly renumbered and fixed both cross-refs but silently dropped a bundled TOC instruction while reporting full success -- the same silent-narrowing pattern found independently on the redline-workspace build. After splitting into single-purpose turns, Run 2 (0/8 checks) hit a different, worse failure on the byte-identical renumber instruction: a false "updated all 10 sections" claim covering near-zero real progress, a downstream turn trusting that false claim instead of checking the document, and a hand-authored TOC replaced with an empty auto-generated widget. Same instruction, two very different executions -- real run-to- run non-determinism, not a wording problem. Full diagnosis in PROGRESS.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…or self-healing's non-determinism Redline workspace: dropped cross_session_search, loaded the risk playbook into the main session as a background document instead (same pattern as the Exhibits). Re-ran for real: now passes 6/6, both required flags land correctly and the reconciliation-order check holds. Also fixed a bug in this repo's own verify() (a fixed-width per-Article window that overran into the next Article's flag on this cleaner run). Original honest-failure record is left intact in PROGRESS.md/README as prior history, not replaced. Self-healing structure agent: wrapped the renumber step (the one that showed real run-to-run non-determinism) in a verify-then-retry loop -- fresh session, fresh document, up to 3 independent attempts, checked against ground truth each time. Converged on attempt 1 in this run. The Table of Contents step failed the same way as before, now confirmed on two separate real runs with different instruction phrasing -- reads as a reproducible platform behavior, left out of scope for this pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Shivansh Kalra
Two additional builds beyond the assigned Task 2 pair, going after harder, multi-document and multi-turn structural editing rather than single-document drafting.
use-cases/shivansh193/owner-contractor-redline-workspace) -- reconciles a base agreement against Supplementary Conditions, then redlines the reconciled document against a risk playbook retrieved viacross_session_search. Reconciliation verified working end to end. The redline step surfaced a reproducible bug:cross_session_searchcan silently re-open a stale snapshot of a document already open and edited in the same session, so a job reportscompletedwith anapproveddiff that never actually lands.use-cases/shivansh193/self-healing-structure-agent) -- repairs Section numbering, cross-references, and a stale Table of Contents against a fully known ground truth. Two identical real runs of the same instruction produced very different outcomes: one clean pass, and one run with a false "updated all 10 sections" claim covering near-zero real progress, a downstream turn trusting that false claim instead of checking the document, and a hand-authored TOC replaced with an empty auto-generated widget.Both READMEs lead with the verified result, pass or fail, and each folder's
PROGRESS.mdhas the full turn-by-turn diagnosis with the underlying job diffs.