fix: describe the unpublished endpoints, and stop publishing other people's drafts

The defect reports kept landing on the same root cause, so this addresses that
rather than only its symptoms.

Forward's published endpoints have a description generated from its server, so a
parser cannot disagree with them for long. The unpublished ones had nothing:
their responses were read key by key, by hand, and their fixtures were written
from the same assumptions as the parsers. When both were wrong together nothing
noticed, which is how a commit id nested under lastCommit came to be read from a
flat key Forward never sends.

They now have real schemas, merged into the generated description, so models are
generated for them exactly as for published responses and parsing validates
against a declared shape. The recorded examples are run through the SDK's own
parsers, and an unpublished operation that documents no shape is a test failure.

Two behaviours found by running both clients against one fake Forward:

publish() staged over pre-existing drafts and committed them. A commit names
paths rather than changes, so an unrelated half-finished edit sitting on a path
being published was committed along with it, and neither party was told. It now
refuses, with overwrite_drafts to say otherwise deliberately.

The 409 strip-and-retry read its paths from the exception's display string when
the error body would not validate. That string embeds the raw body inside
formatting, so splitting it produced fragments matching nothing, only some paths
were stripped, and the retry failed again on a 409 that looked like a different
problem. It now reads the parsed body, or the raw body as JSON, and declines to
strip at all when the paths cannot be identified. This endpoint is unpublished,
so its error envelope is the least guaranteed anywhere in the SDK, which makes
it the worst possible place to key behaviour off a display string. That pattern
is one this SDK exists to remove from its consumers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ERUwz8RFoDEdZfp4hFbGHz

Author: captainpacket

scripts/downconvert_spec.py

@@ -44,6 +44,12 @@ class would be actively harmful here.
 SOURCE_VERSION = "3.2.0"
 TARGET_VERSION = "3.1.0"
 
+#: Endpoints Forward does not publish, described by hand. Merged in here so
+#: models are generated for their responses exactly as for published ones, and
+#: so request validation covers them too. Without this their shapes are read by
+#: hand, which is what lost a commit id nested under `lastCommit`.
+UNPUBLISHED = Path("spec/unpublished.yaml")
+
 STREAMING_MEDIA_TYPES = ("application/jsonl", "application/x-ndjson", "application/json-seq")
 
 # Schemas whose contents are defined by the user's NQE query rather than by the
@@ -248,6 +254,38 @@ def _coerce_scalar(value: Any, declared: str) -> Any | None:
     return None
 
 
+def merge_unpublished(doc: dict[str, Any], extra: dict[str, Any]) -> int:
+    """Fold the hand-written description of unpublished endpoints into ``doc``.
+
+    Their operations are marked so that anything reading the merged document can
+    tell them apart from what Forward publishes.
+    """
+    merged = 0
+    schemas = doc.setdefault("components", {}).setdefault("schemas", {})
+    for name, schema in (extra.get("components", {}).get("schemas") or {}).items():
+        if name in schemas:
+            raise UnsupportedSpecFeature(
+                f"unpublished schema {name!r} collides with a published one; rename it"
+            )
+        schemas[name] = schema
+
+    for path, item in (extra.get("paths") or {}).items():
+        target = doc.setdefault("paths", {}).setdefault(path, {})
+        for method, operation in item.items():
+            if method in target:
+                raise UnsupportedSpecFeature(
+                    f"unpublished {method.upper()} {path} collides with a published operation"
+                )
+            if isinstance(operation, dict):
+                operation.setdefault("x-forward-stability", "unpublished")
+                merged += 1
+            target[method] = operation
+
+    for tag in extra.get("tags") or []:
+        doc.setdefault("tags", []).append(tag)
+    return merged
+
+
 def downconvert(doc: dict[str, Any]) -> tuple[dict[str, Any], dict[str, int]]:
     if doc.get("openapi") != SOURCE_VERSION:
         raise UnsupportedSpecFeature(
@@ -269,11 +307,19 @@ def downconvert(doc: dict[str, Any]) -> tuple[dict[str, Any], dict[str, int]]:
 def main(argv: list[str] | None = None) -> int:
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--input", type=Path, default=Path("spec/forward-openapi.yaml"))
+    parser.add_argument("--unpublished", type=Path, default=UNPUBLISHED)
     parser.add_argument("--output", type=Path, default=Path("spec/forward-openapi-3.1.json"))
     args = parser.parse_args(argv)
 
     doc = yaml.safe_load(args.input.read_text(encoding="utf-8"))
+
+    unpublished = 0
+    if args.unpublished.exists():
+        extra = yaml.safe_load(args.unpublished.read_text(encoding="utf-8")) or {}
+        unpublished = merge_unpublished(doc, extra)
+
     converted, stats = downconvert(doc)
+    stats["unpublished_operations"] = unpublished
     args.output.write_text(json.dumps(converted, indent=2, sort_keys=True) + "\n", encoding="utf-8")
 
     print(f"wrote {args.output}")

scripts/gen_public_models.py

@@ -32,6 +32,12 @@
     "DiffEntryType": "Type32",
 }
 
+#: Generated names the SDK supersedes with a friendlier type of its own. The
+#: generated ones mirror the wire shape (a nested commit object, for instance);
+#: the hand-written ones are flat and are what callers use. Exporting both under
+#: one name would leave which you got depending on import order.
+SUPERSEDED = frozenset({"RepositoryQuery", "DraftChange"})
+
 #: Types the SDK defines itself, alongside the generated ones, because they
 #: describe endpoints Forward does not publish a schema for.
 HAND_WRITTEN = (
@@ -65,7 +71,9 @@ def model_names() -> list[str]:
     return sorted(
         name
         for name in CLASS_DECLARATION.findall(source)
-        if not name.startswith("_") and name not in {"ForwardModel", "OpenEnum"}
+        if not name.startswith("_")
+        and name not in {"ForwardModel", "OpenEnum"}
+        and name not in SUPERSEDED
     )