Skip to content

docs: README gaps on the authoring-to-first-published-record path #205

Description

@rorybyrne

Collected README gaps found while taking the platform from nothing to a working archive, following only the README. Individually small; together they're most of why the first-published-record path took three deploy cycles.

Checked against docs/readme-sdk-workflow @ dd8f3b8 — the branch README is byte-identical to what I reviewed, so none of these are already fixed there.

Items with their own issues (linked below) are omitted here.

1. The "Describe your data" example isn't runnable

It reads as a complete example but can't be run:

  • files= omits max_file_size, which the server requires.
  • Pocket is referenced as a hook return type and never defined.
  • PDBIngester is referenced as ingester= and never shown — see bug: GEOIngestor httpx.AsyncClient is never closed #5 below.
  • The implied package layout fails to build.

Either make it complete and runnable, or label it explicitly as an outline and link a full worked example somewhere.

2. No pyproject.toml is shown beyond the entry point

The README shows only:

[project.entry-points."osa.conventions"]
pockets = "mypkg.convention"

There's no build-system section and no mention that the layout is constrained by how the hook/ingester image is built. That constraint is build-breaking and currently documented nowhere.

3. The --convention slug is not the entry point name

osa ingestion start --convention pockets   # matches the entry point above

The real slug is derived server-side from the convention titletitle="Global Seismic Events" becomes global-seismic-events. osa deploy prints it at the end, which saves you, but the example teaches the wrong mental model. convention()'s docstring ("identity slug is assigned server-side") is the only hint.

4. Supported field types are never listed

The schema fields the server accepts are text, number, date, boolean, term, url. Nothing says so, and there's no mention that Python datetime is not among them — which is the root of the most expensive bug I hit. A four-line table next to the Schema example would have saved that entire debugging session.

5. Ingesters — the hardest part of the SDK — are undocumented

The README references ingester=PDBIngester and never shows one. Writing an ingester requires knowing:

  • pull() is an async generator with a five-argument keyword-only signature (ctx, since, limit, offset, session).
  • IngesterRecord, IngesterFileRef, and IngesterContext (add_file / add_bytes / set_session).
  • The name / schedule / initial_run / max_file_mb / limits ClassVar contract.
  • That session state is the mechanism for incremental pulls.
  • That defining a RuntimeConfig inner class changes how your ingester is constructed — ingester_cls(config) vs ingester_cls() (ingester_entrypoint.py:104).

I recovered all of it by reading osa/authoring/ingester.py and osa/runtime/ingester_entrypoint.py. One worked ingester in the docs would cover nearly all of it.

Also worth documenting: ingest dedup keys on convention_slug:source_id, which has real consequences for re-running a failed ingestion.

6. Smaller things

  • osa init generates osa.yaml with ORCID placeholders and base_role: DEPOSITOR that the quickstart never explains — a mild "should I be configuring this?" pause against the README's "nothing to configure".
  • .data/ is created empty and stays empty; its purpose is never stated.
  • osa test and osa deploy prompt for confirmation with no --yes flag, so scripting them means piping yes |.

What's already good

Worth stating, since the above is all gaps: osa initosa start genuinely is a one-command working stack, up in 12 seconds. The mandatory-docs gate is a strong idea, well implemented, and fails fast with a clear explanation. Auto-generated /SKILL.md staying in sync with the data is exactly right. The /data/ read surface, filter API, and MCP endpoint all worked first try with no configuration.

The gap is concentrated in the authoring-to-first-published-record path — the path every new user walks exactly once.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions