docs: user guide for the substrait.dataframe API (Zensical)

[nielspardon]

What

Adds user-facing documentation for the ergonomic native DataFrame/Expr API
(substrait.dataframe) introduced in #204, plus a README introduction pointing
to it. There was previously no guide for the new API — the README still led with
the verbose raw-proto construction.

Built with Zensical (Material-style, zensical.toml) and
mkdocstrings for the generated API reference.
The guide's code is not inline: every snippet is pulled from a runnable script
that CI executes, so a documented example that goes stale against the API fails
CI instead of silently rendering.

Contents

• docs/ — a multi-page guide: overview, getting started, data sources,
  types & schemas, expressions, the f function namespace, transformations,
  joins, aggregations, set operations, window functions, subqueries, parameters
  & context, DDL & writes, custom extensions, consuming plans, and a
  mkdocstrings-generated API reference.
• examples/guide/ + tests/docs/ — the guide's snippets live in runnable
  scripts, not in the Markdown. Each ```python fence is a pymdownx.snippets
  include (--8<-- "examples/guide/<page>.py:<section>") pulling from a script
  under examples/guide/, and tests/docs/test_guide_snippets.py executes every
  one of those scripts end to end. Snippets that build a DataFrame call
  .to_plan(), so the test validates construction and resolution — this
  surfaced and fixed several previously-broken examples (joins on non-existent
  columns, user_defined(...), a nullable-struct schema). A my_functions.yaml
  fixture backs the custom-extension snippets. The two engine-handoff snippets
  (the DuckDB/ADBC tabs) stay inline since they need network + an external engine;
  they are exercised by examples/{duckdb,adbc}_example.py.
• zensical.toml — theme, nav, site_url, the mkdocstrings Python handler
  (pointed at src/), the mike version provider for multi-version docs, and
  pymdownx.snippets (with check_paths = true, so a bad include path or
  section name fails the build).
• pyproject.toml — a docs dependency group (zensical,
  mkdocstrings-python) plus a pixi docs environment with docs-build /
  docs-serve tasks; a per-file ruff ignore for examples/guide/ (its imports
  intentionally follow the docs rather than module conventions).
• CI — .github/workflows/docs.yml runs a build-check on every PR (which now
  also validates every --8<-- include). The existing test.yml picks up the
  new tests/docs/ suite automatically. example.yml gains dataframe_example.py
  (it was previously omitted from the matrix). docs-deploy.yml publishes
  versioned docs to the gh-pages branch via mike, on release tags (v*.*.*)
  and via workflow_dispatch.
• README.md — new "Building plans with the DataFrame API" section; the
  raw-proto examples are reframed as the low-level API.
• CONTRIBUTING.md — a Documentation section (local preview/build, docstring
  convention) plus a Guide code snippets section describing the
  pymdownx.snippets workflow; also corrects the dev-setup command (uv sync,
  not the non-existent --extra test).
• Docstring pass — converts the substrait.dataframe docstrings from
  reStructuredText (:: literal blocks, :class:/:meth: roles) to Markdown so
  the generated reference renders cleanly. Docstrings only; no behavior change.

Building locally

pixi run docs-serve   # or: uv run --group docs zensical serve

Enabling GitHub Pages (one-time, after merge)

Versioned publishing needs the gh-pages branch to exist before Pages can point
at it:

1. Actions → Deploy documentation → Run workflow (creates gh-pages).
2. Settings → Pages → Source → Deploy from a branch → gh-pages / (root).
3. From then on, each v*.*.* release tag publishes that minor series and
   updates the latest alias. Adjust site_url in zensical.toml if the site
   is served from a custom domain.

Verification

• zensical build completes with no issues (includes link validation); with
  check_paths, a wrong --8<-- path or section name fails the build.
• The full test suite passes, including tests/docs/, which runs every guide
  snippet against the real API. A deliberately broken snippet fails that suite.

🤖 Generated with AI