docs: add TODOs 0194-0196 — directory-scoped policy, cross-field rules, duplicate names

docs/spec/todos/TODO-0194.md

@@ -0,0 +1,104 @@
+---
+id: 194
+title: Directory-scoped schema policy — freeze undeclared fields per path
+status: todo
+priority: high
+created: 2026-08-17
+depends_on: []
+blocks: [195]
+related: [195]
+---
+
+# TODO-0194: Directory-scoped schema policy — freeze undeclared fields per path
+
+## Summary
+
+A frontmatter field that appears in no `[[fields.field]]` entry is never a violation — `check` reports it as informational "new fields" and exits 0, in both auto-update and `--no-update` modes. Add a directory-scoped policy section to `mdvs.toml` that can close that world: under a given path glob, undeclared fields become violations.
+
+## Details
+
+### The gap, measured
+
+Verified against v0.8.4:
+
+```
+$ mdvs check . --no-update
+Checked 2 files — no violations, 1 new field(s)
+
+New fields (1):
+┌ secret_new_field ────────┬──────────────────────────────┐
+│ status                   │ new (not in mdvs.toml)       │
+└──────────────────────────┴──────────────────────────────┘
+
+EXIT: 0
+```
+
+Two mechanisms produce this:
+
+- `cmd/check/validate.rs` iterates `[[fields.field]]` entries and calls `navigate_dotted` to find each declared field's value. It never iterates the *keys present in the file*, so a key mdvs has not been told about is never examined.
+- `schema/json_schema/to_canonical.rs` emits the root schema with `additionalProperties: true`. `cmd/check/field_meta.rs` notes that per-field validation "would never produce `Required` or `AdditionalProperties` errors anyway."
+
+`ViolationKind::Disallowed` exists but means something narrower: a *declared* field appearing at a path outside its `allowed` globs (`validate.rs`, the `!m.allowed.is_match(file_path_str)` branch).
+
+What auto-update does and does not do, also measured — re-inference only ever **adds new fields**. It does not widen an existing field's type or relax its constraints:
+
+| Scenario | auto-update (default) | `--no-update` |
+|---|---|---|
+| Brand-new undeclared field | silently written into `mdvs.toml`, exit 0 | listed under "New fields", exit 0 |
+| Value violating existing `categories` | exit 1 | exit 1 |
+| Wrong type on existing field | exit 1 | exit 1 |
+
+So `--no-update` is not a gate against schema drift; it only prevents `mdvs.toml` from being rewritten mid-run and surfaces the addition in the output.
+
+### Why per-directory rather than global
+
+mdvs already has a directory axis, but only *per field*: `allowed` and `required` are path globs on each `[[fields.field]]`. What is missing is the complementary **closed-world statement** — "under `notes/**`, only these fields may appear." Today `[fields].ignore` is global and the open/closed decision is all-or-nothing.
+
+A realistic vault wants both at once: a curated directory that is frozen, and a scratch directory where anything goes. A global switch cannot express that.
+
+### Proposed shape
+
+A new top-level section, one entry per scope:
+
+```toml
+[[scope]]
+path = "projects/**"
+frozen = true                 # undeclared fields here are violations
+ignore = ["scratch_notes"]    # per-scope escape hatch
+
+[[scope]]
+path = "inbox/**"
+frozen = false                # explicit open world
+```
+
+`path = "**"` covers the whole-repo freeze, so the global case falls out of the same mechanism rather than needing its own flag.
+
+### Design questions
+
+- **Precedence.** When a file matches several scopes, does the most specific glob win, or must every matching scope pass? Most-specific-wins matches how people read directory config; all-must-pass is easier to reason about formally. Needs a decision before implementation.
+- **Relationship to `[fields].ignore`.** Does the global list stay and compose with per-scope `ignore`, or does it become sugar for a `path = "**"` scope? Prefer not having two mechanisms for the same thing.
+- **Default for unmatched paths.** A file matching no scope should presumably stay open-world, preserving today's behavior for every existing vault. Confirm this is the migration story.
+- **Which violation kind.** Reuse `Disallowed`, or add a distinct kind (`Undeclared`?) so the two cases stay legible in the output? `Disallowed` currently carries the rule string `allowed in [...]`, which would read oddly for this case.
+- **Interaction with auto-update.** Inside a frozen scope, should `check`'s auto-update refuse to absorb a new field, rather than adding it and then flagging it? Probably yes, or the two features fight.
+- **Does `frozen` belong per-scope or is it the only policy?** If `[[scope]]` is going to exist, consider what else is directory-scoped (see TODO-0195 for cross-field rules, which want a similar home).
+
+### Interaction with TODO-0195
+
+Both this and [TODO-0195](TODO-0195.md) need the same missing thing: **a place in `mdvs.toml` for rules that are not field-local**. Today every rule hangs off exactly one `[[fields.field]]`. Directory policy and cross-field conditions are both cross-cutting. Design the container once, here, so 0195's rules can live in the same section rather than inventing a second syntax.
+
+### Documentation debt this closes
+
+`book/src/recipes/ci.md` currently documents the gap explicitly and offers a workaround:
+
+```bash
+mdvs check --no-update --output json | jq -e '.new_fields | length == 0'
+```
+
+That section, and the sentence "A schema-level 'freeze this directory' option does not exist yet", are what this TODO replaces.
+
+## Files
+
+- `crates/mdvs/src/schema/config.rs` — the `[[scope]]` section, plus a `MdvsToml::validate()` invariant
+- `crates/mdvs/src/cmd/check/validate.rs` — the undeclared-key pass (needs to iterate file keys, not just declared fields)
+- `crates/mdvs/src/output.rs` — violation kind, if a new one is added
+- `book/src/recipes/ci.md`, `book/src/configuration.md`, `book/src/concepts/validation.md` — docs

docs/spec/todos/TODO-0195.md

@@ -0,0 +1,79 @@
+---
+id: 195
+title: Cross-field rules — conditional requiredness, and the variant-type question
+status: todo
+priority: medium
+created: 2026-08-17
+depends_on: [194]
+blocks: []
+related: [194, 156]
+---
+
+# TODO-0195: Cross-field rules — conditional requiredness, and the variant-type question
+
+## Summary
+
+Every validation rule in mdvs hangs off exactly one `[[fields.field]]`. There is no way to say "`closed_date` is required when `status` is `closed`", nor to model a field whose shape depends on a discriminant (a Rust-style tagged enum). This TODO covers conditional requiredness as the near-term feature, and records the variant-type idea as its expensive endpoint.
+
+## Details
+
+### Part A — conditional requiredness (the buildable half)
+
+Two forms, in increasing cost:
+
+**Presence-based.** "If `a` is present, `b` and `c` are required." Maps directly onto JSON Schema's `dependentRequired`, which the `jsonschema` crate already supports. The violation maps onto the existing `ViolationKind::MissingRequired`. No new evaluation machinery.
+
+**Value-based.** "When `status == closed`, require `closure_reason`." Needs `if` / `then` / `else`. More expressive, and the one that pairs with Part B — but worse error messages, since a failing `if/then` reports at the subschema level rather than naming the triggering field.
+
+**The complement is as important as the rule.** Conditional *requiredness* alone is half the constraint. The other half is conditional *disallowal* — `closure_reason` should be meaningless, and rejected, when `status` is `open`. Required-when plus disallowed-when together are what make the constraint actually exhaustive. Ship both directions or the feature is a footgun.
+
+`schema/json_schema/validate.rs` currently hard-rejects `oneOf`, `$ref`, `if`/`then` and friends via the curated reject list. That gate needs to open selectively for whichever keywords this lands on — the point of the gate is to reject *unsupported* JSON Schema, not to freeze the supported set.
+
+### Part B — Rust-style variant types
+
+The idea: instead of a flat `status` with three categories, model substates with payloads —
+
+```yaml
+status:
+  closed:
+    outcome: completed
+    date: 2027-01-01
+```
+
+**The file format is not the obstacle.** YAML and TOML both represent tagged unions fine; the shape above is externally-tagged, and `{kind: closed, outcome: ..., date: ...}` is internally-tagged. These are exactly serde's enum representations. The obstacles are three deliberate mdvs decisions:
+
+1. **The schema gate rejects `oneOf`**, which a tagged union needs (`schema/json_schema/validate.rs`).
+2. **Storage assumes a fixed shape.** `index/storage.rs::transpose_to_storage_type` builds one nested Arrow Struct for the whole corpus; a variant means the shape varies per row. A sparse-struct encoding works around this — `status.kind`, `status.closed.date`, `status.rejected.reason`, all nullable, only the active variant populated — and stays SQL-queryable via `--where "status.kind = 'closed'"`. Arrow's native `Union` is the wrong tool: Lance and DataFusion support for it is thin.
+3. **Inference.** mdvs infers schemas from the corpus. Deriving "this key always has exactly one of N shapes" is far harder than deriving a scalar type, and `init --force` would likely destroy a hand-written variant declaration. Wave C already rejected `Array(Object{...})` and top-level `Object` on disk for related reasons (see [TODO-0156](TODO-0156.md)).
+
+**Part A is ~90% of Part B at ~10% of the cost.** The flat encoding of the example above is: `status` with `categories = ["open", "closed"]`, plus "when `status == closed`, require `closure_reason` (itself categorical) and `closed_date`", plus "when `status == open`, disallow both". Same guarantees, no storage change, queryable with plain SQL today. What is genuinely lost is the *type-level* guarantee — nothing structurally prevents someone declaring a fourth field that only makes sense for a variant nobody wrote a rule for.
+
+Recommendation: build Part A in both directions, live with it, and only revisit Part B if the flat encoding proves insufficient in practice rather than merely inelegant.
+
+### Where these rules live
+
+Cross-field rules do not fit `[[fields.field]]` — by construction they are about a *relationship between* fields. They need their own section, which is the same conclusion [TODO-0194](TODO-0194.md) reaches for directory policy. **Design the container once, in 0194, and put these rules in it.** Hence `depends_on: [194]`.
+
+Sketch, deliberately not settled:
+
+```toml
+[[rule]]
+when = { field = "status", equals = "closed" }
+require = ["closure_reason", "closed_date"]
+disallow = []
+```
+
+### Design questions
+
+- Presence-based only for v0, or value-based from the start? Value-based is what the motivating example needs, so presence-only may not be worth shipping alone.
+- How does a conditional rule interact with a field's own path-scoped `required` globs? Two mechanisms can now make the same field required; the violation output must stay legible about which one fired.
+- Error message quality. A raw `if/then` failure from the `jsonschema` crate will not name the triggering field. Mapping it back to something like "`closure_reason` is required when `status` is `closed`" likely means evaluating the condition ourselves rather than delegating wholesale.
+- Does inference ever propose these? Almost certainly not for v0 — the correlation heuristics are a research project. Hand-written only, and `init --force` must not silently drop them (today it clobbers hand-tuned constraints, which is already a known sharp edge).
+
+## Files
+
+- `crates/mdvs/src/schema/config.rs` — the rule section + `MdvsToml::validate()` invariants
+- `crates/mdvs/src/schema/json_schema/validate.rs` — open the gate for the chosen keywords
+- `crates/mdvs/src/schema/json_schema/to_canonical.rs` / `from_canonical.rs` — emit and reverse the rules
+- `crates/mdvs/src/cmd/check/` — evaluation + violation mapping
+- `book/src/concepts/validation.md`, `book/src/configuration.md` — docs

docs/spec/todos/TODO-0196.md

@@ -0,0 +1,96 @@
+---
+id: 196
+title: Duplicate `[[fields.field]]` names are silently accepted — last one wins
+status: todo
+priority: high
+created: 2026-08-20
+depends_on: []
+blocks: []
+related: [194, 195]
+---
+
+# TODO-0196: Duplicate `[[fields.field]]` names are silently accepted — last one wins
+
+## Summary
+
+Two `[[fields.field]]` entries with the same `name` are accepted by config load without warning. Downstream lookup is keyed by the bare name, so the later entry silently overwrites the earlier one and the first declaration is discarded. Reject duplicates at config load.
+
+## Details
+
+### Reproduction
+
+Verified against v0.8.4. Config declaring `status` twice, scoped to disjoint directories:
+
+```toml
+[[fields.field]]
+name = "status"
+type = "String"
+allowed = ["blog/**"]
+required = ["blog/**"]
+nullable = false
+
+[[fields.field]]
+name = "status"
+type = "Integer"
+allowed = ["projects/**"]
+required = ["projects/**"]
+nullable = false
+```
+
+With `blog/post.md` containing `status: draft` and `projects/p.md` containing `status: 3`:
+
+```
+Checked 2 files — 2 violation(s)
+
+status │ Wrong type  │ type Integer               │ blog/post.md (got String)
+status │ Not allowed │ allowed in ["projects/**"] │ blog/post.md
+```
+
+The `blog/**` declaration is gone. `blog/post.md` is judged against the `projects/**` entry — wrong type *and* wrong path — while `projects/p.md`, the file the surviving rule was never meant to cover alone, passes. Both violations are artifacts of the collapse, not of the vault.
+
+### Cause
+
+`cmd/check/validate.rs:43`:
+
+```rust
+let field_map: HashMap<&str, _> = config
+    .fields
+    .field
+    .iter()
+    .map(|f| (f.name.as_str(), f))
+    .collect();
+```
+
+`collect()` into a `HashMap` keeps the last value for a repeated key. `FieldValidators::build` (`cmd/check/field_meta.rs`) builds a `HashMap<String, Validator>` the same way, so the compiled validator collapses identically.
+
+`MdvsToml::validate()` has nine invariants (`schema/config.rs`); none concerns duplicate names. Invariant 8 covers *shape* conflicts (a name declared both as a leaf and as a parent of a dotted name) but not two declarations of the same leaf.
+
+### Fix
+
+Add **invariant 10**: no two `[[fields.field]]` entries may share a `name`. Error at config load, naming the field and pointing at the duplicate, in the style of the existing invariant messages.
+
+This is a strict improvement regardless of what happens with directory-scoped policy — it converts a silent wrong answer into a clear error, and it is a small, self-contained change.
+
+### Why the duplicate looks reasonable to write
+
+Someone reaching for this config is trying to express "the same field name means different things in different directories" — `status` as a workflow state under `projects/**` and as a publication state under `blog/**`. The config reads as though scoping should disambiguate, because `allowed` already scopes everything else about a field.
+
+It cannot work as written, for a reason that sits below validation: **field identity is the bare name, and that reaches into storage.** `index/storage.rs` transposes frontmatter into a nested Arrow Struct where `data.status` is one column with one type. Two declarations with two types cannot share it.
+
+So rejecting duplicates is correct, but it leaves the underlying need unmet. The proposed resolution, to be designed in [TODO-0194](TODO-0194.md):
+
+- **One name → one type**, enforced. Type is what storage constrains.
+- **Constraints may vary by scope.** `categories`, ranges, lengths, patterns and requiredness are validation-only, so they can differ per directory without touching the column.
+
+That covers the common real case, where the *values* diverge and the type does not. When the type genuinely differs, the fields are different things and should be named differently.
+
+Implementing invariant 10 now does not foreclose that: a future scoped-constraint syntax would attach constraint blocks to a single `[[fields.field]]` entry rather than repeating the entry, so "one entry per name" remains true either way.
+
+### Open question
+
+Should invariant 10 fire for names differing only by case, or by dotted-path normalisation? Field names are matched exactly elsewhere, so exact-match rejection is the consistent choice, but worth confirming no vault relies on near-duplicate names.
+
+## Files
+
+- `crates/mdvs/src/schema/config.rs` — invariant 10 in `MdvsToml::validate()`, plus a rejection test alongside the existing invariant tests
+- `crates/mdvs/src/cmd/check/validate.rs` — no change required once the invariant lands, but worth a comment noting the map assumes unique names

docs/spec/todos/index.md

@@ -195,3 +195,6 @@
 | [0191](TODO-0191.md) | Auto-rewrite array-field comparisons in `--where` (parser-based, with translation note) | done | medium | 2026-06-23 |
 | [0192](TODO-0192.md) | Don't persist the mock-embedder default to `mdvs.toml` | done | high | 2026-06-23 |
 | [0193](TODO-0193.md) | Support .mdx files — free validation, gated search-body stripping | todo | medium | 2026-07-06 |
+| [0194](TODO-0194.md) | Directory-scoped schema policy — freeze undeclared fields per path | todo | high | 2026-08-17 |
+| [0195](TODO-0195.md) | Cross-field rules — conditional requiredness, and the variant-type question | todo | medium | 2026-08-17 |
+| [0196](TODO-0196.md) | Duplicate `[[fields.field]]` names are silently accepted — last one wins | todo | high | 2026-08-20 |