Server-validated runtime knob catalog (internal.* settings schema)

[haasonsaas]

Context

Today, runtime tuning lives in MAESTRO_* env vars scattered across src/ and is documented mostly by reading source. Examples I can find quickly: MAESTRO_AUTOCOMPACT_PCT, MAESTRO_AUTOCOMPACT_ENABLED, MAESTRO_AUTOCOMPACT_MIN_MESSAGES, MAESTRO_DATABASE_TYPE, MAESTRO_DATABASE_FILE, MAESTRO_PLATFORM_BASE_URL, MAESTRO_EVALOPS_*, EVALOPS_FEATURE_FLAGS_PATH, etc.

Sourcegraph Amp's "Neo" exposes runtime knobs as a typed settings catalog (zod-validated). Examples observed in their bundle:

• internal.compactionThresholdPercent (0–100)
• internal.model (string OR Record<mode, string> for per-mode override)
• internal.oracleReasoningEffort (none | minimal | low | medium | high | xhigh)
• openai.speed (standard | fast)
• painter.model (gemini-3-pro-image | gpt-image-2)
• gemini.thinkingLevel (minimal | low | medium | high)
• agent.skipTitleGenerationIfMessageContains: string[]
• tools.disable: string[], tools.enable: string[]

Two consequences worth ours:

1. The schema is its own audit-able artifact. "What knobs even exist" stops being a code-archaeology question. For a governance/compliance buyer, this is high-leverage: every per-tenant deviation from defaults is now a typed, queryable record.
2. Per-org/per-user without redeploys. No env-var-redeploy cycle to change a threshold.

We have the storage already: db/schema.ts has users.settings: jsonb<UserSettings> and organizations.settings: jsonb<OrganizationSettings> columns sitting empty for this exact use case.

Scope

1. Define MaestroSettingsSchema in packages/contracts/src/ (TypeBox, matching our existing schema style). Initial keys to land — pick the highest-leverage knobs first:
   • Compaction: internal.compactionThresholdPercent, internal.compactionEnabled, internal.compactionMinMessages, internal.compactionKeepRecentTokens (also covered in Promote auto-compaction config to typed per-org/per-user settings #296)
   • Model selection: internal.model (string or per-mode record)
   • Tool gating: tools.disable, tools.enable (these are the equivalents of Amp's gating; for us they overlay the PlatformToolExecutionBridge classification)
   • Provider behavior: openai.speed, gemini.thinkingLevel
   • Agent UX: agent.skipTitleGenerationIfMessageContains
2. Resolution order: org settings → user settings → env var (compat) → built-in default. Document in docs/design/SETTINGS.md.
3. Audit hook: emit a settings_changed event with (org, user, key, old, new, by) tuple — same attribution model as SERVER_REQUEST_RESOLVED_BY_* in the headless proto.
4. Surface to clients: extend the headless ClientCapabilities / handshake to advertise the keys the runtime understands, so a TUI/web/IDE can render the correct picker without hardcoding.
5. CLI: maestro settings get|set|describe for read/write/schema introspection.

Non-goals

• Not migrating every env var. Pick the highest-impact and the ones already covered by Promote auto-compaction config to typed per-org/per-user settings #296. Other env vars get follow-up tickets per surface area.
• No settings UI in this slice. CLI + API only.
• Not building feature flags — EVALOPS_FEATURE_FLAGS_PATH is platform-managed, separate concern.

Acceptance criteria

• The settings schema is a single typed source of truth, importable from @evalops/maestro-contracts.
• A platform admin can POST /api/orgs/:id/settings { "internal.compactionThresholdPercent": 75 } and the next session in that org honors 75.
• Env vars still work for users without org/user settings (back-compat).
• An audit consumer can subscribe to settings_changed and reconstruct the per-tenant settings history.

Why now

The internal.* pattern is a free upgrade for the audit story (the schema is the catalog), and it stops the spread of MAESTRO_*_PCT/MAESTRO_*_ENABLED env vars before more land. It also unblocks #296, where the auto-compaction knob wants a real home.

Reference

Pattern from Sourcegraph Amp's "Neo" rebuild (2026-05-06), settings keys extracted from the zod schema in their published @sourcegraph/amp bundle.