Skip to content

docs(orchestrator): pin and document channel_overrides REPLACE semantics - #146

Open
eldonm wants to merge 1 commit into
mainfrom
docs/channel-overrides-replace-semantics
Open

docs(orchestrator): pin and document channel_overrides REPLACE semantics#146
eldonm wants to merge 1 commit into
mainfrom
docs/channel-overrides-replace-semantics

Conversation

@eldonm

@eldonm eldonm commented Aug 7, 2026

Copy link
Copy Markdown
Member

Summary

Two properties of channel_overrides account for most "channel overrides don't work" reports. Both are by design, neither announces itself, and neither was stated anywhere an operator reads.

  1. List knobs REPLACE, they do not merge. pinned_tools, denied_tools and skill_only_tools inside a channel block stand in for the action-level list on that channel — anything still wanted must be repeated there. This is why a config that works with channel_overrides commented out can stop working once it is uncommented: the block is not adding to the global list, it is standing in for it. An explicit [] means "none here", not "fall back".
  2. Keys match visitor.channel exactly. No prefixes, no aliases — a block for whatsapp does not cover whatsapp_call. Both ship in the example app. A mis-keyed block no-ops with nothing logged.

Related issues

Follow-up to a field report that skill_only_tools and channel_overrides "were not working". Investigation found no code defect — #142's 7 resolution tests pass against unmodified main — so this closes the documentation and coverage gap that let a config error read as a product bug.

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change
  • Documentation
  • Chore / tooling / CI — adds a wire-tier test

Changes

File Change
tests/wire/test_channel_overrides_replace.py new — 5 tests asserting resolution from YAML through a real graph
docs/ORCHESTRATOR.md new "Per-channel overrides" section under Configuration
examples/.../orchestrator_agent/agent.yaml both rules stated at channel_overrides and at skill_only_tools

Checklist

  • I read CONTRIBUTING.md and the local tests/CLAUDE.md.
  • pre-commit run --all-files passes.
  • pytest tests/ passes — 0 failures.
  • Bug fixes cite file:line in the description.
  • Docs updated — that is most of the PR. No CHANGELOG.md entry: no behaviour changes.
  • No ADR was modified in place.

Notes for reviewers

Why a wire test rather than another orchestrator unit test. The existing suites — including #142's — set ex.channel_overrides = {...} in Python. That pins the resolver, but never exercises the trip a real deployment takes: agent.yaml → bootstrap → persisted attribute → resolution. A loader that dropped or reshaped the nested block would leave every one of those tests green while the feature did nothing in production. This tier boots a real graph and reads the orchestrator back out of the database, which is exactly the gap tests/CLAUDE.md §3.1 describes.

Mutation-checking caught a bug in my own test, which is worth reading. Per the wire tier's convention I broke the behaviour deliberately:

Mutation Result
list knobs merge instead of replace 3 tests fail ✅
prefix matching instead of exact passed — the test proved nothing

The exact-match test asserted on whatsapp_call, which has its own block, so the exact lookup succeeded and the prefix fallback was never reached. It now asserts on whatsapp_media — a prefix-sharing channel with no block of its own, the only shape that detects it — and fails under that mutation with assert ['whatsapp__send_flow'] == ['kb__search'].

That failure mode is the same one this PR documents: something that looks correct and silently does nothing.

Relationship to #142. Complementary, no overlap. #142 pins resolution semantics in Python and adds a jvagent validate advisory for the sibling-channel case; this pins the YAML→attribute path and writes the rules down. #142's advisory would have turned this particular report into a startup warning, so it is worth landing.

Steps to test

pytest tests/wire/test_channel_overrides_replace.py -q

To see the trap directly, add to an agent's orchestrator context:

      pinned_tools: ["kb__search"]
      channel_overrides:
        whatsapp:
          pinned_tools: ["whatsapp__send_flow"]   # kb__search is now unpinned on whatsapp

Two properties of channel_overrides account for most "channel overrides
don't work" reports. Both are by design and neither announces itself.

List knobs REPLACE rather than merge. pinned_tools, denied_tools and
skill_only_tools inside a channel block stand in for the action-level
list on that channel, so anything still wanted must be repeated there.
That is why a config which works with the block commented out can stop
working once it is uncommented: the block is not adding to the global
list, it is standing in for it. An explicit [] means "none here", not
"fall back".

Keys match visitor.channel exactly. No prefixes, no aliases, so a block
written for whatsapp does not cover whatsapp_call -- voice is its own
channel string and both ship in the example app. A mis-keyed block
no-ops with nothing logged.

Document both in ORCHESTRATOR.md under Configuration, with a table
mapping each assembly warning to the failure it indicates, and state
them at the two places in the example agent.yaml an operator actually
reads -- the channel_overrides block and skill_only_tools. The example
previously showed pinned_tools inside a channel block without noting
that it replaces.

Add tests/wire/test_channel_overrides_replace.py. The orchestrator
suites set ex.channel_overrides in Python, which pins the resolver but
never exercises agent.yaml -> bootstrap -> persisted attribute ->
resolution; a loader that dropped or reshaped the nested block would
leave them all green while the feature did nothing. These boot a real
graph and read the orchestrator back out of the database.

Mutation-checked per the wire tier's convention, which caught a hole in
the first draft: the exact-match test asserted on whatsapp_call, which
has its own block, so the exact lookup succeeded and a prefix-matching
mutation never reached the fallback path -- the test passed while
proving nothing. It now asserts on whatsapp_media, a prefix-sharing
channel with no block of its own, and fails under that mutation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant