feat: unify the listing field vocabulary across CLI and MCP - #79
Merged
Conversation
Listing formats could only show five fields, all from the node index
entry, so metadata that is already queryable -- type, status, and the
rest -- could not be displayed at all. Single letters cannot address an
arbitrary metadata key, so extend the format language rather than adding
verbs.
Field selectors now reuse the query expression vocabulary. In field
position a bare word names a metadata key, a leading dot names a
statistics field, and tags is reserved for the tag list; id and title
are intrinsics. Formats frame selectors as %{...}, and the legacy verbs
%i %d %c %a %t remain as aliases.
Replace the chain of whole-line strings.Replace calls with a single
left-to-right scanner. The old chain re-expanded values that happened to
contain a verb, which adding selectors would have made far worse. Three
fixes fall out of it:
- %% renders a literal percent. It was documented in eight places and
implemented in none.
- Zero timestamps render empty instead of 0001-01-01T00:00:00Z, which
parses as a real date and silently corrupts downstream sorting.
- Control characters in expanded values collapse to spaces. Each
rendered string is one output line, and .lead routinely contains
newlines.
The three index timestamps resolve from the index entry rather than
stats.json, mirroring resolveStatsCompare, so a displayed value agrees
with the same predicate in a query and the default format stays free of
per-node reads. Only formats naming metadata or a non-timestamp stat
perform I/O, and that pass is wrapped in one keg read boundary: the
boundary is exclusive and re-entrant through the context, so without it
a listing would acquire and release it twice per node.
The format vocabulary was documented inconsistently and, on MCP, incorrectly. Three of the five commands sharing the formatter documented no placeholders at all, links and backlinks stated a default they did not use, and every MCP schema advertised %i, %d, and %t while the implementation had long supported %c and %a as well, so two working verbs were undiscoverable to agents. Share one help block across list, grep, tags, links, and backlinks, and give each a --format completer, which none had. Extract the two byte-identical --query completers into one helper so they cannot drift again. Point the five MCP schemas at the same vocabulary. The descriptions must be tag literals, so a test asserts the generated schema still matches keg.FormatVocabularyDescription, and a second test guards the reflector rule that a description beginning with a token containing "=" panics at server construction -- which documenting a type= or status= selector would otherwise trip. Add the cross-surface conformance cases the vocabulary needs: one parity case per selector kind, so CLI and MCP cannot disagree on what a field is named or resolves to. Document the vocabulary in docs/output-formats.md, and fill the gap in docs/query-expressions.md, which described neither the statistics fields nor the comparison operators.
Rendering a metadata column read each node individually, and on a remote keg every read is an HTTP round trip. Measured against a real hub, a 610-node listing took ~49s where the same listing without metadata took 0.21s -- one round trip versus 610. The cause is architectural. Tap.List assembled the listing on the client: fetch entries, sort, offset, limit, then enrich row by row. That inverts the rule the hub already states, that handlers are the remote Keg surface and the server-side LocalKeg owns orchestration. The N+1 is what client-side orchestration costs once a row needs more than the index carries. Add ListView, which resolves a whole listing page server-side: filter by query, order, page, then project the requested field selectors. Ordering the operations that way means a listing that displays metadata reads only the rows it returns, not every node in the keg. Sorting by a metadata key still resolves a key per matching node, since the whole set must be ordered before it can be paged. Field resolution moves to keg.FieldValue so the client renderer and the server projection cannot disagree about what a selector means, and stays best-effort per row: listings render from an index that is allowed to drift, so one unreadable node yields empty values rather than failing the page. A hub without the route answers 404, reported as ErrListViewUnsupported so Tap.List degrades to assembling the listing itself instead of failing.
The help has always documented the default format as "%i\t%d\t%t", but a shell does not expand \t inside double quotes, so the obvious command produced a literal backslash and a t between columns. Getting tabs required $'...' quoting, which is not discoverable from the help text that suggests \t in the first place. Interpret \t, \n, \r, and \\ in the format string. An unrecognised escape passes through untouched, mirroring the existing rule for unrecognised percent verbs, so a Windows-style path in a template still renders. A real tab arriving from $'...' quoting or a script is unaffected.
A keg whose nodes are distinguished by type and subkind had no way to say so: every caller had to pass --format, and the hosted node list hardcoded its own columns, so the two surfaces disagreed about what a keg looks like. Add listFields to the keg config. It uses the same selector vocabulary as --format and query expressions, and resolves as flag, then keg config, then the built-in default. One setting will drive both `tap list` and the Hub node list. Selectors are validated when the config is parsed rather than when a listing renders, so a typo is reported while editing instead of showing a silently blank column later. It lives on the config rather than on an index entry because normalizeIndexes rebuilds the index list from SystemIndexEntries on every parse, which carries only file and summary -- a field declared on the default nodes.tsv index would be discarded on the next read.
Showing or sorting by a metadata column read one node at a time. A listing that sorts needs a key for every matching node, not just the returned page, so on a 610-node keg that was 610 reads -- and on the hosted backend roughly 1220 queries, because each read also checks existence first. Add RepositoryBatchRead as an optional capability, detected by type assertion in the same way SupportsConcurrentAccess already is. A backend that can answer a whole set at once implements it; a plain filesystem repository does not and keeps reading node by node, so local kegs behave exactly as before. listView now loads values for the page as a whole after paging, and the sort path loads keys for the matched set in one go. A metadata listing costs one batch; sorting then projecting costs two, regardless of keg size. Listings naming only intrinsics and index timestamps still read nothing at all. Reads stay best-effort: a node the index names but the repository cannot produce contributes empty values rather than failing the listing.
The hosted node list offers a plain title search alongside the expression filter, for the common "I half-remember the name" case. It cannot be applied by the caller, because the server pages the result: filtering after paging would report the wrong match count and show the wrong slice. Add TitleContains to ListView, applied after the query and before sorting and paging. It costs nothing, since titles are already carried by the index.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Listing formats could only show five fields, all from the node index entry, so metadata that is already queryable —
type,status, and the rest — could not be displayed at all. Single letters cannot address an arbitrary metadata key, so this extends the format language rather than adding verbs.Field selectors now reuse the query expression vocabulary. In field position a bare word names a metadata key, a leading dot names a statistics field, and
tagsis reserved for the tag list;idandtitleare intrinsics. Formats frame selectors as%{...}, and the legacy verbs%i %d %c %a %tremain as aliases.Why
The same field set had three naming conventions and the surfaces disagreed about which fields even existed:
%i,%d,%twhile the implementation had also supported%cand%afor some time — two working verbs undiscoverable to agents.%%was documented in eight places and implemented in none.linksandbacklinksstated a default they did not use.Notable changes
Single-pass scanner. The old expander was a chain of whole-line
strings.Replacecalls, so an expanded value containing a verb was expanded again — adding selectors would have made that far worse. A left-to-right scan makes re-expansion structurally impossible, and three fixes fall out of it:%%renders a literal percent.0001-01-01T00:00:00Z, which parses as a real date and silently corrupts downstream sorting..leadroutinely contains newlines.Cost is opt-in. The three index timestamps resolve from the index entry rather than
stats.json, mirroringresolveStatsCompare, so a displayed value agrees with the same predicate in a query and the default format stays free of per-node reads. Only formats naming metadata or a non-timestamp stat perform I/O.One read boundary. A local keg's read boundary is exclusive and every per-node read takes it, so the enrichment pass is wrapped in a single
keg.WithReadBoundary. Without it a listing would acquire and release that lock twice per node and block every other process on the keg.Behaviour changes
"%%i"now renders%irather than%followed by the node id. No test or fixture used%%; this is the behaviour the help has always documented.0001-01-01T00:00:00Z.%{...}selector is an error. A bare%Xstill passes through as literal text, so no format string that works today starts failing.Verification
go test ./internal/apidocgo test ./...go test -race ./pkg/keg ./pkg/tapper ./pkg/paritygo test ./pkg/cli/... -run Completiongo vet ./...(one pre-existingtap_schema.go:184finding, present on cleanmain)task render-integrations— no drift%{tags}comma-joins,%%renders one percent, and an unknown stats field lists the valid ones.New cross-surface conformance cases in
pkg/paritycover one case per selector kind, so CLI and MCP cannot drift on what a field is named or resolves to.