diff --git a/CHANGELOG.md b/CHANGELOG.md index fcf861cb..681a6792 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -111,6 +111,28 @@ release below. test queries recognize JavaScript/TypeScript `.test`/`.spec` and `__tests__` conventions and import relationships without claiming execution coverage. +- Graphify search readiness now comes from the installed query reader, not + from the lifecycle alone (#1029). `context-graph build`, `refresh`, `status` + and `connection-status` read the published generation the same way a query + does. They report `search` and a metadata-only `query_reader` verdict beside + the generation state, and exit non-zero or withhold `authorization` unless + the reader can consume the generation. So `status`, `connection-status` and + a real query agree. A known provider/reader mismatch, such as a `doc_ref` + generation meeting the `v1.4.2` reader, reports `reader_incompatible`. Its + remediation names the installed Code Mower and the release that reads it, + and does not call the generation `unreadable`. A generation from an + unreviewed provider gets a rebuild-or-upgrade remediation. Reviewed means + the exact distribution and version (`graphifyy==0.9.58`), not the version + alone. Unknown + node types from the reviewed release still fail closed, and no verdict + carries graph content, targets or local paths. Query summaries now report + `generation_completeness` and `query_completeness` separately. A bounded + answer that stopped at a budget, depth or document limit, or crossed an + ambiguous relationship, stays `available` and usable, discloses + `provider_has_more`, `unresolved_entities` or `document_limit`, and does not + mark the generation incomplete. An answer whose only omission is + `unresolved_entities` is `partial` too. Intended for `v1.5.0` together with #1007. + ## 1.4.2 — published - `code-mower board service` manages a persistent local Board: render a diff --git a/docs/context-graph-lifecycle.md b/docs/context-graph-lifecycle.md index eb1c67d0..67f7554a 100644 --- a/docs/context-graph-lifecycle.md +++ b/docs/context-graph-lifecycle.md @@ -585,6 +585,33 @@ written. `partial` exists because of the requeue defect recorded in the evaluation: a fast incremental repeat is not proof that the graph is complete. +### Search readiness is a separate verdict + +The states above describe the *generation*: present, intact, complete, and +bound to the revision. They do not say whether the installed query reader can +consume it. A generation can be `current` and complete and still carry a +provider node type the installed reader predates. The `v1.4.2` reader, for +example, refuses the `doc_ref` nodes the pinned extractor emits. Since #1029, +`build`, `refresh`, `status` and `connection-status` run the query's own read +over the published generation and report the result beside the state, never in +place of it: + +| `search` | `query_reader.reader` | Meaning | Next action | +| --- | --- | --- | --- | +| `available` | `compatible` | The reader consumed the generation; a query can answer from it. | none | +| `unavailable` | `not_checked` | The generation is not usable; `reason` repeats its state. | build or refresh | +| `unavailable` | `not_checked` | `connection-status` only: the connection is disconnected (`reason: disconnected`), so the graph is not read. | reconnect | +| `unavailable` | `incompatible` | Known provider/reader mismatch. `remediation` names the installed Code Mower, the generation's provider release and the required Code Mower release, when one is known. | upgrade Code Mower, or rebuild with a reviewed provider release | +| `unavailable` | `unreadable` | The reader has no account of what the generation contains, for example an unknown node type from the reviewed provider release. Fails closed. | treat as a defect; do not work around it | + +`status` and `build` exit zero only when the generation is usable **and** +`search` is `available`. `connection-status` reports `authorization: +available` under the same condition. The verdict is metadata only: fixed +reasons, release numbers and the Code Mower vocabulary for a known node type. +It never contains graph content, a query target, an unknown type's spelling, or +a local path. A reader upgrade does not require a rebuild. The published +generation stays valid, and the new reader reads it as it is. + ## Commands ``` @@ -595,8 +622,10 @@ code-mower context-graph remove [--show-local-paths] code-mower context-graph doctor [--pin-file PIN] ``` -`status` exits non-zero when the graph is not usable, so a script can branch on -it. `build` and `refresh` do the same, and for the same reason: a provider that +`status` exits non-zero when the graph is not usable or the installed reader +cannot search it (see +[Search readiness is a separate verdict](#search-readiness-is-a-separate-verdict)), +so a script can branch on it. `build` and `refresh` do the same, and for the same reason: a provider that admitted an incomplete run has published a generation `status` will call `partial` and refuse, so the build prints `partial` and exits non-zero rather than describing it as `current` for as long as it takes to ask again. diff --git a/docs/context-graph-queries.md b/docs/context-graph-queries.md index 63de93fc..c13eeaba 100644 --- a/docs/context-graph-queries.md +++ b/docs/context-graph-queries.md @@ -206,6 +206,17 @@ arrived after `v1.4.2` was published and are on `main` for the next appropriate release. In the published package a `doc_ref` node is an unknown type and the JavaScript/TypeScript conventions are not recognized. +From #1029, a reader that meets `doc_ref` without supporting it reports that +node type as a known provider/reader mismatch and names Code Mower `1.5.0` as +the release that reads it. It does not call the generation `unreadable`. Other +unknown node types stay unexplained refusals. When the generation was built by +a provider this reader was not reviewed against, an unknown node type is +reported as `reader_incompatible` with a rebuild-or-upgrade remediation. +Reviewed means an exact requirement, distribution and version together +(`graphifyy==0.9.58` today, the distribution compared in its normalized +spelling), so another distribution published at `0.9.58` is not reviewed. +Either way, search is unavailable until the reader and the generation agree. + Each traversal is symbol-first: a target resolves to the symbols carrying that name, and only a target that names no symbol at all is read as a path. Each is breadth-first over adjacency sorted by the full relationship identity described @@ -346,6 +357,22 @@ Confidence maps the provider's own qualification onto the contract's vocabulary: | `inferred` | `inferred` | derived, not read directly | | `ambiguous` | `unknown` | resolved to more than one candidate | +A partial packet is still an available answer. When a relationship budget, the +depth limit, the seed bound, or the policy's document limit stops a query, or +the query crosses an ambiguous relationship or leaves a target unresolved, the +status stays `available` and +`dependent_work` stays `usable`. The omissions say what was left out and why. +The query summary reports two completeness claims separately, so a bounded +answer is never read as a broken build: + +| Summary field | Describes | +| --- | --- | +| `generation_completeness` | the published build's own `complete`/`partial`, from its manifest | +| `query_completeness` | this answer: `partial` whenever an omission above applies, including `unresolved_entities` alone | + +`completeness` keeps its earlier meaning, the packet's completeness, and +equals `query_completeness`. + `ambiguous` also raises `unresolved_entities` on the packet, so a recipient sees the uncertainty at the packet level and not only per document. A target name that matches more than one definition does the same. @@ -376,6 +403,13 @@ feature. | `required: true` | `required_unavailable` | `paused` | 1 | | `required: false` | `optional_unavailable` | `usable` | 0 | +A usable generation that the installed reader cannot consume follows the same +table. Its `reason` is `reader_incompatible` when the mismatch is a known one, +and the summary carries the same `remediation` and `next_action` that +`status` and `connection-status` report. It is `unreadable` when the reader +has no account of the content, which stays closed. See +[search readiness](context-graph-lifecycle.md#search-readiness-is-a-separate-verdict). + The words match `context_prepare`, so a caller branches on one vocabulary. Optional unavailable context is not a degraded answer — there is no packet at all, and the next action is to carry on with ordinary repository tools. diff --git a/docs/current-state-and-roadmap.md b/docs/current-state-and-roadmap.md index 235a545d..f34efd8b 100644 --- a/docs/current-state-and-roadmap.md +++ b/docs/current-state-and-roadmap.md @@ -320,7 +320,11 @@ release order. Merged fixes count as on main until a later published package is verified to contain them; #935/#973 and the phase-3 Board PRs are now verified in the published `v1.4.2` artifact, while the merged Graphify compatibility fixes in [PR #1007](https://github.com/codemower-ai/code-mower/pull/1007) are on -main awaiting the next appropriate release. +main awaiting the next appropriate release. They are intended for `v1.5.0` +together with the #1029 search-readiness check. That check makes `status` and +`connection-status` report `search` from the installed query reader, so a +current generation the reader cannot consume is reported as a reader mismatch +with an upgrade action rather than as searchable. Each child issue should produce one reviewable PR with one branch writer, independent current-head review, the normal gate, and package-level validation. diff --git a/docs/graphify-setup.md b/docs/graphify-setup.md index f88efc0b..1f3c45f6 100644 --- a/docs/graphify-setup.md +++ b/docs/graphify-setup.md @@ -167,6 +167,28 @@ code-mower context-graph status --json it. A provider run that admitted an incomplete census publishes a generation `status` calls `partial` and refuses, rather than describing it as `current`. +The rest of this step, up to step 4, is **post-`v1.4.2`** and describes +current `main` (#1029, intended for `v1.5.0`). `status` also reports `search`: +whether the installed query reader can consume the generation. A `current` +generation with `search: unavailable` exits non-zero. Read +`query_reader.next_action`: + +- `reader: incompatible` is a known provider/reader mismatch. The + `remediation` block names your installed Code Mower and the release that + reads the generation. Upgrade Code Mower; the generation needs no rebuild. + If it names an unreviewed provider instead (another release, or another + distribution at the same version), refresh with the pinned + `graphifyy` `0.9.58`. +- `reader: unreadable` means the generation holds something no reader has an + account of. It fails closed on purpose; report it rather than working around + it. + +The published `v1.4.2` package has no such check. There, `status` and +`connection-status` can report a usable graph and `search: available` while +the first dependency query fails with `unreadable` on a generation containing +`doc_ref` nodes. That query failure is this mismatch; upgrading to the release +containing #1007 and #1029 resolves it. + **4. Register the graph as a local context connection.** ```bash @@ -231,8 +253,8 @@ ramp-up above -- acquisition, the separate contained offline build, and steps 1 through 7 -- describe that published package. The paragraphs above that are explicitly marked post-`v1.4.2` describe current `main` instead: the language-extras and runtime-ownership paragraphs under -[Separate acquisition environment](#separate-acquisition-environment) are the -only ones so marked today. +[Separate acquisition environment](#separate-acquisition-environment) and the +search-readiness text in step 3 are the only ones so marked today. [PR #1007](https://github.com/codemower-ai/code-mower/pull/1007) has since merged to `main` with further real-pilot compatibility fixes: a bounded 16 MiB diff --git a/src/code_mower/context_graph_command.py b/src/code_mower/context_graph_command.py index 58eb3ee4..9032a8a4 100644 --- a/src/code_mower/context_graph_command.py +++ b/src/code_mower/context_graph_command.py @@ -114,6 +114,21 @@ def _write_packet(destination: Path, packet: dict) -> None: raise +def _readiness(repository: Path, root: Path | None, report: lifecycle.GenerationStatus) -> dict: + """The query reader's verdict on the published generation, metadata only.""" + return query.search_readiness(lifecycle.GraphStateRoot(repository, root=root), report) + + +def _search_text(readiness: dict) -> str: + """The readiness lines appended to the lifecycle's own status text.""" + lines = [f" search: {readiness['search']} (query reader: {readiness['reader']})"] + if readiness.get("detail") and readiness["reader"] != "not_checked": + lines.append(f" reader: {readiness['detail']}") + if readiness.get("next_action"): + lines.append(f" next: {readiness['next_action']}") + return "\n".join(lines) + "\n" + + def _emit(payload: dict, *, as_json: bool, text: str) -> None: if as_json: print(json.dumps(payload, indent=2, sort_keys=True)) @@ -210,12 +225,19 @@ def main(argv=None) -> int: manifest=manifest, detail="" if complete else "local graph build was incomplete; refresh it", ) - summary = {"status": "published", "usable": published.usable, **manifest.shareable_summary()} - _emit(summary, as_json=args.json, text=lifecycle.render_status_text(published)) - # Publishing an unusable generation is a reportable condition, not - # a crash: exit non-zero for the same reason ``status`` does, so a - # script does not have to re-ask to find out what it just built. - return 0 if published.usable else 1 + # Asked of the generation just published, not of whatever ``status`` + # would find later: the manifest is the one this build wrote. + readiness = _readiness(args.repo_path, args.state_dir, published) + summary = {"status": "published", "usable": published.usable, **manifest.shareable_summary(), + "search": readiness["search"], "query_reader": readiness} + _emit(summary, as_json=args.json, + text=lifecycle.render_status_text(published) + _search_text(readiness)) + # Publishing an unusable or unsearchable generation is a reportable + # condition, not a crash: exit non-zero for the same reason + # ``status`` does, so a script does not have to re-ask to find out + # what it just built. + searchable = readiness["search"] == query.SEARCH_AVAILABLE + return 0 if published.usable and searchable else 1 if args.command == "status": report = lifecycle.graph_status( args.repo_path, @@ -223,10 +245,19 @@ def main(argv=None) -> int: revision=args.revision, require_complete=not args.allow_partial, ) - _emit(report.shareable_summary(), as_json=args.json, text=lifecycle.render_status_text(report)) - # A non-current graph is a normal, reportable condition, not a - # command failure; exit 1 so a script can branch on usability. - return 0 if report.usable else 1 + # ``state``/``usable`` stay the generation's own verdict -- current, + # complete, bound to this revision -- and ``search`` is whether the + # installed reader can answer from it. They are different facts, so + # they are different fields; the exit code needs both. + readiness = _readiness(args.repo_path, args.state_dir, report) + summary = {**report.shareable_summary(), "search": readiness["search"], + "query_reader": readiness} + _emit(summary, as_json=args.json, + text=lifecycle.render_status_text(report) + _search_text(readiness)) + # A non-current or unsearchable graph is a normal, reportable + # condition, not a command failure; exit 1 so a script can branch. + searchable = readiness["search"] == query.SEARCH_AVAILABLE + return 0 if report.usable and searchable else 1 if args.command == "query": authorization = _load_authorization(args.authorization) outcome = query.graph_context( diff --git a/src/code_mower/context_graph_connection.py b/src/code_mower/context_graph_connection.py index 8f50af15..e1edafdf 100644 --- a/src/code_mower/context_graph_connection.py +++ b/src/code_mower/context_graph_connection.py @@ -97,9 +97,15 @@ def saved_state(value: Any, name: str) -> dict[str, Any]: ("repository_root", "repositories", "recipients")})} -def _summary(state: Mapping[str, Any]) -> dict[str, Any]: +def _summary(state: Mapping[str, Any], *, search: str = "available") -> dict[str, Any]: + """The connection's shape. ``search`` is the provider capability by default. + + ``connect`` and ``disconnect`` report the capability the connection offers; + ``status`` passes the observed readiness instead, because that is the one + report an operator reads to decide whether a query will work now. + """ return {"schema": "code_mower.contextConnectionSummary.v1", "provider": PROVIDER, - "kind": CONNECTION_KIND, "status": state["state"], "search": "available", + "kind": CONNECTION_KIND, "status": state["state"], "search": search, "memory": "unavailable", "credential_storage": "none"} @@ -154,10 +160,23 @@ def status(store: ContextStore, name: str, *, root: Path | None = None, """Report the connection and the graph behind it, without minting evidence.""" with store.locked(name) as locked: state = saved_state(locked.read(), name) - report = lifecycle.graph_status(Path(state["repository_root"]), root=root, revision=revision) - return {**_summary(state), "graph": report.shareable_summary(), - "authorization": "available" if state["state"] == "verified" and report.usable - else "unavailable"} + repository = Path(state["repository_root"]) + report = lifecycle.graph_status(repository, root=root, revision=revision) + # The query's own read, not the lifecycle's verdict alone: a current, + # complete generation this reader cannot consume must not be reported as + # searchable and then fail on the first question asked of it. A + # disconnected connection cannot search whatever the reader says, so the + # graph artifact is not opened for it. + verified = state["state"] == "verified" + if verified: + readiness = query.search_readiness(lifecycle.GraphStateRoot(repository, root=root), report) + else: + readiness = query.reader_not_checked("disconnected") + search = readiness["search"] + return {**_summary(state, search=search), "graph": report.shareable_summary(), + "query_reader": readiness, + "authorization": "available" if verified and report.usable + and search == query.SEARCH_AVAILABLE else "unavailable"} def _published(state: Mapping[str, Any], *, root: Path | None, diff --git a/src/code_mower/context_graph_query.py b/src/code_mower/context_graph_query.py index 646a4b16..b73c5da5 100644 --- a/src/code_mower/context_graph_query.py +++ b/src/code_mower/context_graph_query.py @@ -46,6 +46,7 @@ from pathlib import Path from typing import Any, Iterable, Mapping, Sequence +from . import __version__ as CODE_MOWER_VERSION from . import context_graph_lifecycle as lifecycle from .context_contract import ( CAPABILITY_VERSION, @@ -146,6 +147,30 @@ GRAPH_FILE_TYPES = frozenset({"code", "document", "paper", "image", "rationale", "concept", "doc_ref"}) CODE_FILE_TYPE = "code" +#: Provider node types whose reader support arrived after the first Graphify +#: release, and the Code Mower release whose reader first consumed each. A +#: reader that meets one of these without supporting it is a *known* +#: provider/reader mismatch with a concrete fix -- install that release -- and +#: says so instead of calling a valid generation ``unreadable``. Every other +#: unsupported type stays an unexplained refusal: fail closed, name nothing. +KNOWN_PROVIDER_FILE_TYPES: Mapping[str, str] = {"doc_ref": "1.5.0"} + +#: The provider releases this reader was reviewed against, as exact +#: requirements: distribution and version together, the distribution in its +#: normalized spelling. A version alone would let any distribution published at +#: that number inherit the review. A generation whose manifest names another +#: provider and then fails to read is a provider/reader mismatch the operator +#: can resolve by rebuilding with a reviewed release or installing a Code Mower +#: whose reader covers theirs. +READER_PROVIDERS = frozenset({"graphifyy==0.9.58"}) + + +def _reviewed_provider(provider: Mapping[str, Any]) -> bool: + distribution, version = provider.get("distribution"), provider.get("version") + if not isinstance(distribution, str) or not isinstance(version, str): + return False + return f"{lifecycle._normalized_distribution(distribution)}=={version.strip()}" in READER_PROVIDERS + #: The pinned validator's ``VALID_CONFIDENCES``, lowercased. The provider #: writes these uppercase; the packet contract's vocabulary is lowercase, and #: this is the whole of the difference. @@ -474,6 +499,50 @@ def seeds(self, target: str) -> tuple[GraphNode, ...]: return self.seed_matches(target)[0] +class UnsupportedNodeType(ContextError): + """A node type no Code Mower reader is known to read. Always fails closed.""" + + +class ReaderIncompatible(ContextError): + """A valid generation this installed reader cannot consume, and how to fix it. + + Carries only vocabulary this module already owns -- a type from + ``KNOWN_PROVIDER_FILE_TYPES``, release numbers from a validated manifest and + this package -- so its remediation can be shown to anyone who may see a + status report. Nothing from the graph's content, and no local path. + """ + + def __init__(self, message: str, *, minimum_reader: str | None = None, + node_type: str | None = None, provider: str | None = None) -> None: + super().__init__(message) + self.minimum_reader = minimum_reader + self.node_type = node_type + self.provider = provider + + def remediation(self) -> dict[str, Any]: + reviewed = ", ".join(sorted(READER_PROVIDERS)) + value: dict[str, Any] = { + "installed_code_mower": CODE_MOWER_VERSION, + "reader_providers": sorted(READER_PROVIDERS), + } + if self.provider is not None: + value["generation_provider"] = self.provider + if self.minimum_reader is not None: + value["node_type"] = self.node_type + value["required_code_mower"] = self.minimum_reader + value["action"] = ( + f"Upgrade Code Mower to {self.minimum_reader} or later (installed " + f"{CODE_MOWER_VERSION}); the published generation is valid and needs no rebuild." + ) + else: + value["action"] = ( + f"Refresh the graph with a provider release this reader was reviewed against " + f"({reviewed}), or upgrade Code Mower (installed {CODE_MOWER_VERSION}) to a " + "release whose reader supports the generation's provider." + ) + return value + + def _required(value: Any, fields: Sequence[str], *, what: str) -> Mapping[str, Any]: """A provider record with every field its own validator requires. @@ -608,7 +677,15 @@ def _node(value: Any) -> _ParsedNode: # raises TypeError out of a reader whose only failure is ``ContextError``. file_type = _text(record["file_type"], maximum=64) if file_type not in GRAPH_FILE_TYPES: - raise ContextError("unsupported local graph node file type") + # Only a type from the known table is ever named back: an unknown one + # is provider output, and its spelling is not metadata. + if file_type in KNOWN_PROVIDER_FILE_TYPES: + raise ReaderIncompatible( + "local graph carries a provider node type this Code Mower reader predates", + minimum_reader=KNOWN_PROVIDER_FILE_TYPES[file_type], + node_type=file_type, + ) + raise UnsupportedNodeType("unsupported local graph node file type") identifier = _text(record["id"], maximum=512) if file_type != CODE_FILE_TYPE: return _ParsedNode(id=identifier, node=None) @@ -865,7 +942,70 @@ def read_graph(state: lifecycle.GraphStateRoot, status: lifecycle.GenerationStat payload = json.loads(raw) except (ValueError, UnicodeError, RecursionError): raise ContextError("local graph document is not supported JSON") from None - return load_graph(payload, generation=status.generation, commit=status.manifest.commit) + provider = status.manifest.provider + requirement = f"{provider.get('distribution')}=={provider.get('version')}" + try: + return load_graph(payload, generation=status.generation, commit=status.manifest.commit) + except ReaderIncompatible as error: + error.provider = requirement + raise + except UnsupportedNodeType: + # The same unknown type means two different things depending on who + # built the generation. From a reviewed release it is a document this + # reader has no account of, and stays an unexplained refusal. From an + # unreviewed release it is the expected consequence of a provider the + # reader was never checked against, which has a concrete fix. Reviewed + # means this exact distribution at this exact version. + if _reviewed_provider(provider): + raise + raise ReaderIncompatible( + "local graph was built by a provider release this Code Mower reader was not reviewed against", + provider=requirement, + ) from None + + +#: Machine-readable search readiness, reported by ``build``, ``status`` and +#: ``connection-status`` beside -- never folded into -- the generation state. +READINESS_SCHEMA = "code_mower.contextGraphSearchReadiness.v1" +SEARCH_AVAILABLE = "available" +SEARCH_UNAVAILABLE = "unavailable" + + +def reader_not_checked(reason: str) -> dict[str, Any]: + """The readiness verdict when the graph was not opened: never searchable.""" + return {"schema": READINESS_SCHEMA, "search": SEARCH_UNAVAILABLE, + "reader": "not_checked", "reason": reason} + + +def search_readiness(state: lifecycle.GraphStateRoot, status: lifecycle.GenerationStatus) -> dict[str, Any]: + """Whether the installed reader can answer from the published generation. + + ``graph_status`` answers a different question: is the generation present, + intact, complete and bound to this revision. A generation can be all of + those and still be one this reader cannot consume, and a status that said + ``search: available`` on the strength of the lifecycle alone would disagree + with the first real query. So the check here is the query's own read -- + ``read_graph``, the same function ``graph_context`` calls -- and not a + second, cheaper approximation of it that could drift. + + Metadata only: the verdict, a fixed reason, and for a known mismatch the + remediation, which names releases and never graph content or paths. + """ + if not status.usable: + return reader_not_checked(status.state) + try: + read_graph(state, status) + except ReaderIncompatible as error: + remediation = error.remediation() + return {"schema": READINESS_SCHEMA, "search": SEARCH_UNAVAILABLE, + "reader": "incompatible", "reason": "reader_incompatible", + "detail": str(error), "remediation": remediation, + "next_action": remediation["action"]} + except ContextError as error: + return {"schema": READINESS_SCHEMA, "search": SEARCH_UNAVAILABLE, + "reader": "unreadable", "reason": "unreadable", "detail": str(error)} + return {"schema": READINESS_SCHEMA, "search": SEARCH_AVAILABLE, "reader": "compatible", + "installed_code_mower": CODE_MOWER_VERSION} @dataclass(frozen=True) @@ -1369,8 +1509,13 @@ def build_packet( # ``provider_partial`` covers both of its sources -- the generation's own # partial build, added just above, and a traversal that read a relationship # whose far end the document never declared. Neither is truncation, and a - # packet carrying either must not call itself complete. - packet_completeness = "partial" if truncated or "provider_partial" in omissions else "complete" + # packet carrying either must not call itself complete. Nor may one whose + # walk left an entity unresolved: ambiguity alone is a partial answer. + packet_completeness = ( + "partial" + if truncated or "provider_partial" in omissions or "unresolved_entities" in omissions + else "complete" + ) expiry = min( _timestamp(connection["expires_at"]), current + timedelta(seconds=limits["max_age_seconds"]), @@ -1400,6 +1545,14 @@ def build_packet( "graph_generation": result.generation, "documents": len(documents), "completeness": packet_completeness, + # Two different completeness claims, reported separately so a bounded + # answer is never read as a broken build. ``generation_completeness`` is + # the published build's own; ``query_completeness`` is this answer's, + # partial whenever a budget, a depth or the document limit stopped it, + # the walk crossed a relationship the provider could not state, or it + # left an entity unresolved. + "generation_completeness": completeness, + "query_completeness": packet_completeness, "truncated": truncated, "omissions": omissions, "recipients": list(connection["recipients"]), @@ -1494,6 +1647,13 @@ def graph_context( try: graph = read_graph(state, status) census = lifecycle.read_tracked_census(state.repository, manifest.commit) + except ReaderIncompatible as error: + # The same reason and the same remediation ``search_readiness`` reports, + # so status and the query that follows it cannot tell two stories. + remediation = error.remediation() + return _unavailable(required, "reader_incompatible", { + "detail": str(error), "remediation": remediation, "next_action": remediation["action"], + }) except ContextError as error: return _unavailable(required, "unreadable", {"detail": str(error)}) result = run_query(graph, question=question, target=target, depth=depth, node_budget=node_budget) @@ -1533,17 +1693,26 @@ def graph_context( "GraphContext", "GraphEdge", "GraphNode", + "KNOWN_PROVIDER_FILE_TYPES", "MAX_NODE_BUDGET", "OPTIONAL_UNAVAILABLE", "PacketDraft", "QUESTIONS", "QUERY_SCHEMA", "QueryResult", + "READER_PROVIDERS", + "READINESS_SCHEMA", "REQUIRED_UNAVAILABLE", + "ReaderIncompatible", "Relation", + "SEARCH_AVAILABLE", + "SEARCH_UNAVAILABLE", + "UnsupportedNodeType", "build_packet", "graph_context", "load_graph", "read_graph", + "reader_not_checked", "run_query", + "search_readiness", ) diff --git a/tests/test_context_graph_connection.py b/tests/test_context_graph_connection.py index f1805b2e..0c7d1955 100644 --- a/tests/test_context_graph_connection.py +++ b/tests/test_context_graph_connection.py @@ -27,15 +27,18 @@ from types import SimpleNamespace from unittest.mock import patch +from code_mower import __version__ as CODE_MOWER_VERSION from code_mower import context_delivery, context_packets, context_prepare, context_session +from code_mower import context_graph_command as command from code_mower import context_graph_connection as connection from code_mower import context_graph_lifecycle as lifecycle from code_mower import context_graph_query as query from code_mower.context_contract import ContextError, ContextRequest from code_mower.context_store import ContextStore +from code_mower.package_manifest import PACKAGE_FILES from test_context_connections import MemoryVault from test_context_graph_query import ( - PIN, git, graph_document, indexer, make_repository, wide_graph_document, + PIN, edge, git, graph_document, indexer, make_repository, node, wide_graph_document, ) @@ -717,5 +720,292 @@ def test_saved_state_refuses_another_providers_connection(self) -> None: connection.saved_state({"schema": connection.GRAPH_SCHEMA}, "local-graph") +def doc_ref_document() -> dict: + """A complete generation whose pinned extractor also emitted ``doc_ref`` nodes.""" + document = graph_document() + document["nodes"].append( + node("n-doc-ref", "README.md", "example_pkg/config.py", 1, file_type="doc_ref") + ) + document["edges"].extend([ + edge("n-config", "n-doc-ref", "references"), + edge("n-doc-ref", "n-load", "references"), + ]) + return document + + +def unknown_type_document() -> dict: + """A generation carrying a node type no Code Mower reader has an account of.""" + document = graph_document() + document["nodes"].append( + node("n-hologram", "hologram-label", "example_pkg/config.py", 1, file_type="hologram_ref") + ) + return document + + +@unittest.skipUnless(os.name == "posix", "private context needs POSIX protections") +class SearchReadinessTests(unittest.TestCase): + """``status``, ``connection-status`` and a real query agree on search (#1029). + + The lifecycle alone can say a generation is current and complete. Whether + the installed reader can answer from it is a second fact, and every + surface that reports ``search`` must report that one -- the same verdict + the query itself reaches -- rather than inferring it from the first. + """ + + def setUp(self) -> None: + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name).resolve() + self.repository = make_repository(self.root) + self.private = self.root / "private" + self.private.mkdir(mode=0o700) + self.store: ContextStore | None = None + + def publish(self, document: dict, *, pin=PIN): + return lifecycle.build_graph( + self.repository, pin=pin, indexer=indexer(document), root=self.private, + ) + + def connected(self) -> ContextStore: + if self.store is None: + self.store = ContextStore(self.private, vault=MemoryVault()) + connection.connect(self.store, "local-graph", { + "repository_root": str(self.repository), + "repositories": ["owner/repo"], + "recipients": RECIPIENTS, + }) + return self.store + + def readiness(self) -> dict: + return query.search_readiness( + lifecycle.GraphStateRoot(self.repository, root=self.private), + lifecycle.graph_status(self.repository, root=self.private), + ) + + def status_command(self, *, as_json: bool = True): + stream = io.StringIO() + with redirect_stdout(stream): + code = command.main([ + "status", "--repo-path", str(self.repository), "--state-dir", str(self.private), + *(["--json"] if as_json else []), + ]) + output = stream.getvalue() + return code, (json.loads(output) if as_json else output) + + def ask(self, **overrides) -> query.GraphContext: + store = self.connected() + with store.locked("local-graph") as locked: + envelope = connection.authorize_locked(locked, "local-graph", root=self.private) + arguments = { + "question": "impact", "target": "parse_config", "envelope": envelope, + "policy": POLICY, "context_repository": "owner/repo", "work_item": "WORK-1", + "root": self.private, + } + arguments.update(overrides) + return query.graph_context(self.repository, **arguments) + + def observed(self): + """Every surface at once: the status command, the connection, and a query.""" + code, status = self.status_command() + report = connection.status(self.connected(), "local-graph", root=self.private) + return code, status, report, self.ask() + + def assert_private(self, *values) -> None: + """No graph content, target, or local path in any reported verdict.""" + text = json.dumps(values, sort_keys=True) + for forbidden in ("README.md", "parse_config", "example_pkg", "n-doc-ref", + "hologram", str(self.root), str(self.repository), str(self.private)): + self.assertNotIn(forbidden, text) + + def test_a_complete_generation_with_doc_refs_validates_and_is_queryable(self) -> None: + self.publish(doc_ref_document()) + readiness = self.readiness() + self.assertEqual((readiness["search"], readiness["reader"]), ("available", "compatible")) + self.assertEqual(readiness["installed_code_mower"], CODE_MOWER_VERSION) + code, status, report, context = self.observed() + self.assertEqual(code, 0, status) + self.assertEqual((status["state"], status["search"]), ("current", "available")) + self.assertEqual(status["build"]["completeness"], lifecycle.COMPLETE) + self.assertEqual((report["search"], report["authorization"]), ("available", "available")) + self.assertEqual(context.status, query.AVAILABLE) + self.assertEqual(context.summary["generation_completeness"], lifecycle.COMPLETE) + self.assertTrue(context.packet["documents"]) + + def test_an_old_reader_mismatch_is_an_upgrade_action_everywhere(self) -> None: + """The v1.4.2 reader, simulated: the same code path without ``doc_ref``. + + The generation is valid, current and complete; only the reader is + behind. Status must not say ``search: available``, and neither it nor + the query may call the graph ``unreadable`` -- the fix is to upgrade. + """ + self.publish(doc_ref_document()) + older = query.GRAPH_FILE_TYPES - {"doc_ref"} + with patch.object(query, "GRAPH_FILE_TYPES", older): + readiness = self.readiness() + code, status, report, context = self.observed() + text_code, text = self.status_command(as_json=False) + for verdict in (readiness, status["query_reader"], report["query_reader"]): + self.assertEqual(verdict["search"], "unavailable") + self.assertEqual((verdict["reader"], verdict["reason"]), ("incompatible", "reader_incompatible")) + remediation = verdict["remediation"] + self.assertEqual(remediation["required_code_mower"], "1.5.0") + self.assertEqual(remediation["installed_code_mower"], CODE_MOWER_VERSION) + self.assertEqual(remediation["node_type"], "doc_ref") + self.assertEqual(remediation["generation_provider"], "graphifyy==0.9.58") + self.assertIn("Upgrade Code Mower to 1.5.0 or later", verdict["next_action"]) + # The generation itself is still the lifecycle's ``current``: this is a + # reader verdict beside it, not a rewrite of it. + self.assertEqual(code, 1) + self.assertEqual((status["state"], status["usable"], status["search"]), + ("current", True, "unavailable")) + self.assertEqual((report["search"], report["authorization"]), ("unavailable", "unavailable")) + self.assertEqual(context.status, query.REQUIRED_UNAVAILABLE) + self.assertEqual(context.summary["reason"], "reader_incompatible") + self.assertEqual(context.summary["next_action"], readiness["next_action"]) + self.assertEqual(text_code, 1) + self.assertIn("search: unavailable (query reader: incompatible)", text) + self.assertIn("Upgrade Code Mower to 1.5.0", text) + self.assert_private(readiness, status, report, context.summary, text) + + def test_an_unreviewed_provider_release_names_a_rebuild_or_an_upgrade(self) -> None: + other = lifecycle.GraphifyPin(distribution="graphifyy", version="0.9.99", wheel_sha256="a" * 64) + self.publish(unknown_type_document(), pin=other) + readiness = self.readiness() + self.assertEqual((readiness["search"], readiness["reason"]), ("unavailable", "reader_incompatible")) + self.assertEqual(readiness["remediation"]["generation_provider"], "graphifyy==0.9.99") + self.assertNotIn("required_code_mower", readiness["remediation"]) + self.assertIn("0.9.58", readiness["next_action"]) + context = self.ask() + self.assertEqual(context.summary["reason"], "reader_incompatible") + self.assert_private(readiness, context.summary) + + def test_a_same_version_provider_from_another_distribution_is_not_reviewed(self) -> None: + """The review covers ``graphifyy==0.9.58``, not every distribution at 0.9.58.""" + other = lifecycle.GraphifyPin(distribution="other-provider", version="0.9.58", wheel_sha256="a" * 64) + self.publish(unknown_type_document(), pin=other) + readiness = self.readiness() + self.assertEqual( + (readiness["search"], readiness["reader"], readiness["reason"]), + ("unavailable", "incompatible", "reader_incompatible"), + ) + remediation = readiness["remediation"] + self.assertEqual(remediation["generation_provider"], "other-provider==0.9.58") + self.assertEqual(remediation["reader_providers"], ["graphifyy==0.9.58"]) + self.assertNotIn("required_code_mower", remediation) + self.assertNotIn("node_type", remediation) + self.assertIn("graphifyy==0.9.58", readiness["next_action"]) + code, status, report, context = self.observed() + self.assertEqual(code, 1) + self.assertEqual(status["query_reader"]["reason"], "reader_incompatible") + self.assertEqual((report["search"], report["authorization"]), ("unavailable", "unavailable")) + self.assertEqual(context.status, query.REQUIRED_UNAVAILABLE) + self.assertEqual(context.summary["reason"], "reader_incompatible") + self.assert_private(readiness, status, report, context.summary) + self.assertNotIn("hologram", json.dumps([readiness, status, report, context.summary])) + + def test_the_reviewed_provider_is_matched_by_its_normalized_requirement(self) -> None: + for distribution, version, reviewed in ( + ("graphifyy", "0.9.58", True), + ("Graphifyy", "0.9.58", True), + ("graphifyy", "0.9.59", False), + ("other-provider", "0.9.58", False), + ("graphify", "0.9.58", False), + (None, "0.9.58", False), + ): + with self.subTest(distribution=distribution, version=version): + self.assertIs( + query._reviewed_provider({"distribution": distribution, "version": version}), reviewed, + ) + + def test_unknown_node_types_still_fail_closed(self) -> None: + with self.assertRaises(query.UnsupportedNodeType): + query.load_graph(unknown_type_document(), generation="a" * 32, commit="b" * 40) + self.publish(unknown_type_document()) + readiness = self.readiness() + self.assertEqual( + (readiness["search"], readiness["reader"], readiness["reason"]), + ("unavailable", "unreadable", "unreadable"), + ) + self.assertNotIn("remediation", readiness) + code, status, report, context = self.observed() + self.assertEqual(code, 1) + self.assertEqual(status["search"], "unavailable") + self.assertEqual((report["search"], report["authorization"]), ("unavailable", "unavailable")) + self.assertEqual(context.status, query.REQUIRED_UNAVAILABLE) + self.assertEqual(context.summary["reason"], "unreadable") + self.assert_private(readiness, status, report, context.summary) + + def test_a_disconnected_status_does_not_read_the_graph(self) -> None: + """Disconnected cannot search, so the reader check is not paid for.""" + self.publish(doc_ref_document()) + store = self.connected() + with patch.object(query, "read_graph", wraps=query.read_graph) as reader: + report = connection.status(store, "local-graph", root=self.private) + self.assertEqual(reader.call_count, 1) + self.assertEqual(report["query_reader"]["reader"], "compatible") + connection.disconnect(store, "local-graph") + reader.reset_mock() + report = connection.status(store, "local-graph", root=self.private) + reader.assert_not_called() + self.assertEqual(report["query_reader"], { + "schema": query.READINESS_SCHEMA, "search": "unavailable", + "reader": "not_checked", "reason": "disconnected", + }) + self.assertEqual((report["search"], report["authorization"]), ("unavailable", "unavailable")) + # The generation's own verdict is still reported beside it. + self.assertEqual(report["graph"]["state"], "current") + self.assert_private(report) + + def test_a_bounded_partial_query_stays_available_and_says_why(self) -> None: + """Partial answer, complete generation: two facts, reported apart.""" + document = wide_graph_document(7) + document["edges"][0]["confidence"] = "AMBIGUOUS" + self.publish(document) + code, status, report, _ = self.observed() + self.assertEqual(code, 0, status) + self.assertEqual((status["state"], status["search"]), ("current", "available")) + self.assertEqual(status["build"]["completeness"], lifecycle.COMPLETE) + self.assertEqual(report["authorization"], "available") + # Six relationships of seven, five documents of six, one ambiguous claim. + context = self.ask(node_budget=6) + self.assertEqual((context.status, context.dependent_work), (query.AVAILABLE, "usable")) + self.assertEqual(context.summary["generation_completeness"], lifecycle.COMPLETE) + self.assertEqual(context.summary["query_completeness"], "partial") + for omission in ("provider_has_more", "unresolved_entities", "document_limit"): + self.assertIn(omission, context.packet["omissions"]) + self.assertIn(omission, context.summary["omissions"]) + self.assertNotIn("provider_partial", context.packet["omissions"]) + self.assertTrue(context.packet["truncated"]) + + def test_an_ambiguity_only_answer_is_partial_but_usable(self) -> None: + """No budget, depth or document limit applies; one relationship is ambiguous.""" + document = wide_graph_document(3) + document["edges"][0]["confidence"] = "AMBIGUOUS" + self.publish(document) + code, status, report, _ = self.observed() + self.assertEqual(code, 0, status) + self.assertEqual((status["search"], report["authorization"]), ("available", "available")) + context = self.ask() + self.assertEqual((context.status, context.dependent_work), (query.AVAILABLE, "usable")) + self.assertEqual(context.summary["generation_completeness"], lifecycle.COMPLETE) + self.assertEqual(context.summary["query_completeness"], "partial") + self.assertEqual(context.summary["completeness"], "partial") + self.assertEqual(context.packet["completeness"], "partial") + self.assertFalse(context.packet["truncated"]) + self.assertEqual(context.packet["omissions"], ["unresolved_entities"]) + self.assertEqual(context.summary["omissions"], ["unresolved_entities"]) + + def test_the_graphify_modules_remain_in_the_package_inventory(self) -> None: + manifest = json.loads( + (Path(__file__).resolve().parents[1] / "code-mower-package-manifest.json").read_text(encoding="utf-8") + ) + packaged = {entry[0] for entry in PACKAGE_FILES} + written = {entry.get("source") for entry in manifest["files_written"]} + for module in ("context_graph", "context_graph_lifecycle", "context_graph_query", + "context_graph_command", "context_graph_connection"): + self.assertIn(f"src/code_mower/{module}.py", packaged) + self.assertIn(f"src/code_mower/{module}.py", written) + + if __name__ == "__main__": # pragma: no cover - direct invocation unittest.main() diff --git a/tests/test_context_graph_lifecycle.py b/tests/test_context_graph_lifecycle.py index f858ce44..1f97a2b0 100644 --- a/tests/test_context_graph_lifecycle.py +++ b/tests/test_context_graph_lifecycle.py @@ -4567,8 +4567,16 @@ def test_the_verbs_address_one_checkout_from_any_directory_inside_it(self) -> No Built through the injected indexer rather than the launcher, because what is under test is which state the verbs address, not containment. + The artifact is a real (empty) provider graph document: ``status`` + exits zero only when the query reader can consume the generation too. """ - self.build() + document = b'{"nodes": [], "edges": []}' + buffer = io.BytesIO() + with tarfile.open(fileobj=buffer, mode="w", format=tarfile.PAX_FORMAT) as archive: + info = tarfile.TarInfo("graph.json") + info.size = len(document) + archive.addfile(info, io.BytesIO(document)) + self.build(indexer=recording_indexer(buffer.getvalue())) inside = ["--repo-path", str(self.repository / "example_pkg"), "--state-dir", str(self.state), "--json"] code, output = self.run_command("status", *inside)