Repository navigation
feat: add grounded incident runbook investigator #48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
72986f7
c048a7f
b680754
7409b33
043f3e7
dd0fae1
cbfed88
1a92e68
f8827d5
a803452
e2c4732
5775eff
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| # Incident investigator | ||
|
|
||
| The incident investigator retrieves evidence from approved local Markdown runbooks. It is deliberately retrieval-grounded: it returns ranked excerpts and provenance, rather than generating root-cause claims or executing remediation. | ||
|
|
||
| ## Runbook contract | ||
|
|
||
| Each runbook must be a UTF-8 `.md` file below the selected directory and start with bounded front matter: | ||
|
|
||
| ```markdown | ||
| --- | ||
| id: database-timeout | ||
| title: Database timeout response | ||
| version: 1.0.0 | ||
| --- | ||
| # Response steps | ||
| ... | ||
| ``` | ||
|
|
||
| At investigation time, operators must provide an immutable source-control revision for the complete runbook set. The result records: | ||
|
|
||
| - runbook ID, version, and relative source path; | ||
| - supplied repository commit revision; | ||
| - SHA-256 digest of the exact runbook file; | ||
| - deterministic relevance score and a bounded matching excerpt. | ||
|
|
||
| This lets an incident record show exactly which approved guidance was retrieved. The command never downloads runbooks, calls a hosted model, modifies infrastructure, or presents its response as an authorization to act. | ||
|
|
||
| ## CLI usage | ||
|
|
||
| Pin the runbooks to the revision checked out by your deployment or incident workspace: | ||
|
|
||
| ```bash | ||
| logsight investigate "database timeout during checkout" \ | ||
| --runbooks ./runbooks \ | ||
| --revision "$(git rev-parse HEAD)" \ | ||
| --evidence "ERROR connection pool timeout" \ | ||
| --evidence "trace_id=abc123" | ||
| ``` | ||
|
|
||
| Alternatively, provide the revision through deployment configuration: | ||
|
|
||
| ```bash | ||
| export LOGSIGHT_RUNBOOK_REVISION="$(git rev-parse HEAD)" | ||
| logsight investigate "queue consumer lag" --runbooks ./runbooks | ||
| ``` | ||
|
|
||
| Review cited excerpts and the surrounding versioned runbook before any production action. A result with no citations is intentionally inconclusive; it should prompt human triage or authoring of an approved runbook, not an invented recommendation. | ||
|
|
||
| ## Promotion controls | ||
|
|
||
| - Store runbooks with code review and immutable release/commit history. | ||
| - Require incident responders to include the returned revision and digest in an incident record. | ||
| - Review changes to runbook wording as operational changes, including rollback implications. | ||
| - Use access-controlled repositories and deployment-owned checkout paths for production runbooks. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,13 +3,15 @@ | |
| from __future__ import annotations | ||
|
|
||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| import click | ||
| from rich.console import Console | ||
| from rich.markup import escape | ||
| from rich.table import Table | ||
|
|
||
| from logsight.analyzer import AnomalyReport, detect_anomalies, error_rate_spike_details | ||
| from logsight.investigator import IncidentInvestigator, RunbookRepository | ||
| from logsight.parser import parse_file, parse_lines | ||
| from logsight.reasoning import EvidenceExplanation, explain_report | ||
|
|
||
|
|
@@ -169,6 +171,64 @@ def stdin_cmd(threshold: float, explain: bool) -> None: | |
| _print_explanations(explain_report(report)) | ||
|
|
||
|
|
||
| @main.command("investigate") | ||
| @click.argument("query") | ||
| @click.option( | ||
| "--runbooks", | ||
| type=click.Path(exists=True, file_okay=False, readable=True, path_type=Path), | ||
| default=Path("runbooks"), | ||
| show_default=True, | ||
| help="Directory containing approved, versioned Markdown runbooks.", | ||
| ) | ||
| @click.option( | ||
| "--revision", | ||
| envvar="LOGSIGHT_RUNBOOK_REVISION", | ||
| required=True, | ||
| help="Immutable commit revision for the runbook set.", | ||
| ) | ||
| @click.option( | ||
| "--evidence", | ||
| multiple=True, | ||
| help="Additional incident evidence; repeat this option for each item.", | ||
| ) | ||
| @click.option( | ||
| "--limit", | ||
| default=3, | ||
| type=click.IntRange(1, 10), | ||
| show_default=True, | ||
| help="Maximum number of grounded runbook citations to return.", | ||
| ) | ||
| def investigate_cmd( | ||
| query: str, | ||
| runbooks: Path, | ||
| revision: str, | ||
| evidence: tuple[str, ...], | ||
| limit: int, | ||
| ) -> None: | ||
| """Retrieve approved runbook evidence for an incident QUERY.""" | ||
|
|
||
| try: | ||
| repository = RunbookRepository(runbooks, revision=revision) | ||
| investigation = IncidentInvestigator(repository.load()).investigate( | ||
| query, evidence=evidence, limit=limit | ||
| ) | ||
| except (OSError, ValueError) as exc: | ||
| raise click.ClickException(str(exc)) from exc | ||
|
|
||
| console.print("[bold]Investigation provenance[/bold]") | ||
| console.print(f" revision : [cyan]{investigation.revision}[/cyan]") | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Preserve the supplied revision when no runbooks load. If Preserve the supplied revision for this result, or reject an empty runbook set with an explicit error.
|
||
| console.print(f" guidance : {investigation.guidance}") | ||
| for citation in investigation.citations: | ||
| console.print( | ||
| f"\n[bold]{escape(citation.title)}[/bold] " | ||
| f"({escape(citation.identifier)} v{escape(citation.version)})" | ||
| ) | ||
| console.print(f" source : {escape(citation.path)} @ {escape(citation.revision)}") | ||
| console.print(f" digest : {citation.digest}") | ||
| console.print(f" score : {citation.score:.4f}") | ||
| console.print(f" excerpt : {escape(citation.excerpt)}") | ||
|
|
||
|
|
||
| @main.command("health") | ||
| def health_cmd() -> None: | ||
| """Verify that LogSight-AI is installed and operational.""" | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,180 @@ | ||
| """Version-pinned, retrieval-grounded incident investigation helpers.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import hashlib | ||
| import re | ||
| from collections.abc import Sequence | ||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
|
|
||
| _MAX_RUNBOOKS = 100 | ||
| _MAX_RUNBOOK_CHARS = 64_000 | ||
| _MAX_QUERY_CHARS = 2_000 | ||
| _TOKEN = re.compile(r"[a-z0-9][a-z0-9_-]{1,}") | ||
| _REVISION = re.compile(r"[A-Za-z0-9._-]{7,64}") | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class Runbook: | ||
| """An approved runbook with explicit version and repository provenance.""" | ||
|
|
||
| identifier: str | ||
| title: str | ||
| version: str | ||
| revision: str | ||
| path: str | ||
| content: str | ||
| digest: str | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class RunbookCitation: | ||
| """A ranked, bounded excerpt suitable for a human incident review.""" | ||
|
|
||
| identifier: str | ||
| title: str | ||
| version: str | ||
| revision: str | ||
| path: str | ||
| digest: str | ||
| score: float | ||
| excerpt: str | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class Investigation: | ||
| """Evidence-first investigation outcome with immutable source provenance.""" | ||
|
|
||
| query: str | ||
| revision: str | ||
| citations: tuple[RunbookCitation, ...] | ||
| guidance: str | ||
|
|
||
|
|
||
| class RunbookRepository: | ||
| """Load bounded Markdown runbooks with versioned front matter.""" | ||
|
|
||
| def __init__(self, root: str | Path, *, revision: str) -> None: | ||
| self.root = Path(root).resolve() | ||
| if not _REVISION.fullmatch(revision): | ||
| raise ValueError("runbook revision must be a 7-64 character identifier") | ||
| self.revision = revision | ||
|
|
||
| def load(self) -> list[Runbook]: | ||
| """Load approved runbooks under the configured root in stable path order.""" | ||
|
|
||
| if not self.root.is_dir(): | ||
| raise FileNotFoundError(f"runbook directory was not found: {self.root}") | ||
| paths = sorted(self.root.rglob("*.md")) | ||
| if len(paths) > _MAX_RUNBOOKS: | ||
| raise ValueError(f"runbook directory exceeds {_MAX_RUNBOOKS} documents") | ||
| return [self._load_path(path) for path in paths] | ||
|
|
||
| def _load_path(self, path: Path) -> Runbook: | ||
| resolved = path.resolve() | ||
| try: | ||
| relative_path = resolved.relative_to(self.root) | ||
| except ValueError as exc: | ||
| raise ValueError("runbook path must stay within the configured directory") from exc | ||
| content = resolved.read_text(encoding="utf-8") | ||
| if len(content) > _MAX_RUNBOOK_CHARS: | ||
| raise ValueError(f"runbook exceeds {_MAX_RUNBOOK_CHARS} characters: {relative_path}") | ||
|
Comment on lines
+80
to
+82
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bound the read before loading the complete runbook.
Read at most
|
||
| metadata, body = _split_front_matter(content) | ||
| identifier = _required_metadata(metadata, "id") | ||
| title = _required_metadata(metadata, "title") | ||
| version = _required_metadata(metadata, "version") | ||
| if not body.strip(): | ||
| raise ValueError(f"runbook has no body: {relative_path}") | ||
| return Runbook( | ||
| identifier=identifier, | ||
| title=title, | ||
| version=version, | ||
| revision=self.revision, | ||
| path=relative_path.as_posix(), | ||
| content=body.strip(), | ||
| digest=hashlib.sha256(content.encode("utf-8")).hexdigest(), | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Compute the runbook digest from the file bytes. The digest on line 96 hashes the decoded text. Read the bytes once, hash them, and decode the same bytes for parsing: Suggested fix
|
||
| ) | ||
|
|
||
|
|
||
| class IncidentInvestigator: | ||
| """Retrieve approved runbook excerpts without inventing operational actions.""" | ||
|
|
||
| def __init__(self, runbooks: Sequence[Runbook]) -> None: | ||
| self.runbooks = tuple(runbooks) | ||
|
|
||
| def investigate( | ||
| self, query: str, *, evidence: Sequence[str] = (), limit: int = 3 | ||
| ) -> Investigation: | ||
| """Rank version-pinned runbooks against an incident query and evidence.""" | ||
|
|
||
| if not query.strip() or len(query) > _MAX_QUERY_CHARS: | ||
| raise ValueError("query must be non-empty and within the configured limit") | ||
| if not 1 <= limit <= 10: | ||
| raise ValueError("limit must be between 1 and 10") | ||
| query_tokens = _tokens(" ".join([query, *evidence])) | ||
| if not query_tokens: | ||
| raise ValueError("query must contain searchable terms") | ||
|
|
||
| ranked: list[tuple[float, Runbook]] = [] | ||
| for runbook in self.runbooks: | ||
| document_tokens = _tokens(f"{runbook.title} {runbook.content}") | ||
| overlap = len(query_tokens & document_tokens) | ||
| if overlap: | ||
| score = overlap / len(query_tokens) | ||
| ranked.append((score, runbook)) | ||
| ranked.sort(key=lambda item: (-item[0], item[1].identifier, item[1].path)) | ||
| citations = tuple( | ||
| _citation(runbook, score, query_tokens) for score, runbook in ranked[:limit] | ||
| ) | ||
| revision = self.runbooks[0].revision if self.runbooks else "none" | ||
| guidance = ( | ||
| "Review the cited approved runbook excerpts before acting." | ||
| if citations | ||
| else "No approved runbook matched the supplied incident evidence." | ||
| ) | ||
| return Investigation(query=query, revision=revision, citations=citations, guidance=guidance) | ||
|
|
||
|
|
||
| def _split_front_matter(content: str) -> tuple[dict[str, str], str]: | ||
| if not content.startswith("---\n"): | ||
| raise ValueError("runbook must begin with YAML-style front matter") | ||
| closing = content.find("\n---\n", 4) | ||
| if closing < 0: | ||
| raise ValueError("runbook front matter is not terminated") | ||
| metadata: dict[str, str] = {} | ||
| for line in content[4:closing].splitlines(): | ||
| key, separator, value = line.partition(":") | ||
| if not separator or not key or not value.strip(): | ||
| raise ValueError("runbook front matter must contain key: value entries") | ||
| metadata[key.strip()] = value.strip() | ||
| return metadata, content[closing + 5 :] | ||
|
|
||
|
|
||
| def _required_metadata(metadata: dict[str, str], key: str) -> str: | ||
| value = metadata.get(key, "") | ||
| if not value or len(value) > 128: | ||
| raise ValueError(f"runbook metadata requires a bounded {key}") | ||
| return value | ||
|
|
||
|
|
||
| def _tokens(value: str) -> set[str]: | ||
| return set(_TOKEN.findall(value.lower())) | ||
|
|
||
|
|
||
| def _citation(runbook: Runbook, score: float, query_tokens: set[str]) -> RunbookCitation: | ||
| lines = runbook.content.splitlines() | ||
| matching = next( | ||
| (line.strip() for line in lines if query_tokens & _tokens(line)), | ||
| lines[0].strip(), | ||
| ) | ||
| return RunbookCitation( | ||
| identifier=runbook.identifier, | ||
| title=runbook.title, | ||
| version=runbook.version, | ||
| revision=runbook.revision, | ||
| path=runbook.path, | ||
| digest=runbook.digest, | ||
| score=round(score, 4), | ||
| excerpt=matching[:500], | ||
| ) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| id: database-timeout | ||
| title: Database timeout response | ||
| version: 1.0.0 | ||
| --- | ||
| # Database timeout response | ||
|
|
||
| 1. Confirm the error-rate increase and affected service in the normalized log stream. | ||
| 2. Correlate the event fingerprint with database connection, latency, and saturation telemetry. | ||
| 3. Preserve the incident timeline and request owner approval before changing connection-pool or failover settings. | ||
| 4. Record the runbook version and repository revision used for the decision. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| id: queue-backlog | ||
| title: Queue backlog response | ||
| version: 1.0.0 | ||
| --- | ||
| # Queue backlog response | ||
|
|
||
| 1. Confirm producer rate, consumer lag, and retry volume for the affected topic or queue. | ||
| 2. Identify the affected consumer group and correlate it with trace and deployment evidence. | ||
| 3. Preserve offsets and the incident timeline before scaling consumers or replaying work. | ||
| 4. Record the runbook version and repository revision used for the decision. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Document that the revision is a caller-supplied label, not an enforced pin.
The example reads the current working-tree runbooks.
RunbookRepositorychecks the revision's syntax but does not compare those files with that commit. If an operator edits a runbook without committing it, the command cites the edited guidance under the unchangedHEADrevision.State that the command does not verify the revision-to-content relationship. Require a clean, deployment-owned checkout whose runbook content matches the supplied commit before using this example for incident provenance. Apply the same clarification to the README example.