Document shipping the audit log to an OTel collector (+ examples/otel-collector.yaml)

[plusky]

SCOPE EXPANDED 2026-08-18 (decision): not only document — bugwarden
implements OTLP export natively. The paragraph below recording that "the
guard does not speak OTLP, does not export anything itself" is superseded
for export (the file remains authoritative); its rationale survives as the
failure-semantics boundary: the JSONL file keeps today's fail-closed
semantics untouched, and OTLP delivery is best-effort on top.

Implementation (supersedes the docs-only proposal in part)

• Audit records → OTLP log records: one per AuditEvent, structured
  attributes (kind, tool, verdict, rule, session, seq) plus the full JSON
  body; trace_id/span_id from the record's TraceContext (Populate audit trace context from params._meta (traceparent) #28) so
  collector-side correlation works. Export happens after the file write.
• Diagnostics → OTLP logs: the server's own tracing output gains an
  OTLP layer alongside stderr.
• Config: standard OTel env vars, off by default, matching the
  container env convention (feat(server): let the environment set allowed hosts and the auth header #104/Per-caller authentication on the MCP endpoint #32): OTEL_EXPORTER_OTLP_ENDPOINT
  (unset or empty ⇒ no export, zero new behaviour), _HEADERS (secret
  material under I12 — never logged), _PROTOCOL, OTEL_SERVICE_NAME
  (default bugwarden).
• Transport: OTLP/HTTP + protobuf over the existing reqwest/rustls
  stack; no gRPC/tonic.
• Failure semantics: best-effort — bounded batch queue, drop counter
  logged at powers of two; a collector outage never stops serving. The
  file sink's FailMode is unchanged and remains the only load-bearing one.
• Invariants: exported stream carries exactly what the file carries (I12);
  the pipeline is reachable through no MCP surface (I15); guard behaviour
  untouched (I9).

The original deliverables below still ship, extended: the collector
example carries BOTH receivers (OTLP for native export, filelog for the
sidecar/tamper-evidence path), and the sensitivity doc covers both.

REVISED 2026-08-18 (decision, supersedes the best-effort paragraphs above):

• OTLP export is fail-closed when configured. If an OTLP endpoint is
  configured and delivery of audit records does not work — at startup or at
  any point while serving — the server stops accepting work, mirroring the
  audit file's FailMode semantics (refuse tool calls with the uniform text;
  recover when delivery recovers). No silent drops of audit records: the
  bounded-queue drop counter survives only for diagnostics, never for audit
  records.
• The JSONL audit file becomes optional. A deployment chooses its audit
  sinks: file, OTLP, both, or none. Container deployments may run
  OTLP-only (no volume). Running with no audit sink at all is permitted
  (revised 2026-08-18: an unconfigured sink is a deployment choice, not an
  error); what is not permitted is a configured sink that does not work —
  any configured sink is load-bearing and gates serving when it fails.
  When both are configured, the file write still precedes export and
  either sink failing gates serving.
• Diagnostics export stays best-effort — a dropped log line must not
  halt the guard; only audit records carry the fail-closed guarantee.

Depends on #28 — the document should describe correlating guard records
with client traces, which only works once the trace field is actually
populated. The tailing setup itself can be written before that; the
correlation section cannot.

The audit log is a versioned JSONL file on disk and deliberately stays that
way: the guard does not speak OTLP, does not export anything itself, and does
not write the audit stream to stderr. Shipping it off the host is an operator
task, and off-host shipping is what turns the log into a tamper-evident record
rather than a file the host can rewrite.

Right now we ship no guidance for doing that, so every operator has to work out
the same collector configuration.

Proposal

• examples/otel-collector.yaml — a working collector configuration using the
  filelog receiver to tail the audit file, parse the JSON, and lift the
  fields that matter for querying (event type, tool, verdict, session, and the
  trace ids from Populate audit trace context from params._meta (traceparent) #28) so records land as structured entries rather than opaque
  strings.
• A short document covering the parts a sample config cannot express: file
  permissions and which user the collector runs as (the log is written 0600),
  how the built-in rotation interacts with the receiver, and what the log
  contains — denied bug ids and matched rule names, which makes the shipped
  stream at least as sensitive as the bugs it describes and its destination a
  policy decision, not a detail.
• The container layout. Future deployments are containers configured
  entirely by environment (settled 2026-08-18), and the guard still writes
  only a file — so the doc must show the audit directory on a shared volume
  with the collector tailing it as a sidecar, where the 0600/user question
  above stops being a footnote and becomes the setup's one hard part.
  Docker examples pass configuration with -e (BUGZILLA_SERVER,
  BUGZILLA_API_KEY_FILE, ports), matching the clap env bindings every
  flag already has.
• A README pointer to it from the audit section.

Acceptance criteria

• The example config is valid and tails a rotating audit file without
  losing or duplicating records.
• The document states the sensitivity of the stream and the permissions
  model explicitly.
• The container layout is covered: audit dir on a shared volume,
  collector as a sidecar, permissions that survive both.
• Documentation nowhere suggests OTLP export replaces the file: the JSONL
  audit file stays the authoritative, fail-closed record, and export is
  described as a best-effort copy of it. (Reworded 2026-08-18: the original
  criterion — no suggestion the guard exports telemetry at all — predates the
  scope expansion above and would contradict it.)

---

See also #34 §4(f) — the 2026-07-28 revision deprecates the MCP Logging feature outright and points at stderr or OpenTelemetry, i.e. exactly this shape.