Release 3.0.0

CHANGELOG.md

@@ -11,6 +11,241 @@ target.
 
 ## [Unreleased]
 
+## [3.0.0] — 2026-10-03
+
+The request guards — outbound redaction, tool-call inspection and the
+content filter — now share their rule model with the desktop gateway:
+the same policy shape, built-in rule catalog, validation, rule view and
+sample trial, from thinkwatch-core. Each guard has three modes (off,
+observe, and one named for what it does: replace, cut off, enforce), and
+the content filter can delete what a rule matches as well as refuse or
+record it, and match by code point. Hidden characters become content
+filter rules, and the per-model output length guardrail becomes a cap on
+the output tokens a request may ask for. Settings saved by an earlier
+version are converted at the first start: read the first section before
+deploying. It is a major release because setting keys, console API
+routes and the `models` table change; the conversion keeps each
+deployment's behaviour.
+
+### Read before upgrading
+
+- **Back up `system_settings` and `models` first, and plan for no way
+  back to 2.2.** The upgrade converts the guard settings and drops a
+  column (next item). Version 2.2 cannot run on the converted database:
+  its settings are gone, so it would start on its seeded defaults, and
+  it can no longer read the models table. Take a copy before deploying:
+
+  ```sh
+  pg_dump --data-only --table=system_settings --table=models "$DATABASE_URL" > thinkwatch-2.2-guards.sql
+  ```
+
+- **Stop every replica of 2.2 before the first 3.0 one starts.** A 2.2
+  replica still running when 3.0 converts finds its settings gone (it
+  then filters and redacts nothing) and cannot rebuild its router; one
+  that restarts writes its seeded defaults back. 3.0 does not convert
+  those a second time — it removes them at its next start and logs a
+  warning — but the 2.2 replica runs on them until then. With the Helm
+  chart (release `thinkwatch`, namespace `thinkwatch` here):
+
+  ```sh
+  kubectl -n thinkwatch scale deployment/thinkwatch-server --replicas=0
+  kubectl -n thinkwatch wait --for=delete pod \
+    -l app.kubernetes.io/name=think-watch,app.kubernetes.io/component=server --timeout=5m
+  helm upgrade thinkwatch deploy/helm/think-watch -n thinkwatch   # with your usual values
+  kubectl -n thinkwatch scale deployment/thinkwatch-server --replicas=<replicas you run>
+  ```
+
+  The last command matters when `autoscaling.enabled` is on: the chart
+  then leaves the replica count alone, and it stays at 0. With Docker
+  Compose, `docker compose stop server` before pulling and starting the
+  new image.
+- **The old guard settings are converted at the first start, and
+  behave as before.** `security.content_filter_patterns`,
+  `security.hidden_text`, `security.pii_redactor_patterns` and
+  `security.tool_inspection` become `security.content`,
+  `security.redact` and `security.inspect_tools`, and are deleted, in
+  one transaction during the boot migration. The conversion is recorded
+  in `security.legacy_converted` (when, which version, what it
+  converted); with that record present, old keys or the old column
+  that show up again are removed, never converted, so the policies in
+  force are not overwritten. A content rule identical to a built-in
+  rule becomes that rule, switched on, any other a custom rule; a list
+  with rules in it runs in enforce mode with every built-in rule it did
+  not name switched off. The four seeded PII patterns become the
+  built-in rules for the same data (`cn-resident-id`, `bank-card`,
+  `email`, `cn-mobile-phone`), any other pattern a custom rule whose
+  label is its old placeholder prefix. Whatever the old runtime was
+  skipping (a rule that did not compile, a built-in rule id it did not
+  know, an `output_guardrails` value it could not read) is left out,
+  each with a warning in the start-up log. A model's
+  `output_guardrails` length cap becomes `max_output_tokens` (below),
+  and the column is dropped. To see what was converted, read the three
+  keys from Settings or `system_settings` afterwards; the start-up log
+  lists them too.
+- **Placeholders are written `<<TW_EMAIL_1>>`, not `{{EMAIL_1}}`.** The
+  label of a custom rule is upper case letters, digits and
+  underscores (an old prefix is converted: `REDACTED-SSN` →
+  `REDACTED_SSN`); the built-in identity number and bank card rules
+  use `ID_NUMBER` and `CARD_NUMBER`. Anything that looked for the old
+  form in answers or logs needs the new one.
+- **Redaction searches the whole request**, not only the user's
+  messages: the system prompt, earlier answers and tool-call arguments
+  are redacted too. Base64 payloads (images, files, signatures) are
+  still left alone. Rules run on the request as it is sent, as JSON,
+  where a custom pattern's match ends at a quote or a backslash: a
+  pattern written to match across a `"` in the decoded text needs
+  rewriting.
+- **Built-in credential rules start replacing on deployments that were
+  redacting.** API keys and tokens with a known prefix, private keys,
+  JWTs and connection-string passwords are built-in rules that ship
+  switched on. A deployment whose PII list had patterns in it runs
+  redaction in enforce mode after the upgrade, so these values are now
+  replaced as well. Switch the ones you do not want off on the
+  console's security page. With an empty PII list, redaction
+  converts to observe mode: it records what it finds and changes
+  nothing.
+- **The built-in rules that replace the seeded PII patterns are
+  stricter.** An identity number has to have a real province code, a
+  real date of birth and a matching check digit; a card number a known
+  network's prefix and length and a valid Luhn digit, and published
+  test card numbers do not count. Numbers the old regexes took for
+  them — any 18 digits, any 16 — are no longer replaced. A deployment
+  that relied on the looser match can add its old regex back as a
+  custom rule.
+- **Tool calls are judged as the client receives them, and two built-in
+  rules are new.** Inspection now reads a tool call converted to the
+  caller's format and with redacted values restored — what the client
+  would run — where it used to read the placeholders. The new
+  `secret-to-unknown-host` rule cuts (in enforce mode) a call that sends
+  a recognised API key or private key to a host that is neither local
+  nor the key's own provider; `upload-file-to-host` records a call that
+  uploads a local file to an outside host. A deployment running
+  tool-call inspection in enforce mode starts cutting the first; add it
+  to `disable` if that is not wanted.
+- **"Warn" and "log" are one action now, "record only"**, and hidden
+  characters are content filter rules: `unicode-tags` and
+  `bidi-controls`, plus `zero-width` and `private-use`, which ship off.
+  `security.hidden_text: block` converts to those two rules refusing,
+  `warn` and `log` to recording, `off` to switching them off. Recording
+  writes an audit event: `hidden_text: log` used to reach only the
+  application log, and now writes `gateway.content_flagged`.
+- **A deployment with no content rules starts recording.** An empty
+  content filter list converts to observe mode with the built-in rules
+  that ship on, so requests matching them (`ignore previous
+  instructions` and the like) write `gateway.content_flagged` events
+  where 2.2 wrote nothing. Nothing on the wire changes.
+- **The output length guardrail is replaced by a model's maximum output
+  tokens — check each model's after upgrading.** A cap of N bytes on
+  the answer converts to `ceil(N / 4)` output tokens, stored as
+  converted. The answer is no longer measured or cut: a request asking
+  for more tokens than the cap is lowered to it, in whichever field its
+  API uses (a Chat request that sets both `max_tokens` and
+  `max_completion_tokens` has both lowered), and the upstream stops
+  there; one asking for less keeps its own. A request that sets no limit
+  is held to the cap only when the cap is within what the gateway knows
+  the model's family to take (32,000 tokens for Claude models, 8,192 for
+  others); a Chat request is then given `max_completion_tokens` on
+  OpenAI's own endpoint, whose reasoning models refuse `max_tokens`, and
+  `max_tokens` elsewhere. Above that figure, the request goes out
+  without a limit rather than with one the model could refuse, and the
+  model's own default applies: a 100,000-byte cap on a non-Claude model
+  converts to 25,000 tokens, so its requests that set no limit are not
+  capped at all. To hold every request to a cap, have the clients send
+  a limit, or set the cap to that family figure or below. Reasoning
+  (thinking) tokens count towards the cap on the APIs that bill them as
+  output, so a cap that fit an answer can cut short a model that thinks
+  first. An Anthropic request with extended thinking has its thinking
+  budget lowered below the cap too, or thinking turned off when the cap
+  is 1,024 tokens or less, the smallest budget Anthropic takes. The
+  model API's `output_guardrails` field is gone: a request that still
+  sets one (anything but `null` or `[]`) is refused with `400`, so a
+  script cannot believe answers are still capped. `max_output_tokens`
+  (1 to 2147483647, `null` for no limit) replaces it.
+- **A new installation observes by default.** Every guard starts in
+  observe mode, with only the built-in rules that rarely misfire
+  switched on (personal data such as e-mail addresses and phone
+  numbers ships off). Nothing is refused, replaced or deleted until a
+  guard is switched to its third mode.
+- **A content filter refusal is `403`**, with the error type of the
+  caller's API (`permission_error` for OpenAI-style APIs). Keyword and
+  regex rules used to refuse with `400`. In `gateway_logs`, its
+  `error_type` is `PolicyBlocked`, where it was `TransformError`.
+- **A rule that deletes text changes what is stored.** The request a
+  content rule stripped goes upstream, is redacted and is captured as
+  the stripped one: the audit log's request body is what was sent, not
+  what the caller typed.
+- **Changing a guard policy takes `settings:write` and the guard's own
+  permission**: `pii_redactor:write` for `security.redact`,
+  `content_filter:write` for `security.content` and
+  `security.inspect_tools`, through `PATCH /api/admin/settings`. 2.2
+  checked `settings:write` on the server and the guard permission in
+  the console; both are checked on the server now. Trying a sample
+  takes `pii_redactor:read` or `content_filter:read`. Reading the
+  policies — `GET /api/admin/security`, which the console's security
+  page loads — takes `settings:read`. The seeded `admin` and
+  `super_admin` roles hold all of them.
+- **Console API changes.** `GET /api/admin/security` lists each guard's
+  mode and every rule, and `POST /api/admin/security/{guard}/test` tries
+  a sample; they replace `/api/admin/settings/content-filter/test`,
+  `/content-filter/presets`, `/pii-redactor/test`,
+  `/tool-inspection/rules` and `/tool-inspection/test`, which are gone.
+- **Audit events.** Every rule that matches writes one event per
+  request: `gateway.content_flagged`, `gateway.content_stripped` and
+  `gateway.content_blocked`; `gateway.redaction_flagged` and
+  `gateway.redaction_replaced`; `gateway.tool_call_flagged` and
+  `gateway.tool_call_blocked` as before. `gateway.hidden_text_flagged`
+  and `gateway.hidden_text_blocked` are gone; hidden characters are
+  content events. **No event carries the request's text**: a content
+  event names the rule, the outcome, how many matches and whether they
+  were in a tool result; a redaction event names the rule and counts
+  the values and their occurrences, and only a built-in rule's lists a
+  few in masked form (`sk-an…7f9c`) — a custom rule's values are not
+  written at all. A request writes at most 20 events per guard (the
+  rules that changed it, or matched most, first), each with the number
+  of rules that matched (`rules_in_request`). With
+  `audit.body_redact_pii` on, captured bodies are redacted with the
+  outbound redaction rules, built-in ones included, whatever the
+  redaction mode.
+- **Metrics renamed.** `gateway_hidden_text_total` is gone: hidden
+  characters count in `gateway_content_matched_total{outcome,custom}`
+  with every content rule. `pii_pattern_invalid_total{pattern}` is now
+  `guard_policy_invalid_total{guard}` (a stored policy with a rule that
+  does not compile; the rule is left out), next to
+  `guard_policy_unreadable_total{guard}` (a stored policy that cannot be
+  read; the guard runs on its factory policy). Redaction counts values
+  in `gateway_redaction_found_total{kind,outcome}`.
+
+### Added
+
+- **Deleting what a content rule matches.** A content rule can refuse
+  the request, delete the matched text from the caller's messages and
+  tool results and send the rest, or only record. Text deleted joins
+  back what it separated, so the request is checked again afterwards.
+- **Code point rules.** A content rule can match characters by code
+  point (`U+200B`, `U+E0000–U+E007F`), for invisible characters a
+  keyword cannot be written for.
+- **Every rule visible and switchable**, built-in and custom, in each
+  guard, with what it does in the third mode and what it did out of the
+  box; a sample can be tried against one rule, an unsaved one, or all
+  of them.
+
+### Changed
+
+- **Core crates at ThinkWatch-Core v0.59.0.** `tw-dialect`, `tw-guard`,
+  `tw-breaker` and `tw-bedrock` move from v0.55.0; the shared guard model
+  described above comes with them.
+
+### Fixed
+
+- **A credential in a matched tool call no longer reaches the audit
+  log.** The excerpt of a tool call that inspection cut or recorded is
+  masked with the redaction rules before it is written; 2.2 stored the
+  matched arguments as they were, a key the model wrote in them
+  included.
+- **A request's audit events and its log row carry the same id** when
+  the caller sends no `x-trace-id`. The log row of a request that went
+  through used to carry a second, unrelated id.
+
 ## [2.2.0] — 2026-10-01
 
 This release fixes authorization. The gateways never checked
@@ -907,7 +1142,8 @@ unreleased builds should: stop the gateway, run `db/schema.sql`
 against PostgreSQL, restart against this tag. The schema is
 idempotent end-to-end, so the apply is safe to repeat.
 
-[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v2.2.0...HEAD
+[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.0.0...HEAD
+[3.0.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.0.0
 [2.2.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v2.2.0
 [2.1.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v2.1.0
 [2.0.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v2.0.0