DLP redact patterns corrupt the JSON request body: match spans are computed on unescaped text (upstream 400 'invalid escape')

[Spymack]

DLP request-body redaction corrupts the JSON: match spans computed on unescaped text

Package: @clawshell/clawshell 0.2.1 (native binary clawshell-linux-x64)
Area: [dlp] redaction of the request body (src/dlp.rs, src/app.rs)
Severity: request-loss — every affected request is rejected by the upstream, and a retrying client turns it into a multi-minute stall

Summary

When [dlp] is configured with action = "redact" patterns, ClawShell rewrites the
request body before forwarding. The rewrite breaks the JSON whenever a redacted match
sits immediately after a JSON escape sequence: the offending escape's last character is
consumed, leaving a bare backslash, and the upstream rejects the whole request:

Upstream 400: Failed to parse the request body as JSON:
  messages[17].reasoning_content: invalid escape at line 1 column 137594

(Message wording/column are from an axum + serde_path_to_error upstream; any strict JSON
parser reports the same class of error.)

Configuration that triggers it

[dlp]
scan_responses = true
patterns = [
  { name = "email",    regex = '\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b', action = "redact" },
  { name = "phone_fr", regex = '(?:\+33|0[1-9])(?:[\s.\-]?\d{2}){4}\b',               action = "redact" },
  # nir_fr / iban_eu / card patterns likewise
]

Minimal, deterministic reproduction

Send an OpenAI-compatible POST /v1/chat/completions body in which one message string
contains an e-mail address at the start of a line, so that the serialized JSON contains
an escaped newline (backslash + n) immediately followed by the address.

Result matrix (ClawShell 0.2.1, action = "redact"):

Body content	Result
e-mail in the middle of a line (preceded by a space)	200 OK, no corruption
e-mail at the very start of the string	200 OK, no corruption
e-mail immediately after an escaped newline	upstream 400 — invalid escape
25 e-mails, each after an escaped newline, no other escape in the text	upstream 400 — invalid escape
phone number / IBAN after an escaped newline	200 OK, but the character before the match is silently eaten (text mangled)
body without any PII pattern match	200 OK

Log lines on the failing request:

INFO clawshell::app: Sensitive data detected in request, PII redacted from request body before forwarding
INFO clawshell::app: Forwarding request to upstream
INFO clawshell::app: Request completed ... status=400 Bad Request latency_ms=627

Note the 400 arrives within a second: the corruption is in the body ClawShell produced,
not an upstream timeout. Removing the [dlp] patterns makes every one of the cases above
return 200.

Likely cause

The match spans used by the redaction appear to be computed on the unescaped body (or
with each escape sequence counted as a single character) and then applied to the escaped
body that is actually forwarded. Every escape sequence occurring before a match therefore
shifts the span by one character: when the match is adjacent to an escape, the replacement
range starts one byte early and eats the escape's last character, leaving an invalid escape
sequence (\[, \", \u... fragments). The invalid match span assertion in src/dlp.rs
looks like the same invariant, hit from the other side.

Because the shift is proportional to the number of preceding escapes, the damage is silent
for text-only payloads and destructive for payloads that are long and escape-dense — e.g.
reasoning_content from a reasoning model, which contains newlines plus model-generated
e-mail/host-like tokens.

Suggested fix

Redact the deserialized values (walk the parsed JSON strings, then re-serialize with a real
JSON writer), or compute the spans on exactly the representation you slice. Either removes
the whole class of bug; the current "scan the raw body" approach cannot be correct for
escape-sensitive patterns.

Workaround in production

We neutralized the section (patterns = [], scan_responses = false) and moved PII
redaction to our own JSON-aware proxy in front of ClawShell. That restored normal operation
immediately (verified: the reproductions above all return 200).