feat: evolve error grouping with normalization, cause-chain fingerprints and persistent history

[AllenMuu]

Background

The current StackWatch pipeline already has the right high-level shape:

ErrorEvent
  -> Fingerprinter
  -> L1 fingerprint cache
  -> L2 vector similarity
  -> L3 LLM RCA

The existing Fingerprinter already does useful deterministic work such as filtering framework frames, removing line-number volatility through Class#method, using SHA-256, and versioning fingerprints.

However, several gaps prevent L1 from acting as a durable error identity / aggregation layer:

1. Normalization is implicit inside StackFrame / Fingerprinter instead of being an explicit preprocessing stage.
2. ErrorEvent only carries the outer exception and outer stack; Caused by / root-cause chains are lost.
3. L1 history is currently backed by Caffeine TTL cache, so restart / expiry loses exact-match history.
4. L1 cache hits reuse RCA but do not update memberCount / lastSeen / occurrence history.
5. Application-frame detection relies mainly on framework denylist; third-party libraries can be misclassified as business frames.
6. Message content is currently excluded from exact fingerprints. This avoids volatile IDs, but can over-merge different exceptions thrown from the same location.
7. L2 persistence still depends on in-memory cluster state for actual cosine matching, so historical grouping is not fully durable across restart.

The goal of this issue is to make the deterministic path a stable, explainable and persistent identity layer before vector similarity and LLM reasoning.

---

Target architecture

Error source
   |
   v
ErrorEvent                 // raw facts
   |
   v
ErrorNormalizer            // deterministic normalization
   |
   v
NormalizedError
   |
   v
Fingerprinter V2
   |
   +--> strict fingerprint exact match
   |       |
   |       +--> record occurrence + reuse RCA
   |
   +--> loose fingerprint exact match
   |       |
   |       +--> candidate merge / grouping
   |
   v
Vector similarity
   |
   v
LLM RCA

Principle:

• Fingerprint = deterministic error identity / deduplication
• Vector = semantic similarity / nearby error family
• LLM = root-cause reasoning

---

P0 — Preserve the full cause chain in ErrorEvent

Problem

Both the Logback IThrowableProxy path and direct Throwable path currently only extract the outer exception.

For example:

RuntimeException
  at OrderService...
Caused by: RedisCommandTimeoutException
  at Redisson...

StackWatch should not fingerprint this only as RuntimeException.

Proposed model

Introduce a structured throwable representation:

record ThrowableInfo(
    String type,
    String message,
    List<RawStackFrame> frames,
    ThrowableInfo cause
) {}

Update ErrorEvent conceptually toward:

record ErrorEvent(
    String eventId,
    String appName,
    String env,
    String release,
    Instant occurredAt,
    ThrowableInfo exception,
    String loggerName,
    String threadName,
    String traceId,
    String spanId,
    Map<String, String> attributes
) {}

The exact field expansion can be incremental, but the cause chain is required.

Collection rules

• Logback path: recursively traverse IThrowableProxy#getCause().
• Direct Throwable path: recursively traverse Throwable#getCause().
• Preserve raw values; do not normalize in the collector.
• Add recursion depth / cycle safety.

---

P0 — Add explicit ErrorNormalizer

Normalization should be a first-class preprocessing stage instead of being embedded in the hash generator.

Introduce:

record NormalizedError(
    String outerExceptionType,
    String effectiveExceptionType,
    String normalizedMessage,
    String normalizedRootCauseMessage,
    List<NormalizedFrame> applicationFrames,
    List<NormalizedFrame> rootCauseFrames,
    int causeDepth
) {}

Suggested components:

preprocess/
  ErrorNormalizer
  CauseResolver
  MessageNormalizer
  StackFrameNormalizer
  Fingerprinter

Fingerprinter should hash an already-normalized structure and should not own parsing / classification policy.

---

P0 — Cause resolution rules

Resolve the effective exception before fingerprinting.

Suggested policy:

outer exception
   -> cause
   -> cause
   -> deepest meaningful cause

Known wrapper types may be skipped when selecting the effective type, e.g.:

CompletionException
ExecutionException
InvocationTargetException
UndeclaredThrowableException

Fingerprint selection priority:

1. root-cause application frames
2. outer application frames
3. root-cause top frames
4. outer top frames

Do not immediately fall back from too few application frames to arbitrary outer framework frames if a more meaningful root-cause frame set exists.

---

P0 — Message normalization

Do not hash the raw exception message directly.

Normalize volatile tokens into placeholders, for example:

Order 981273 not found
-> Order <NUM> not found

requestId=550e8400-e29b-41d4-a716-446655440000
-> requestId=<UUID>

Failed at 2026-08-29T10:22:12
-> Failed at <TIME>

Initial normalization candidates:

• UUID -> <UUID>
• IPv4 / IPv6 -> <IP>
• timestamps / dates -> <TIME>
• long hex / hashes -> <HEX>
• long business IDs -> <NUM>
• selected URL query values -> <VALUE>
• repeated whitespace -> single whitespace

Do not blindly replace every number.

Preserve semantic codes where possible, including:

• HTTP status
• SQLState
• errno / DB error code
• gRPC status
• stable application error codes

The implementation should be deterministic, testable and versioned.

---

P0 — Application frame detection: prefer allowlist

Current framework-prefix denylist is useful as a fallback, but it can treat third-party libraries such as Hikari, MySQL, Redisson, Jackson or MyBatis as application frames.

Add configuration such as:

stackwatch:
  fingerprint:
    application-packages:
      - com.mycompany
      - cn.mycompany

Selection rule:

configured application package match
  -> application frame
else
  -> library/framework frame

Keep the denylist only as a fallback when no application package is configured.

---

P0 — Fingerprint V2

Keep V1 compatibility and introduce V2 rather than silently changing existing hashes.

Strict fingerprint

Purpose: identify the same deterministic error.

Canonical input proposal:

version=v2
app=order-service
type=java.lang.NullPointerException
message=cannot invoke <VALUE> because <VALUE> is null
frame=com.foo.OrderService#create
frame=com.foo.OrderFacade#submit
frame=com.foo.OrderController#create

Then:

SHA-256(canonical input)

Recommended strict inputs:

appName
+ effective/root-cause exception type
+ normalized message template
+ top N normalized application frames

Loose fingerprint

Purpose: catch same exception/call-path variants without relying on semantic vector search yet.

Recommended input:

appName
+ effective/root-cause exception type
+ top N normalized application frames

No normalized message.

Scope rules

• Include appName in the fingerprint identity.
• Do not include env in the fingerprint identity; keep environment as an aggregation dimension.
• Keep FingerprintVersion in persistence / lookup keys.
• Keep user-facing fingerprint record parts for explainability.

---

P0 — Persistent fingerprint history / ErrorGroup

Caffeine must remain an accelerator, not the source of truth.

Introduce a persistent repository, e.g.:

interface ErrorGroupRepository {
    Optional<ErrorGroup> findByFingerprint(String appName,
                                           FingerprintVersion version,
                                           String fingerprint);

    ErrorGroup create(...);

    ErrorGroup recordOccurrence(...);
}

Suggested persisted error_group fields:

id
app_name
fingerprint
fingerprint_version
loose_fingerprint
exception_type
root_cause_type
message_template
first_seen
last_seen
occurrence_count
cluster_id
analysis_id
created_at
updated_at

Recommended uniqueness:

UNIQUE(app_name, fingerprint_version, fingerprint)

Lookup path:

Caffeine / Redis
   -> miss
persistent ErrorGroupRepository

Cache may expire. Error identity must not.

---

P0 — Record every occurrence on L1 hit

Current exact-cache-hit behavior should not return immediately without updating the group.

Expected behavior:

exact fingerprint hit
  -> occurrence_count += 1
  -> last_seen = event.occurredAt
  -> optionally persist / sample the occurrence
  -> reuse prior RCA

This should make the following correct:

first event  -> count 1
second event -> count 2
third event  -> count 3

regardless of whether the RCA came from cache.

---

P1 — Add ErrorOccurrence

Separate the durable error identity from individual events.

Concept:

ErrorGroup      = what error is this?
ErrorOccurrence = this concrete occurrence of that error

Suggested occurrence fields:

id
group_id
event_id
occurred_at
env
trace_id
span_id
exception_message
raw_stack_trace

Retention may be sampled / capped later; group counters must remain accurate.

This will support:

• first seen / last seen
• frequency trends
• weekly digest
• environment breakdown
• trace drill-down

---

P1 — Durable L2 history

The current PgVector implementation still uses an in-memory clusterIndex for actual cosine matching. After restart, saved vector documents do not reconstruct the full historical matching index.

After the deterministic layer is fixed, evolve L2 so historical cluster similarity survives restart.

Options may include:

• true pgvector nearest-neighbor query
• repository-level vector lookup independent of process memory
• startup reconstruction only as an interim fallback

This is intentionally lower priority than persistent L1 identity.

---

Proposed runtime flow

ErrorEvent
   |
   v
ErrorNormalizer.normalize(event)
   |
   v
NormalizedError
   |
   v
Fingerprinter.generateV2(normalized)
   |
   +--> strict exact lookup
   |       |
   |       +--> hit -> record occurrence -> reuse RCA
   |
   +--> loose exact lookup
   |       |
   |       +--> optional candidate grouping policy
   |
   v
Embedding / vector similarity
   |
   +--> hit -> merge group/cluster -> persist occurrence
   |
   v
LLM RCA
   |
   +--> create new cluster + ErrorGroup + first occurrence

---

Suggested implementation order

• P0: introduce ThrowableInfo and collect full cause chain
• P0: introduce NormalizedError
• P0: extract CauseResolver
• P0: extract StackFrameNormalizer
• P0: implement deterministic MessageNormalizer
• P0: add configurable application-packages allowlist
• P0: implement strict / loose Fingerprint V2 while keeping V1 compatibility
• P0: introduce persistent ErrorGroupRepository
• P0: make Caffeine / Redis a cache in front of persistent fingerprint history
• P0: update L1 hit path to record occurrence_count / last_seen
• P1: introduce persisted / sampled ErrorOccurrence
• P1: make vector-cluster history queryable after restart
• P2: define fingerprint V1 -> V2 migration / dual-read strategy

---

Acceptance criteria

Cause chain

• Two outer RuntimeExceptions with different meaningful root causes produce different strict fingerprints when appropriate.
• Equivalent exceptions wrapped by CompletionException / ExecutionException resolve to the same effective underlying type and fingerprint when stacks/messages normalize identically.

Stack normalization

• Line-number-only changes do not change fingerprints.
• Framework/library version stack churn does not change fingerprints when business frames are stable.
• Configured application package allowlist takes priority over generic denylist behavior.

Message normalization

• Order 12345 not found and Order 98765 not found can produce the same normalized message template.
• Stable semantic error codes such as HTTP 401 vs 500 or SQLState values are not accidentally collapsed.

Persistence

• Exact fingerprint matches still work after process restart.
• Exact fingerprint matches still work after cache TTL expiry.
• Caffeine/Redis cache eviction does not destroy grouping history.

Occurrences

• Every L1 exact hit increments group occurrence count.
• Every L1 exact hit updates lastSeen.
• firstSeen remains unchanged after subsequent occurrences.

Compatibility

• Existing V1 fingerprints remain readable during migration.
• V2 fingerprints are explicitly versioned.
• Existing L2/L3 behavior continues to work when no exact deterministic match exists.

---

Non-goals for this issue

• Replacing the LLM RCA pipeline.
• Designing automatic remediation / runbook execution.
• Performing semantic clustering entirely in the normalization layer.
• Treating vector similarity as a replacement for deterministic fingerprinting.

The intent is to strengthen the deterministic foundation so later semantic grouping and RCA operate on stable, durable error identities.

[AllenMuu]

Design decisions locked (2026-09-04)

The following decisions resolve the open design forks before implementation.

1. Delivery boundary: this issue implements the P0 deterministic identity layer only. Deferred P1/P2 work is tracked by child follow-up: durable occurrence history, restart-safe L2, and V1 migration #6.
2. Persistence: add an explicit stackwatch.error-history.enabled feature flag. When enabled, PostgreSQL/Flyway is the ErrorGroup source of truth; Caffeine remains only an accelerator. The default zero-infrastructure startup behavior remains unchanged.
3. Cause-chain API: add an optional nested ThrowableInfo exception payload while retaining legacy outer exception fields. Requests that provide both forms are rejected. All collector paths produce the raw structured form before preprocessing.
4. Cause resolver: follow only the primary cause chain; identity-cycle guard and maximum depth 32; skip declared wrapper types; select the deepest non-wrapper with a type. Frame priority is effective-cause app frames, outer app frames, effective-cause raw frames, then outer raw frames. Suppressed exceptions are preserved but do not affect V2.
5. Fingerprints: strict V2 uses app name, effective type, normalized message, and normalized application frames. Loose V2 excludes the message and is a candidate signal only: it must not automatically reuse RCA or merge occurrence counts.
6. Normalization/config: message rules and their ordering are versioned and regression-tested; semantic HTTP/SQLState/errno/gRPC codes are retained. A configured application-package allowlist takes priority; denylist fallback applies only when it is absent.
7. Correct counting: use the composite identity (appName, fingerprintVersion, strictFingerprint) everywhere. P0 accepts an optional event ID and persists a minimal idempotency key, so retry duplicates do not increment the group. Updates are atomic: firstSeen=min, lastSeen=max, and the count changes once per accepted event.
8. Compatibility: new writes are V2; lookup is V2-primary then V1 read-only. V1 hits are not silently migrated or merged. The observability and retirement/backfill plan is deferred to follow-up: durable occurrence history, restart-safe L2, and V1 migration #6.

Child issue: #6.