fix(highlights): anchor report comparison sentiment on impact_score formula (GLOOK-20)

[msogin]

Summary

• Fixes hallucinated narrative in the home page report comparison summary where the LLM was labeling raw metric changes as "improved" or "worse" without understanding what each metric means
• The root cause: the system prompt gave no definition of improvement — the LLM guessed direction naively (e.g., complexity dropping from 3.1→3.0 labeled "improved" even though complexity is not independently directional)

Changes

Prompt rewrite (prompts/report-highlights-system.txt): adds the PERFORMANCE FORMULA definition so the LLM understands impact_score is the only authoritative measure, and restricts positive/warning sentiment to impact_score movement only. All other metrics (commits, complexity, AI%, lines) must be described neutrally.

Data (service.ts): adds avg_impact to both report totals so the LLM has the aggregate performance signal to anchor on.

Cache guard (service.ts): adds _v: 2 versioning to the report_comparisons cache. Rows stored with the old prompt (bare array format) are transparently treated as stale and regenerated on the next page load.

Test plan

• All 742 unit tests pass
• Home page comparison summary uses neutral language for complexity/commit/AI% changes
• Positive/warning sentiment only appears when impact_score moved in that direction
• Old cached comparison rows regenerate automatically (not served as stale)

🤖 Generated with Claude Code

[msogin]

Claude Code Review

Reviewed by two parallel personas (Sr. Software Engineer + LLM Systems & Data Integrity Engineer) via review-loop + Smartling claude-pr-review-local.

---

🟡 Important — service.ts:~46 — Missing try/catch around JSON.parse

A corrupt or truncated report_comparisons row throws a SyntaxError that surfaces as an unhandled 500, bypassing the stale-cache fallthrough that was just added. Parse failure should be treated identically to _v !== 2 — fall through to regenerate.

let data: any = null;
try {
  data = typeof cached[0].highlights_json === 'string'
    ? JSON.parse(cached[0].highlights_json)
    : cached[0].highlights_json;
} catch { /* malformed — fall through to regenerate */ }
if (data && !Array.isArray(data) && data._v === 2) {

---

🟡 Important — service.ts:~93,101 — avgImpact headcount dilution skews the exclusive sentiment gate

avgImpact is computed as an arithmetic mean across all developers. When headcount grows between periods, the average drops from dilution alone — even if every existing developer improved. The prompt now gates "warning" sentiment exclusively on avg_impact, so a net-positive period with one new low-scoring developer can produce a spurious "warning" bullet.

Options: (a) filter to same-developer cohort for the avg, (b) pass both cohort-avg and total-avg to the LLM, or (c) add a note to the prompt acknowledging headcount-driven shifts. This is a conscious design decision — flagging it so the choice is intentional.

---

🟡 Important — prompts/report-highlights-system.txt:5 — Max: 9.3 is inconsistent

The formula coefficients sum to 10.3 at max-normalized inputs (2.0 + 2.7 + 3.5 + 1.1 + 0.5 + 0.5). The user message sends scores like impact=80.0 and avg_impact=65.00, which are also inconsistent with a theoretical max of 9.3. The LLM has no coherent frame for calibrating score magnitudes. Either document the normalization assumptions (inputs are pre-scaled 0–1 proportions? scores are ×100?) or remove the Max: 9.3 claim to avoid confusing the model.

---

🟡 Important — service.ts — Deploy burst: no migration for stale rows

All existing report_comparisons rows in the old bare-array format will fall through to LLM regeneration on first access post-deploy. For orgs with many cached report pairs this is a synchronous LLM burst. Consider running a backfill migration before deploying (UPDATE report_comparisons SET highlights_json = JSON_OBJECT('_v', 2, 'highlights', highlights_json) WHERE ...) or document the expected blast radius.

---

🟣 Question — prompts/report-highlights-system.txt — "or" in warning rule

- "warning": use ONLY when avg_impact or individual impact_score declined.

This means a single developer's decline triggers "warning" even when the org average improved. Is this intentional? If so, what should the LLM do when individual decline contradicts org-level improvement? No rule currently disambiguates this case.

---

🟣 Question — service.ts:~52–60 — Forward-compatibility on unknown _v

When data._v is present but not 2 (e.g., _v: 3 written by a future version, then the code rolls back), the current code falls through to regenerate indefinitely on every request until the row is overwritten. Is silent-regenerate the intended behavior for unknown future versions, or should this be guarded?

---

🟣 Question — snapshot — rank #undefined is pre-existing

The snapshot shows rank #undefined for all developers in TOP5_A/TOP5_B. formatDev is called on raw statsA/statsB rows which have no rank property — rank is only attached inside mapA/mapB. The snapshot currently asserts this broken output. Not introduced by this PR, but the snapshot was regenerated here, so this is the natural moment to fix it.

---

🔵 Suggestion — service.ts:~50 — Simplify destructuring

// before
const { _v: _, highlights } = data;
// after
const highlights = data.highlights;

---

🔵 Suggestion — service.ts — Extract cache version to a constant

2 is repeated in both the read check and the write path. Extract: const HIGHLIGHTS_CACHE_VERSION = 2;

---

📁 Note — docs/superpowers/ files

docs/superpowers/plans/2026-06-16-glook-20-highlights-prompt.md (268 lines) and docs/superpowers/specs/2026-06-16-glook-20-highlights-prompt-design.md (91 lines) are point-in-time planning artifacts. The content is already captured in this PR description and GLOOK-20. These will drift from reality immediately — worth removing before merge.

---

Ready to merge? With fixes — the JSON.parse try/catch is a real production reliability risk, and the avgImpact dilution issue can produce systematically incorrect sentiment given the prompt's exclusive reliance on it.

🤖 Generated with Claude Code