Conversation checks and opt-in Claude replies for follow-ups (0.3.5)

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "jevpromptcoach",
   "displayName": "Jev (Prompt Coach)",
-  "version": "0.3.0",
+  "version": "0.3.5",
   "description": "Jev (Prompt Coach) scores how well you prompt a coding agent and shows your habits improving over time. Runs on TypeSafe's Jev model.",
   "license": "MIT",
   "keywords": [

.gitignore

@@ -5,7 +5,9 @@ node_modules/
 # Eval fixtures are real prompts from real work. They stay on the machine that
 # made them. See test/fixtures/README.md.
 test/fixtures/prompts.json
+test/fixtures/conversations.json
 test/eval-raw.json
+test/eval-conversations-raw.json
 
 # No lockfile at the plugin root, deliberately.
 #

CLAUDE.md

@@ -41,6 +41,18 @@ easy to break with a change that looks reasonable.
   test asserts it on the wire.
 - **`metadata_only` means no text leaves the machine.** Any new code path that
   sends text must check for it.
+- **Claude's replies are filtered in `src/conversation.ts`.** Only the closing
+  visible text of a turn is read; tool calls, tool output, thinking and
+  subagent records never leave that module, and an exchange whose prompt was
+  bypassed with `*` is dropped with its reply. Replies are on by default and go
+  through `applyPrivacy` like prompts; `JEVPROMPTCOACH_SESSION_REPLIES=0` turns
+  them off. The wire tests cover each filter.
+- **Conversation checks are calibrated separately.** Each check's
+  `conversation` block in `src/checks.ts` has its own threshold and inline
+  eligibility, set from `npm run eval -- --conversations` on the gitignored
+  `test/fixtures/conversations.json`. Only constraints and verification are
+  inline-eligible so far; the labels behind them were set by the agent, not a
+  person, and the set is 40 follow-ups, so treat them as provisional.
 - **`dist/` is committed and must match `src/`.** Claude Code installs with
   `--ignore-scripts`, so nothing is ever built at install time. Run
   `npm run build` after any source change; CI fails if it drifts.

README.es.md

@@ -235,6 +235,16 @@ misma sesión, y solo se puntúa el nuevo. Activado por defecto;
 `JEVPROMPTCOACH_SESSION_CONTEXT=0` lo desactiva. Los umbrales se calibraron con
 prompts puntuados solos.
 
+**También las respuestas de Claude.** Un seguimiento se envía con los dos últimos
+intercambios (tus prompts y el texto final de las respuestas de Claude, nunca las
+herramientas ni su salida), y cada comprobación se hace en su forma
+conversacional. Un intercambio saltado con `*` se descarta junto con su
+respuesta. Las respuestas pasan por la misma depuración que tus prompts.
+Activado por defecto; `JEVPROMPTCOACH_SESSION_REPLIES=0` envía solo tus prompts
+anteriores. Por ahora solo dos
+comprobaciones conversacionales se muestran en línea: lo que no debe cambiar y
+los pasos de verificación.
+
 ## Privacidad
 
 Los prompts contienen código, rutas y a veces secretos.
@@ -249,9 +259,16 @@ de tu máquina salvo durante un comando que tú hayas ejecutado.
 | `raw` | El texto tal cual. Las cadenas con forma de credencial se eliminan **igualmente**. |
 
 Se eliminan en todos los niveles, incluido `raw`: `sk-`, `sk-ant-`, `sk-proj-`,
-`ghp_` y similares, `AKIA`/`ASIA`, `AIza`, los `xox*` de Slack, JWT, bloques
-PEM, secretos de cliente de Azure, tokens `Bearer`, y cualquier cosa asignada a
-un nombre que acabe en `KEY`/`TOKEN`/`SECRET`/`PASSWORD`.
+`ghp_` y similares, `github_pat_`, `AKIA`/`ASIA`, `AIza`, los `xox*` de Slack,
+Stripe `sk_live_`/`rk_live_`, `npm_`, SendGrid `SG.`, URL de webhooks de Slack y
+Discord, JWT, bloques PEM, secretos de cliente y firmas SAS de Azure, tokens
+`Bearer`, la contraseña de cualquier URL `esquema://usuario:contraseña@host`,
+todo lo etiquetado como `password`, y cualquier cosa asignada a un nombre que
+acabe en `KEY`/`TOKEN`/`SECRET`/`PASSWORD` o `_PASS`/`_PWD`/`_AUTH`. Sin prefijo
+ni etiqueta, dos formas se eliminan igualmente: cualquier secuencia de 16 o más
+caracteres hexadecimales pasa a `[HEX]`, y un token de aspecto aleatorio de 20 o
+más caracteres pasa a `[KEY]`. Solo un secreto que no es hexadecimal ni aleatorio
+y no lleva etiqueta, como una contraseña con forma de palabra, no se elimina.
 
 **Qué se envía exactamente, y cuándo:**
 
@@ -261,6 +278,7 @@ un nombre que acabe en `KEY`/`TOKEN`/`SECRET`/`PASSWORD`.
 | `/jevpromptcoach:report` | Los prompts registrados sin puntuar, depurados, por lotes |
 | `config backfill` | Tu historial, depurado, por lotes — **después** de una estimación de coste y una confirmación explícita |
 | modo `always` | Cada prompt al enviarlo, depurado, más hasta dos prompts anteriores de la misma sesión como contexto, también depurados |
+| modo `always`, respuestas de Claude | Por defecto, el contexto son los dos últimos intercambios: tus prompts y el texto final de las respuestas de Claude, depurados. `JEVPROMPTCOACH_SESSION_REPLIES=0` quita las respuestas |
 | En cualquier otro momento | Nada |
 
 Sin telemetría. Sin ningún otro destino de red. La clave de API se lee del

README.fr.md

@@ -244,6 +244,15 @@ même session, et seul le nouveau est noté. Activé par défaut ;
 `JEVPROMPTCOACH_SESSION_CONTEXT=0` le désactive. Les seuils ont été calibrés sur
 des prompts notés seuls.
 
+**Les réponses de Claude aussi.** Une relance est envoyée avec les deux derniers
+échanges (vos prompts et le texte final des réponses de Claude, jamais les outils
+ni leurs sorties), et chaque vérification est posée dans sa forme
+conversationnelle. Un échange contourné par `*` est retiré avec sa réponse. Les
+réponses passent par le même expurgeage que vos prompts. Activé par défaut ;
+`JEVPROMPTCOACH_SESSION_REPLIES=0` n'envoie que vos prompts précédents. Pour l'instant, seules deux
+vérifications conversationnelles s'affichent en ligne : ce qui ne doit pas changer
+et les étapes de vérification.
+
 ## Confidentialité
 
 Les prompts contiennent du code, des chemins, et parfois des secrets.
@@ -258,9 +267,17 @@ quitte votre machine en dehors d'une commande que vous avez lancée.
 | `raw` | Le texte tel qu'écrit. Les chaînes en forme d'identifiant sont **quand même** retirées. |
 
 Retiré à tous les niveaux, y compris `raw` : `sk-`, `sk-ant-`, `sk-proj-`,
-`ghp_` et apparentés, `AKIA`/`ASIA`, `AIza`, les `xox*` de Slack, les JWT, les
-blocs PEM, les secrets clients Azure, les jetons `Bearer`, et tout ce qui est
-assigné à un nom finissant par `KEY`/`TOKEN`/`SECRET`/`PASSWORD`.
+`ghp_` et apparentés, `github_pat_`, `AKIA`/`ASIA`, `AIza`, les `xox*` de Slack,
+Stripe `sk_live_`/`rk_live_`, `npm_`, SendGrid `SG.`, les URL de webhook Slack et
+Discord, les JWT, les blocs PEM, les secrets clients et signatures SAS Azure, les
+jetons `Bearer`, le mot de passe de toute URL `schéma://utilisateur:motdepasse@hôte`,
+tout ce qui est étiqueté `password`, et tout ce qui est assigné à un nom
+finissant par `KEY`/`TOKEN`/`SECRET`/`PASSWORD` ou `_PASS`/`_PWD`/`_AUTH`. Sans
+préfixe ni étiquette, deux formes partent quand même : toute suite de 16
+caractères hexadécimaux ou plus devient `[HEX]`, et un jeton d'apparence
+aléatoire de 20 caractères ou plus devient `[KEY]`. Seul un secret qui n'est ni
+hexadécimal ni aléatoire et sans étiquette, comme un mot de passe en forme de
+mot, n'est pas retiré.
 
 **Ce qui est envoyé, et quand :**
 
@@ -270,6 +287,7 @@ assigné à un nom finissant par `KEY`/`TOKEN`/`SECRET`/`PASSWORD`.
 | `/jevpromptcoach:report` | Les prompts journalisés pas encore notés, expurgés, par lots |
 | `config backfill` | Votre historique, expurgé, par lots — **après** une estimation de coût et une confirmation explicite |
 | mode `always` | Chaque prompt au moment où vous l'envoyez, expurgé, plus jusqu'à deux prompts précédents de la même session comme contexte, expurgés eux aussi |
+| mode `always`, réponses de Claude | Par défaut, le contexte est les deux derniers échanges : vos prompts et le texte final des réponses de Claude, expurgés. `JEVPROMPTCOACH_SESSION_REPLIES=0` retire les réponses |
 | Sinon, jamais | Rien |
 
 Aucune télémétrie. Aucune autre destination réseau. La clé d'API est lue depuis

README.md

@@ -292,15 +292,31 @@ Guarantees:
   confidence, often three or four of them, so a 0 said less than it looked. When
   none of those pass, the notice shows what is missing and leaves the number off.
 
-**Follow-ups are read in context.** The first prompt of a session is scored on
-its own, because it has to carry everything the agent needs. A later prompt is
-sent with the two prompts before it in the same session, and only the new one
-is scored: "commit and push all changes" is a fine follow-up once the earlier
-prompt said what the change was. This is on by default. Set
-`JEVPROMPTCOACH_SESSION_CONTEXT=0`, in your environment or in
-`~/.claude/jevpromptcoach/.env`, to score every prompt alone. The thresholds
-were tuned on prompts scored alone; follow-up scores have not yet been through
-the eval.
+**Follow-ups are read in conversation.** "Yes, commit it" can only be judged
+against what Claude offered. The first prompt of a session is scored on its
+own, because it has to carry everything the agent needs. A later prompt is sent
+with the conversation before it, up to the last two exchanges: your prompt and
+the closing text of Claude's reply, twice, then the new prompt, with whatever
+is available earlier in a session. Only the new prompt is scored, and each
+check is asked in its conversation form, which gives credit for what the
+conversation already settled, such as accepting a change Claude described in a
+named file.
+
+Only Claude's visible closing text is read, never tool calls, tool output or
+subagent work, and an exchange whose prompt was bypassed with `*` is dropped
+along with its reply. Replies go through the same redaction as your prompts.
+Both are on by default: `JEVPROMPTCOACH_SESSION_REPLIES=0` leaves Claude's
+replies out and sends only your earlier prompts, and
+`JEVPROMPTCOACH_SESSION_CONTEXT=0` scores every prompt alone. Either goes in
+your environment or in `~/.claude/jevpromptcoach/.env`.
+
+Measured on 40 labelled follow-ups (see
+[test/fixtures/README.md](test/fixtures/README.md)), Claude's replies make
+"which file or function" rank noticeably better than judging the follow-up
+alone, and leave the other checks level; your earlier prompts on their own add
+nothing measurable. Only two conversation checks are steady enough to show
+inline so far, what must not change and the verification steps; the rest are
+recorded for the report and stay off the line.
 
 `always` does not use the mechanism the docs suggest. Writing to stderr with a
 non-zero exit displays nothing on Claude Code 2.1.277; a top-level
@@ -324,9 +340,22 @@ your machine except during a command you ran.
 | `raw` | Prompt text as written. Credential-shaped strings are **still** stripped. |
 
 Stripped at every level, including `raw`: `sk-`, `sk-ant-`, `sk-proj-`, `ghp_`
-and friends, `AKIA`/`ASIA`, `AIza`, Slack `xox*`, JWTs, PEM blocks, Azure client
-secrets, `Bearer` tokens, and anything assigned to a name ending in
-`KEY`/`TOKEN`/`SECRET`/`PASSWORD`.
+and friends, `github_pat_`, `AKIA`/`ASIA`, `AIza`, Slack `xox*`, Stripe
+`sk_live_`/`rk_live_`, `npm_`, SendGrid `SG.`, Slack and Discord webhook URLs,
+JWTs, PEM blocks, Azure client secrets and SAS signatures, `Bearer` tokens, the
+password in any `scheme://user:password@host` URL, anything labelled `password`
+(JSON keys and "password is …" included), and anything assigned to a name
+ending in `KEY`/`TOKEN`/`SECRET`/`PASSWORD` or `_PASS`/`_PWD`/`_AUTH`. With no
+prefix and no label, two shapes still go: any run of 16 or more hex characters
+becomes `[HEX]` (commit SHAs too; the marker keeps the fact that an identifier
+was named), and a random-looking token of 20 or more characters becomes
+`[KEY]`.
+
+Redaction works by shape, and that has a limit: a secret that is neither hex
+nor random-looking and carries no label, such as a word-like password on its
+own, is not removed. That applies to your prompts and to Claude's replies
+alike; set `JEVPROMPTCOACH_SESSION_REPLIES=0` if you would rather replies never
+leave the machine.
 
 **Exactly what is sent, and when:**
 
@@ -335,7 +364,7 @@ secrets, `Bearer` tokens, and anything assigned to a name ending in
 | `/jevpromptcoach:score` | The one prompt you passed, redacted |
 | `/jevpromptcoach:report` | Any logged prompts not yet scored, redacted, batched |
 | `config backfill` | Your history, redacted, batched — **after** a cost estimate and an explicit confirmation |
-| `always` mode | Each prompt as you submit it, redacted, plus up to two earlier prompts from the same session as context, redacted again at the current level |
+| `always` mode | Each prompt as you submit it, redacted, plus the conversation before it as context: up to the last two exchanges, your prompts and the closing text of Claude's replies, redacted again at the current level. `JEVPROMPTCOACH_SESSION_REPLIES=0` drops the replies; `JEVPROMPTCOACH_SESSION_CONTEXT=0` drops the context |
 | Ever, otherwise | Nothing |
 
 No telemetry. No other network destination. The API key is read from the