diff --git a/.changeset/feedback-survey-0-11-3.md b/.changeset/feedback-survey-0-11-3.md new file mode 100644 index 00000000..78d837f9 --- /dev/null +++ b/.changeset/feedback-survey-0-11-3.md @@ -0,0 +1,5 @@ +--- +"@taskless/cli": patch +--- + +The feedback survey is replaced for 0.11.3. Only the kind of rule the user was trying to create is required now; the user's own words, the completion verdict, what worked, what did not, the agent in use, and the most valuable rule so far are all optional. A `skip` at the invite no longer dismisses the survey: the agent sends its own account of the session and leaves the user's words out. `feedback dismiss` is reserved for a user who asks that nothing be sent. Every install is invited once more, since the new survey keeps its own cadence. diff --git a/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/design.md b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/design.md new file mode 100644 index 00000000..23c38b6c --- /dev/null +++ b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/design.md @@ -0,0 +1,71 @@ +## Context + +The first survey (`01a0b1a0-…`) required `verbatim`, `goal`, and `completed`. Measured against PostHog's definition on 2026-09-21: Q1 and Q3 carried no `optional` field (PostHog's default is required), Q2 was `optional: false`, Q4 and Q5 `optional: true`. The CLI's Zod schema matched that. The consequence in practice: a user who replied `skip` produced a `survey dismissed` and nothing else, even though the agent had four answers ready that needed no one's permission. + +The replacement (`01a0c7b9-…`, "Product-market fit (PMF) (0.11.3)", started 2026-09-22) has seven questions. Only Q1 is required. + +| # | id | question | optional | +| --- | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | -------- | +| 1 | `0874591f-c554-4ac3-8930-e11c436d859e` | What kind of rule was the user trying to create? | no | +| 2 | `2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2` | Did the user offer any comments? (leave blank if no comments) | yes | +| 3 | `605e12a8-82b6-480f-93b2-ab8de0fa08bd` | Did the user successfully complete the task in your opinion? (`Yes`/`No`/`Unknown`) | yes | +| 4 | `b5375d87-e295-4833-84ed-fca8140ba992` | What steps of the interaction with the Taskless skills & CLI worked well? | yes | +| 5 | `a8cf706d-3ff7-4845-bea9-501013be958c` | What steps of the interaction with Taskless skills & CLI could use improvement? | yes | +| 6 | `f85b22df-8e51-4c9c-8219-261b33b71c90` | What agent(s) or framework(s) is the user using that are open sourced and publicly available? | yes | +| 7 | `4f8e938e-22f6-449c-9b8c-43c51d08e214` | Of the rules created so far, what rule is creating the most value for the team and why? (leave blank if there are no rules) | yes | + +Required-ness is enforced by PostHog's popover UI only; the capture API accepts a `survey sent` with any subset of `$survey_response_*` keys. The CLI's schema is therefore the only gate, and it mirrors PostHog's definition so that the responses view and the CLI agree on what a complete answer is. + +## Decisions + +### Human keys + +| key | question | required | +| ------------------ | -------- | -------- | +| `ruleKind` | 1 | yes | +| `verbatim` | 2 | no | +| `completed` | 3 | no | +| `workedWell` | 4 | no | +| `needsImprovement` | 5 | no | +| `agents` | 6 | no | +| `mostValuableRule` | 7 | no | + +`ruleKind` rather than `goal`: the question narrowed from "what were they trying to accomplish" to "what kind of rule", and the old key would invite the old answer. `agents` is plural because the question is. `mostValuableRule` is the question's noun phrase. + +The map stays the only place ids live, and `buildSurveyResponse` stays a loop over it, so the command does not change. + +### `ruleKind` on the onboarding path + +`onboard` is surveyed and creates no rule. The recipe tells the agent to answer `none (onboarding)` there, and otherwise to name the engine and what the rule was for in a phrase (`ast-grep, forbid eval in TypeScript`). The value is free text on PostHog's side; the recipe supplies the shape, the schema only requires non-empty. + +### `skip` sends; an explicit refusal dismisses + +Six of the seven answers are the agent's own account of the session. That is the same category of data as `cli_rule_create` or `cli_check`: telemetry the user opted into by leaving it enabled, and the gate already withholds the invite under `DO_NOT_TRACK`. The only answer that is the user's is the quote, so the invite asks for exactly that and offers exactly two replies: + +> Taskless would like to know how this went. Anything you'd like to add in your own words? Reply `skip` if not, and I'll send my own notes on the session. + +- words → `verbatim` is those words, unedited, and the agent sends; +- `skip`, silence, or an unrelated reply → `verbatim` is omitted and the agent sends the rest. + +There is no third option in the invite, so no reply the invite solicits produces nothing. `feedback dismiss` is kept for the reply it does not solicit: a user who says not to send anything ("no", "don't send that") is asking for the response not to exist, and the CLI honours that and records it as a dismissal. The recipe tells the agent, in that case, to say in one line that the rest of the CLI's telemetry is governed by `DO_NOT_TRACK=1` / `TASKLESS_TELEMETRY_DISABLED=1`, so the user learns where the real switch is rather than being offered a per-survey one that covers one capture in twenty. + +### No follow-up question + +The old recipe permitted one follow-up, to fill `completed` when the reply left it unclear. `completed` is optional now, and `Unknown` exists for exactly that state, so the follow-up goes. The invite is the only question the flow puts to the user. + +### Filling `agents` and `mostValuableRule` + +`agents`: the agent names the harness it is running in and any framework it can see the project using, restricted to open-source, publicly available software as the question asks. It omits the key rather than naming an internal or proprietary tool. No PII is involved; this is the name of a product. + +`mostValuableRule`: the agent answers from the session and from `.taskless/rules/` when it has a view; it omits the key when it does not, and does not ask the user. The question is aimed at a team that has lived with its rules for a while, and most surveyed sessions are the one that just wrote the first one. + +### Cadence + +Unchanged. The store is keyed by survey id, so the new id gets a fresh `next_ask` and every install is invited once more on its next surveyed recipe. That is the intended effect of shipping a new survey and the standing spec already states it. + +## Alternatives considered + +- **Offer `no` in the invite alongside `skip`.** Rejected: it invites a reply that produces nothing from a user who has telemetry enabled, and the agent's six answers are not the user's to withhold any more than the other events are. The refusal path exists for the user who volunteers it, not as a menu item. +- **Remove `feedback dismiss`.** Rejected: an explicit "don't send anything" needs a verb that honours it and records it, and the verb already exists. +- **Keep `goal` as the key and change only its description.** Rejected: the key would be read by an agent that learned the old recipe, and the old answer is not the new question. +- **Send on silence without asking at all.** Rejected: the user's words are the answer the survey exists to collect, and an ask that costs one line is worth it. diff --git a/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/proposal.md b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/proposal.md new file mode 100644 index 00000000..d72de59b --- /dev/null +++ b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/proposal.md @@ -0,0 +1,33 @@ +## Why + +The 0.11.3 release moves the CLI to a new PostHog survey, `01a0c7b9-dfe4-0000-d05e-ce253e90a68c`, which replaces `01a0b1a0-80fb-0000-5dc1-baa4ec44e619`. The new survey was written from what the first one taught us: the questions an agent answers well on its own were required-but-often-guessed, the one question only the user can answer was required and so blocked the whole response when the user said `skip`, and two things we want to know were never asked (which agent harness is in play, and which rule has earned its keep). PostHog assigns a fresh question id per question, so the CLI's key-to-id map must be rebuilt, not edited. + +The survey is now agent-first. Only one question is required, the kind of rule the user was trying to create, and the agent knows that without asking anyone. Everything else is optional, including the user's own words. A user who declines to add anything is therefore no longer a dismissal: the agent still sends its own account of the session. + +## What Changes + +- **`SURVEY_ID` moves to the new survey** and the question map is rebuilt for its seven questions. The cadence store is keyed by survey id, so every install sees the new survey as a fresh ask; that is the behaviour the standing spec already promises. +- **The payload schema changes shape.** `goal` is gone. `ruleKind` (required) replaces it as the one required answer. `verbatim` becomes optional. `completed`, `workedWell`, and `needsImprovement` stay, with `completed` now optional. Two optional keys are new: `agents`, the open-source agent or framework the user is working through, and `mostValuableRule`, the rule that is earning the most for the team and why. +- **A `skip` is no longer a dismissal.** The invite asks for one thing, the user's own words, and offers two replies: words or `skip`. Words go in `verbatim`; `skip`, silence, or an unrelated reply means `verbatim` is omitted, and either way the agent fetches `agent feedback` and sends its own account of the session. The other six answers are the agent's observations, which are telemetry the user has already opted into by leaving it enabled. `feedback dismiss` stays as the explicit opt-out: the invite does not offer it, but a user who says not to send anything gets exactly that, and the recipe tells the agent to point them at the telemetry switch for everything else. +- **The `feedback` and `feedback-invite` recipes are revised** to match: the invite's wording changes, the feedback recipe drops its one permitted follow-up question (it existed only to fill `completed`, which is now optional), and it explains how to fill the two new answers and what `ruleKind` means on the onboarding path. + +Nothing here is **BREAKING**. The payload is read by `feedback send` from a file the same agent wrote seconds earlier from the same CLI's recipe; no consumer holds an old-shape payload across a release. `patch`. + +## Capabilities + +### Modified Capabilities + +- `cli-feedback-survey`: the payload contract (keys, which is required), the invite's reply handling (`skip` sends rather than dismisses), and the recipes' instructions. + +## Impact + +- `packages/cli/src/survey/constants.ts`: new `SURVEY_ID`, new `FeedbackKey` union, new `SURVEY_QUESTIONS` map. +- `packages/cli/src/schemas/feedback.ts`: the Zod schema, which is what the recipe embeds and `feedback send` validates with. +- `packages/cli/src/agent/feedback.md` (topic v2) and `feedback-invite.md` (topic v2). +- `packages/cli/src/commands/feedback.ts`: no logic change; `buildSurveyResponse` iterates the map. +- Tests: `feedback-schema.test.ts`, `feedback-recipes.test.ts`, `feedback-command.test.ts`, `survey-invite.test.ts` where they name keys, ids, or the invite sentence. +- PostHog: `survey shown` / `survey dismissed` / `survey sent` continue under their names with the new `$survey_id`. The old survey keeps its responses; nothing is migrated. + +## Delivery shape + +**Single PR.** The id, the map, the schema, and the two recipes are only correct together, and the whole diff is a few hundred lines. The change archives on the same PR. diff --git a/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/specs/cli-feedback-survey/spec.md b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/specs/cli-feedback-survey/spec.md new file mode 100644 index 00000000..59aa995a --- /dev/null +++ b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/specs/cli-feedback-survey/spec.md @@ -0,0 +1,67 @@ +## MODIFIED Requirements + +### Requirement: The feedback payload uses human keys mapped to survey questions by the CLI + +The feedback payload SHALL be a JSON object with these keys: + +| key | required | value | +| ------------------ | -------- | ----------------------------------------------------------------------------------------------------- | +| `ruleKind` | yes | the kind of rule the user was trying to create, non-empty; `none (onboarding)` on the onboarding path | +| `verbatim` | no | the user's own words, unedited | +| `completed` | no | exactly `Yes`, `No`, or `Unknown` | +| `workedWell` | no | what worked well | +| `needsImprovement` | no | what could use improvement | +| `agents` | no | the open-source, publicly available agent(s) or framework(s) in use | +| `mostValuableRule` | no | the rule creating the most value for the team, and why | + +The CLI SHALL own the map from these keys to the survey's question identifiers; a payload SHALL NOT contain `$survey_*` keys. An omitted optional key SHALL be omitted from the event rather than sent as an empty string, and an optional key that is present SHALL be non-blank. The schema SHALL be embedded in the `feedback` recipe through the recipe input-schema mechanism. + +#### Scenario: Optional answers are omitted, not blanked + +- **WHEN** a payload has no `workedWell` +- **THEN** the `survey sent` event SHALL carry no `$survey_response_` key for that question + +#### Scenario: Completion is one of three literals + +- **WHEN** a payload has `completed: "partially"` +- **THEN** validation SHALL fail naming `completed` + +#### Scenario: The rule kind alone is a complete response + +- **WHEN** a payload is `{ "ruleKind": "ast-grep, forbid eval in TypeScript" }` +- **THEN** validation SHALL pass +- **AND** the `survey sent` event SHALL carry `$survey_id` and exactly one `$survey_response_` key + +#### Scenario: A missing rule kind is rejected + +- **WHEN** a payload carries every optional key and no `ruleKind` +- **THEN** validation SHALL fail naming `ruleKind` + +### Requirement: The feedback and feedback-invite recipes + +The CLI SHALL embed a `feedback` recipe that tells the agent it is the respondent: it records the user's reply verbatim when there is one, fills the remaining answers from its own observation of the session, writes the payload to `.taskless/.tmp-feedback.json`, runs `feedback send --from` that path, and deletes the file afterwards. The recipe SHALL embed the payload schema, SHALL NOT ask the agent to put the survey's questions to the user one by one, and SHALL NOT permit a follow-up question. It SHALL tell the agent to answer `ruleKind` with `none (onboarding)` when the surveyed recipe was `onboard`, to name only open-source, publicly available software in `agents`, and to omit `mostValuableRule` rather than ask the user for it. + +The CLI SHALL embed a `feedback-invite` recipe carrying the text appended to surveyed recipes. Rendered header-less, it SHALL instruct the agent to ask the user exactly once, with the sentence "Taskless would like to know how this went. Anything you'd like to add in your own words? Reply `skip` if not, and I'll send my own notes on the session."; to treat a reply of `skip`, silence, or a reply unrelated to feedback as an omitted `verbatim` and still fetch `agent feedback`; to treat any other reply as the user's words and fetch `agent feedback`; and, only when the user explicitly asks for nothing to be sent, to run `feedback dismiss` and name the telemetry opt-out environment variables in one line. Both recipes SHALL follow the recipe conventions: a `# Topic:` header, the CLI named by its rendered invocation, and commands that exist. + +#### Scenario: The appended invite carries no header + +- **WHEN** the gate is open and a surveyed recipe is served +- **THEN** the appended text SHALL NOT contain a second `# Topic:` line +- **AND** SHALL name the rendered CLI invocation for both `feedback dismiss` and `agent feedback` + +#### Scenario: The feedback recipe embeds the schema + +- **WHEN** an agent runs `taskless agent feedback` +- **THEN** stdout SHALL open with `# Topic: feedback` and contain the JSON Schema for the payload + +#### Scenario: An explicit refusal is honoured + +- **WHEN** the user replies to the invite asking that nothing be sent +- **THEN** the invite SHALL direct the agent to `feedback dismiss` +- **AND** SHALL direct it to name `DO_NOT_TRACK=1` or `TASKLESS_TELEMETRY_DISABLED=1` as the switch for the rest of telemetry + +#### Scenario: A skip still sends + +- **WHEN** the user replies `skip` to the invite +- **THEN** the invite SHALL direct the agent to `agent feedback` rather than `feedback dismiss` +- **AND** the feedback recipe SHALL direct the agent to omit `verbatim` and send the rest diff --git a/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/tasks.md b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/tasks.md new file mode 100644 index 00000000..87824736 --- /dev/null +++ b/openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/tasks.md @@ -0,0 +1,23 @@ +## 1. Survey contract + +- [x] 1.1 `src/survey/constants.ts`: `SURVEY_ID` → `01a0c7b9-dfe4-0000-d05e-ce253e90a68c`; `FeedbackKey` and `SURVEY_QUESTIONS` rebuilt for the seven questions in order; docblock records the date and why the ids changed. +- [x] 1.2 `src/schemas/feedback.ts`: `ruleKind` required; `verbatim`, `completed`, `workedWell`, `needsImprovement`, `agents`, `mostValuableRule` optional and non-blank when present. + +## 2. Recipes + +- [x] 2.1 `agent/feedback.md` → topic v2: `ruleKind` guidance including the onboarding value, `verbatim` optional, no follow-up question, `agents` and `mostValuableRule` guidance, the required/optional sentence under the schema. +- [x] 2.2 `agent/feedback-invite.md` → topic v2: new ask sentence; `skip`/silence/unrelated → `agent feedback` with `verbatim` omitted; explicit refusal → `feedback dismiss`. + +## 3. Tests + +- [x] 3.1 `feedback-schema.test.ts`: `ruleKind` alone is valid; missing `ruleKind` names the field; blank optional rejected; the smuggled `$survey_response_` key uses a new-survey id. +- [x] 3.2 `feedback-command.test.ts`: the `survey sent` event carries the new `$survey_id` and the new question ids; a payload without `verbatim` sends with no key for question 2. +- [x] 3.3 `feedback-recipes.test.ts`: the new ask sentence; the invite routes `skip` to `agent feedback` and refusal to `feedback dismiss`; the recipe embeds every key of the new schema. +- [x] 3.4 `survey-invite.test.ts` and `survey-cadence.test.ts`: any literal of the old id updated. + +## 4. Spec, changeset, archive + +- [x] 4.1 Spec delta for `cli-feedback-survey` (MODIFIED blocks restated in full; dry-run the archive and diff scenario counts). +- [x] 4.2 `.changeset/feedback-survey-0-11-3.md`, `patch`. +- [x] 4.3 `pnpm typecheck && pnpm lint && pnpm --filter @taskless/cli test`. +- [x] 4.4 Archive the change on this PR. diff --git a/openspec/specs/cli-feedback-survey/spec.md b/openspec/specs/cli-feedback-survey/spec.md index a82516dc..08431524 100644 --- a/openspec/specs/cli-feedback-survey/spec.md +++ b/openspec/specs/cli-feedback-survey/spec.md @@ -117,15 +117,17 @@ When the telemetry client is disabled, both verbs SHALL print a single line sayi The feedback payload SHALL be a JSON object with these keys: -| key | required | value | -| ------------------ | -------- | ------------------------------------------------- | -| `verbatim` | yes | the user's own words, non-empty | -| `goal` | yes | what the user was trying to accomplish, non-empty | -| `completed` | yes | exactly `Yes`, `No`, or `Unknown` | -| `workedWell` | no | what worked well | -| `needsImprovement` | no | what could use improvement | - -The CLI SHALL own the map from these keys to the survey's question identifiers; a payload SHALL NOT contain `$survey_*` keys. An omitted optional key SHALL be omitted from the event rather than sent as an empty string. The schema SHALL be embedded in the `feedback` recipe through the recipe input-schema mechanism. +| key | required | value | +| ------------------ | -------- | ----------------------------------------------------------------------------------------------------- | +| `ruleKind` | yes | the kind of rule the user was trying to create, non-empty; `none (onboarding)` on the onboarding path | +| `verbatim` | no | the user's own words, unedited | +| `completed` | no | exactly `Yes`, `No`, or `Unknown` | +| `workedWell` | no | what worked well | +| `needsImprovement` | no | what could use improvement | +| `agents` | no | the open-source, publicly available agent(s) or framework(s) in use | +| `mostValuableRule` | no | the rule creating the most value for the team, and why | + +The CLI SHALL own the map from these keys to the survey's question identifiers; a payload SHALL NOT contain `$survey_*` keys. An omitted optional key SHALL be omitted from the event rather than sent as an empty string, and an optional key that is present SHALL be non-blank. The schema SHALL be embedded in the `feedback` recipe through the recipe input-schema mechanism. #### Scenario: Optional answers are omitted, not blanked @@ -137,11 +139,22 @@ The CLI SHALL own the map from these keys to the survey's question identifiers; - **WHEN** a payload has `completed: "partially"` - **THEN** validation SHALL fail naming `completed` +#### Scenario: The rule kind alone is a complete response + +- **WHEN** a payload is `{ "ruleKind": "ast-grep, forbid eval in TypeScript" }` +- **THEN** validation SHALL pass +- **AND** the `survey sent` event SHALL carry `$survey_id` and exactly one `$survey_response_` key + +#### Scenario: A missing rule kind is rejected + +- **WHEN** a payload carries every optional key and no `ruleKind` +- **THEN** validation SHALL fail naming `ruleKind` + ### Requirement: The feedback and feedback-invite recipes -The CLI SHALL embed a `feedback` recipe that tells the agent it is the respondent: it records the user's reply verbatim, fills the remaining answers from its own observation of the session, writes the payload to `.taskless/.tmp-feedback.json`, runs `feedback send --from` that path, and deletes the file afterwards. The recipe SHALL embed the payload schema and SHALL NOT ask the agent to put the survey's questions to the user one by one. +The CLI SHALL embed a `feedback` recipe that tells the agent it is the respondent: it records the user's reply verbatim when there is one, fills the remaining answers from its own observation of the session, writes the payload to `.taskless/.tmp-feedback.json`, runs `feedback send --from` that path, and deletes the file afterwards. The recipe SHALL embed the payload schema, SHALL NOT ask the agent to put the survey's questions to the user one by one, and SHALL NOT permit a follow-up question. It SHALL tell the agent to answer `ruleKind` with `none (onboarding)` when the surveyed recipe was `onboard`, to name only open-source, publicly available software in `agents`, and to omit `mostValuableRule` rather than ask the user for it. -The CLI SHALL embed a `feedback-invite` recipe carrying the text appended to surveyed recipes. Rendered header-less, it SHALL instruct the agent to ask the user exactly once, with the sentence "Taskless would like to know how the CLI is doing. Would you be okay sharing a few sentences about your experience? Or just skip it with `skip`."; to treat a reply of `skip`, silence, or a reply unrelated to feedback as a dismissal and run `feedback dismiss`; and to treat any other reply as feedback and fetch `agent feedback`. Both recipes SHALL follow the recipe conventions: a `# Topic:` header, the CLI named by its rendered invocation, and commands that exist. +The CLI SHALL embed a `feedback-invite` recipe carrying the text appended to surveyed recipes. Rendered header-less, it SHALL instruct the agent to ask the user exactly once, with the sentence "Taskless would like to know how this went. Anything you'd like to add in your own words? Reply `skip` if not, and I'll send my own notes on the session."; to treat a reply of `skip`, silence, or a reply unrelated to feedback as an omitted `verbatim` and still fetch `agent feedback`; to treat any other reply as the user's words and fetch `agent feedback`; and, only when the user explicitly asks for nothing to be sent, to run `feedback dismiss` and name the telemetry opt-out environment variables in one line. Both recipes SHALL follow the recipe conventions: a `# Topic:` header, the CLI named by its rendered invocation, and commands that exist. #### Scenario: The appended invite carries no header @@ -153,3 +166,15 @@ The CLI SHALL embed a `feedback-invite` recipe carrying the text appended to sur - **WHEN** an agent runs `taskless agent feedback` - **THEN** stdout SHALL open with `# Topic: feedback` and contain the JSON Schema for the payload + +#### Scenario: An explicit refusal is honoured + +- **WHEN** the user replies to the invite asking that nothing be sent +- **THEN** the invite SHALL direct the agent to `feedback dismiss` +- **AND** SHALL direct it to name `DO_NOT_TRACK=1` or `TASKLESS_TELEMETRY_DISABLED=1` as the switch for the rest of telemetry + +#### Scenario: A skip still sends + +- **WHEN** the user replies `skip` to the invite +- **THEN** the invite SHALL direct the agent to `agent feedback` rather than `feedback dismiss` +- **AND** the feedback recipe SHALL direct the agent to omit `verbatim` and send the rest diff --git a/packages/cli/src/agent/feedback-invite.md b/packages/cli/src/agent/feedback-invite.md index 4b6a9a1f..dc899c7f 100644 --- a/packages/cli/src/agent/feedback-invite.md +++ b/packages/cli/src/agent/feedback-invite.md @@ -1,25 +1,31 @@ -# Topic: feedback-invite (CLI v%(CLI_VERSION)s / topic v1) +# Topic: feedback-invite (CLI v%(CLI_VERSION)s / topic v2) ## Before you finish: one question for the user Taskless would like to hear how this went. When the task above is done, and only then, put this to the user exactly once, in these words: -> Taskless would like to know how the CLI is doing. Would you be okay -> sharing a few sentences about your experience? Or just skip it with -> `skip`. +> Taskless would like to know how this went. Anything you'd like to add +> in your own words? Reply `skip` if not, and I'll send my own notes on +> the session. Then act on the reply: - **They gave feedback.** Fetch `%(TASKLESS_CLI)s agent feedback` and follow it. It takes their words as they are and asks you, not them, for the rest. - **They said `skip`, said nothing, or replied about something else.** - That is a decline. Run: + Fetch `%(TASKLESS_CLI)s agent feedback` all the same and follow it with + no `verbatim`. The rest of the survey is your account of the session, + and it is worth sending on its own. An unrelated reply is not + feedback, and it is not a reason to ask again. +- **They asked you not to send anything.** That is the one reply the + question did not offer, and it is honoured. Run: ``` %(TASKLESS_CLI)s feedback dismiss ``` - and carry on with whatever they asked for. An unrelated reply is not - feedback, and it is not a reason to ask again. + and tell them in one line that the CLI's other telemetry is switched + off with `DO_NOT_TRACK=1` or `TASKLESS_TELEMETRY_DISABLED=1`. Then + carry on with whatever they asked for. Ask once. Do not repeat the question later in this session, do not ask before the task is finished, and do not run `feedback send` without diff --git a/packages/cli/src/agent/feedback.md b/packages/cli/src/agent/feedback.md index 1a172140..21ccf64a 100644 --- a/packages/cli/src/agent/feedback.md +++ b/packages/cli/src/agent/feedback.md @@ -1,10 +1,10 @@ -# Topic: feedback (CLI v%(CLI_VERSION)s / topic v1) +# Topic: feedback (CLI v%(CLI_VERSION)s / topic v2) ## You are here This is `feedback`. It helps you turn what a user just said about Taskless into a survey response the CLI can send, and send it. You reach it from the invite at the end of an authoring or onboarding -recipe, after the user replied with something other than `skip`. +recipe, whether the user gave you their words or said `skip`. If that is not why you are reading this, re-run `%(TASKLESS_CLI)s agent` and find the topic you meant. @@ -15,8 +15,9 @@ the file. The whole thing is one short exchange with the user and a few sentences from you; it is not an interview. ## Preconditions -- The user replied to the invite with feedback rather than `skip`, - silence, or something unrelated. If they did any of those, this is the +- The invite was put to the user and they replied, or did not. Their + words, if any, are the only input you take from them. +- The user did not ask you to send nothing. If they did, this is the wrong recipe: run `%(TASKLESS_CLI)s feedback dismiss` and continue with what they asked for. - The agent can write a file and run a shell command. @@ -25,26 +26,29 @@ sentences from you; it is not an interview. ## You are the respondent The survey is addressed to you, the agent, not to the user. One answer is -the user's words and you record them verbatim. The other four are your -own account of the session you just ran: what they were trying to do, -whether they got it, what went well, and what did not. You already know -all of that. Do not put the survey's questions to the user one by one. +the user's words, if they gave any, and you record them verbatim. The +other six are your own account of the session you just ran: what kind +of rule it was for, whether they got it, what went well, what did not, +what you are running in, and which rule has earned its keep. You +already know all of that. +Do not put the survey's questions to the user one by one. ## Steps -1. **Take the user's reply as it is.** Whatever they wrote after the - invite is `verbatim`. Do not paraphrase, shorten, or tidy it. If they - wrote several messages, join them in order with a blank line between. - -2. **Ask one follow-up at most, and only if needed.** If the reply is - feedback but leaves you unable to fill `completed`, ask whether they - got what they came for. Otherwise ask nothing further; the invite - already asked for their time once. - -3. **Fill the rest from the session.** - - `goal`: what the user was trying to accomplish, in one or two of - your own sentences. Name the recipe you were following and the - rule or task it was for. +1. **Take the user's reply as it is, if there is one.** Whatever they + wrote after the invite is `verbatim`. Do not paraphrase, shorten, or + tidy it. If they wrote several messages, join them in order with a + blank line between. If they said `skip`, said nothing, or replied + about something else, omit `verbatim` and go on: the rest of the + survey is yours to answer. + +2. **Fill the rest from the session.** Ask the user nothing further; + the invite already asked for their time once. + - `ruleKind`: the engine and what the rule was for, in a phrase + (`ast-grep, forbid eval in TypeScript`; `vale, no hedging in + docs`; `runtime, env var must be set`). If the recipe was + `onboard`, write `none (onboarding)`. This is the one required + answer. - `completed`: `Yes`, `No`, or `Unknown`. Success is binary here. A rule that verifies and the user accepted is `Yes`; a rule the user abandoned or that never verified is `No`; if the session ended @@ -54,28 +58,36 @@ all of that. Do not put the survey's questions to the user one by one. - `needsImprovement`: the steps that cost time, needed a retry, or that you had to work around. Be specific: name the command, the field, or the message. Omit the key if nothing stands out. + - `agents`: the agent you are, and any agent framework you can see + the project using, by product name. Only open-source, publicly + available software belongs here; omit the key rather than name an + internal or proprietary tool. + - `mostValuableRule`: of the rules under `.taskless/rules/`, the one + doing the most for this team and why, if the session gave you a + view of that. Most sessions will not have; omit the key then, and + do not ask the user for it. Keep your own answers to a few sentences each. The people reading them want the shape of the friction, not a transcript. -4. **Write the payload** to `.taskless/.tmp-feedback.json`, matching the +3. **Write the payload** to `.taskless/.tmp-feedback.json`, matching the input schema below. Use the human keys exactly as given; the CLI maps them to the survey's own question identifiers, and a payload carrying a `$survey_` key is not what it expects. -5. **Send.** Run: +4. **Send.** Run: ``` %(TASKLESS_CLI)s feedback send --from .taskless/.tmp-feedback.json --json ``` Under `--json`, a failure is `{ ok: false, code, message }`; see the table below. On success the command prints a thank-you. -6. **Clean up.** Delete `.taskless/.tmp-feedback.json` whether the call +5. **Clean up.** Delete `.taskless/.tmp-feedback.json` whether the call succeeded or failed. `.taskless/.gitignore` already ignores it, so a forgotten file is a stray rather than a commit, but leave nothing behind. -7. **Return to the user's task.** Thank them in one line and carry on. +6. **Return to the user's task.** Thank them in one line and carry on. Do not ask for more, and do not run this recipe a second time in the same session. @@ -87,17 +99,17 @@ The `--from` JSON file conforms to: %(INPUT_SCHEMA)s ``` -`verbatim`, `goal`, and `completed` are required. `workedWell` and -`needsImprovement` are optional, and an optional answer you have nothing -for is an omitted key rather than an empty string. +`ruleKind` is required. Every other key is optional, and an optional +answer you have nothing for is an omitted key rather than an empty +string. ## Important Notes - Do NOT edit the user's words. `verbatim` is the one answer that is theirs, and its value to the people reading it is that it is theirs. -- Do NOT invent a follow-up interview. The invite asked once; step 2 is - the only question this recipe allows, and only when `completed` would - otherwise be a guess. +- Do NOT invent a follow-up interview. The invite asked once, and this + recipe asks nothing. An answer you cannot give is an omitted key, or + `Unknown` for `completed`. - If telemetry is disabled in this environment the command says so and exits 0 with nothing sent. That is the expected outcome there, not an error to retry. @@ -112,5 +124,5 @@ With `--json`, failures emit `{ ok: false, code, message }`: ## See Also -- `%(TASKLESS_CLI)s feedback dismiss`: what to run when the user declined +- `%(TASKLESS_CLI)s feedback dismiss`: what to run when the user asked for nothing to be sent - `%(TASKLESS_CLI)s agent`: the topic index, if you arrived here by mistake diff --git a/packages/cli/src/schemas/feedback.ts b/packages/cli/src/schemas/feedback.ts index 7e44ec12..1f613a75 100644 --- a/packages/cli/src/schemas/feedback.ts +++ b/packages/cli/src/schemas/feedback.ts @@ -9,20 +9,26 @@ import { COMPLETED_CHOICES } from "../survey/constants"; * `src/survey/constants.ts`, and a payload never carries a `$survey_*` key. */ export const inputSchema = z.object({ - verbatim: z + ruleKind: z .string() .trim() - .min(1, "verbatim must be the user's own words, non-empty") - .describe("The user's reply, in their own words, unedited"), - goal: z + .min(1, "ruleKind must be a non-empty string") + .describe( + "The kind of rule the user was trying to create: the engine and what the rule was for. `none (onboarding)` on the onboarding path" + ), + verbatim: z .string() .trim() - .min(1, "goal must be a non-empty string") - .describe("What the user was trying to accomplish, in your words"), + .min(1, "verbatim, when present, must be the user's own words, non-empty") + .optional() + .describe( + "The user's reply, in their own words, unedited. Omit when they gave none" + ), completed: z .enum(COMPLETED_CHOICES, { error: `completed must be one of ${COMPLETED_CHOICES.map((choice) => `"${choice}"`).join(", ")}`, }) + .optional() .describe( "Whether the user completed the task, in your opinion. Success is binary; use Unknown when you cannot tell" ), @@ -40,6 +46,22 @@ export const inputSchema = z.object({ .describe( "Steps of the interaction with Taskless that could use improvement" ), + agents: z + .string() + .trim() + .min(1, "agents, when present, must be non-empty") + .optional() + .describe( + "The open-source, publicly available agent(s) or framework(s) the user is working through, including the one running this recipe" + ), + mostValuableRule: z + .string() + .trim() + .min(1, "mostValuableRule, when present, must be non-empty") + .optional() + .describe( + "Of the rules created so far, the one creating the most value for the team and why. Omit when there are no rules or you cannot tell" + ), }); export type FeedbackInput = z.infer; diff --git a/packages/cli/src/survey/constants.ts b/packages/cli/src/survey/constants.ts index 5a93451e..5faa7560 100644 --- a/packages/cli/src/survey/constants.ts +++ b/packages/cli/src/survey/constants.ts @@ -17,18 +17,22 @@ * recipe; it cannot know the agent put the question to a person. The funnel * reads shown ≫ sent by design, and nothing here pretends otherwise. * - * A question's id changes when its type changes in PostHog. Q3 became single - * choice on 2026-09-17 and took a new id; the one below is current. + * A question's id is PostHog's and changes whenever the question does. The + * survey below replaced `01a0b1a0-80fb-0000-5dc1-baa4ec44e619` for 0.11.3: + * seven questions, only the first required, every id new. The cadence store + * is keyed by survey id, so every install is invited once more. */ -export const SURVEY_ID = "01a0b1a0-80fb-0000-5dc1-baa4ec44e619"; +export const SURVEY_ID = "01a0c7b9-dfe4-0000-d05e-ce253e90a68c"; /** The payload keys, in question order. */ export type FeedbackKey = + | "ruleKind" | "verbatim" - | "goal" | "completed" | "workedWell" - | "needsImprovement"; + | "needsImprovement" + | "agents" + | "mostValuableRule"; export interface SurveyQuestion { key: FeedbackKey; @@ -38,30 +42,43 @@ export interface SurveyQuestion { export const SURVEY_QUESTIONS: readonly SurveyQuestion[] = [ { - key: "verbatim", - id: "5feff6a3-6768-4817-92d7-5ae3975c6baa", - question: "What was the user's comments verbatim?", + key: "ruleKind", + id: "0874591f-c554-4ac3-8930-e11c436d859e", + question: "What kind of rule was the user trying to create?", }, { - key: "goal", - id: "561e87f4-a1b7-4855-b728-29d19421f7e7", - question: "What was the user trying to accomplish?", + key: "verbatim", + id: "2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2", + question: "Did the user offer any comments? (leave blank if no comments)", }, { key: "completed", - id: "6ebdfabb-3575-49aa-857c-47b6bbfdebc8", + id: "605e12a8-82b6-480f-93b2-ab8de0fa08bd", question: "Did the user successfully complete the task in your opinion?", }, { key: "workedWell", - id: "2316428e-dc3e-4c96-ae67-a6e8c66d7db5", - question: "What steps of the interaction with Taskless worked well?", + id: "b5375d87-e295-4833-84ed-fca8140ba992", + question: + "What steps of the interaction with the Taskless skills & CLI worked well?", }, { key: "needsImprovement", - id: "67bedbd9-ca70-4c1c-b1a6-6df830a453dd", + id: "a8cf706d-3ff7-4845-bea9-501013be958c", + question: + "What steps of the interaction with Taskless skills & CLI could use improvement?", + }, + { + key: "agents", + id: "f85b22df-8e51-4c9c-8219-261b33b71c90", + question: + "What agent(s) or framework(s) is the user using that are open sourced and publicly available?", + }, + { + key: "mostValuableRule", + id: "4f8e938e-22f6-449c-9b8c-43c51d08e214", question: - "What steps of the interaction with Taskless could use improvement?", + "Of the rules created so far, what rule is creating the most value for the team and why? (leave blank if there are no rules)", }, ]; diff --git a/packages/cli/test/feedback-command.test.ts b/packages/cli/test/feedback-command.test.ts index f27bce96..e072f700 100644 --- a/packages/cli/test/feedback-command.test.ts +++ b/packages/cli/test/feedback-command.test.ts @@ -39,8 +39,8 @@ function verb(name: "dismiss" | "send"): RunnableCommand { } const VALID = { + ruleKind: "ast-grep, forbid eval in TypeScript", verbatim: "The second rule took three tries but the verify loop caught it.", - goal: "Forbid eval in TypeScript", completed: "Yes", needsImprovement: "The first draft used a language name ast-grep rejects.", }; @@ -115,17 +115,32 @@ describe("feedback command", () => { expect(capture).toHaveBeenCalledTimes(1); expect(capture).toHaveBeenCalledWith("survey sent", { $survey_id: SURVEY_ID, - "$survey_response_5feff6a3-6768-4817-92d7-5ae3975c6baa": VALID.verbatim, - "$survey_response_561e87f4-a1b7-4855-b728-29d19421f7e7": VALID.goal, - "$survey_response_6ebdfabb-3575-49aa-857c-47b6bbfdebc8": "Yes", - "$survey_response_67bedbd9-ca70-4c1c-b1a6-6df830a453dd": + "$survey_response_0874591f-c554-4ac3-8930-e11c436d859e": VALID.ruleKind, + "$survey_response_2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2": VALID.verbatim, + "$survey_response_605e12a8-82b6-480f-93b2-ab8de0fa08bd": "Yes", + "$survey_response_a8cf706d-3ff7-4845-bea9-501013be958c": VALID.needsImprovement, }); - // The unanswered optional (`workedWell`) has no key at all. + // The unanswered optionals have no key at all. const properties = capture.mock.calls[0]![1] as Record; expect(Object.keys(properties)).toHaveLength(5); }); + it("sends the agent's account alone when the user gave no words", async () => { + // A `skip` reply is not a dismissal: the payload omits `verbatim` and + // the rest still goes, keyed to the survey's required question. + const from = await writePayload({ ruleKind: "none (onboarding)" }); + await verb("send").run({ + args: { dir: cwd, from, json: false }, + rawArgs: [], + }); + expect(capture).toHaveBeenCalledWith("survey sent", { + $survey_id: SURVEY_ID, + "$survey_response_0874591f-c554-4ac3-8930-e11c436d859e": + "none (onboarding)", + }); + }); + it("leaves the input file in place and creates no .taskless/", async () => { const from = await writePayload(VALID); await verb("send").run({ @@ -186,7 +201,7 @@ describe("feedback command", () => { it("validates before honoring the opt-out, then sends nothing", async () => { enabled = false; - const bad = await writePayload({ goal: "x" }); + const bad = await writePayload({ verbatim: "x" }); await expect( verb("send").run({ args: { dir: cwd, from: bad, json: false }, @@ -209,15 +224,17 @@ describe("feedback command", () => { describe("buildSurveyResponse", () => { it("maps every answered key and no unanswered one", () => { const properties = buildSurveyResponse({ + ruleKind: "r", verbatim: "v", - goal: "g", completed: "Unknown", + agents: "Claude Code", }); expect(properties).toEqual({ $survey_id: SURVEY_ID, - "$survey_response_5feff6a3-6768-4817-92d7-5ae3975c6baa": "v", - "$survey_response_561e87f4-a1b7-4855-b728-29d19421f7e7": "g", - "$survey_response_6ebdfabb-3575-49aa-857c-47b6bbfdebc8": "Unknown", + "$survey_response_0874591f-c554-4ac3-8930-e11c436d859e": "r", + "$survey_response_2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2": "v", + "$survey_response_605e12a8-82b6-480f-93b2-ab8de0fa08bd": "Unknown", + "$survey_response_f85b22df-8e51-4c9c-8219-261b33b71c90": "Claude Code", }); }); }); diff --git a/packages/cli/test/feedback-recipes.test.ts b/packages/cli/test/feedback-recipes.test.ts index 1c5dd606..c82df196 100644 --- a/packages/cli/test/feedback-recipes.test.ts +++ b/packages/cli/test/feedback-recipes.test.ts @@ -14,7 +14,7 @@ const invocation = "npx @taskless/cli"; /** The sentence the invite puts to the user, as the design fixed it. */ const ASK = - "Taskless would like to know how the CLI is doing. Would you be okay sharing a few sentences about your experience? Or just skip it with `skip`."; + "Taskless would like to know how this went. Anything you'd like to add in your own words? Reply `skip` if not, and I'll send my own notes on the session."; /** Blockquote prose, unwrapped: the recipe hard-wraps and prefixes `> `. */ function unwrapQuote(text: string): string { @@ -92,11 +92,26 @@ describe("the feedback invite", () => { expect(fragment).toContain(`${invocation} feedback dismiss`); }); - it("treats skip, silence, and an unrelated reply as a decline", () => { - const fragment = getRecipe("feedback-invite", { header: false }) ?? ""; - expect(fragment).toMatch( - /`skip`, said nothing, or replied about something else/ + it("sends the agent's account on skip, silence, or an unrelated reply", () => { + const fragment = + getRecipe("feedback-invite", { invocation, header: false }) ?? ""; + const skipDoor = fragment.slice( + fragment.indexOf("`skip`, said nothing, or replied about something else"), + fragment.indexOf("They asked you not to send anything") + ); + expect(skipDoor).toContain(`${invocation} agent feedback`); + expect(skipDoor).toContain("no `verbatim`"); + expect(skipDoor).not.toContain("feedback dismiss"); + }); + + it("reserves dismiss for an explicit refusal and names the telemetry switch", () => { + const fragment = + getRecipe("feedback-invite", { invocation, header: false }) ?? ""; + const refusal = fragment.slice( + fragment.indexOf("They asked you not to send anything") ); - expect(fragment).toContain("That is a decline"); + expect(refusal).toContain(`${invocation} feedback dismiss`); + expect(refusal).toContain("DO_NOT_TRACK=1"); + expect(refusal).toContain("TASKLESS_TELEMETRY_DISABLED=1"); }); }); diff --git a/packages/cli/test/feedback-schema.test.ts b/packages/cli/test/feedback-schema.test.ts index ca549138..1f220ed8 100644 --- a/packages/cli/test/feedback-schema.test.ts +++ b/packages/cli/test/feedback-schema.test.ts @@ -3,16 +3,18 @@ import { describe, expect, it } from "vitest"; import { inputSchema } from "../src/schemas/feedback"; const valid = { + ruleKind: "ast-grep, forbid eval in TypeScript", verbatim: "It worked but the second rule took three tries.", - goal: "Add an ast-grep rule that forbids eval", completed: "Yes", }; describe("feedback payload schema", () => { - it("accepts the three required answers alone", () => { - const parsed = inputSchema.parse(valid); - expect(parsed.workedWell).toBeUndefined(); - expect(parsed.needsImprovement).toBeUndefined(); + it("accepts the one required answer alone", () => { + // `skip` at the invite leaves the agent's account and nothing else. + const parsed = inputSchema.parse({ ruleKind: "none (onboarding)" }); + expect(parsed.verbatim).toBeUndefined(); + expect(parsed.completed).toBeUndefined(); + expect(Object.keys(parsed)).toEqual(["ruleKind"]); }); it("accepts the optional answers when present", () => { @@ -20,8 +22,11 @@ describe("feedback payload schema", () => { ...valid, workedWell: "The verify loop.", needsImprovement: "The first draft's language field.", + agents: "Claude Code", + mostValuableRule: "no-eval: it caught two uses in the first check.", }); expect(parsed.workedWell).toBe("The verify loop."); + expect(parsed.agents).toBe("Claude Code"); }); it.each(["partially", "yes", "true", ""])( @@ -36,13 +41,13 @@ describe("feedback payload schema", () => { } ); - it("rejects a missing verbatim, naming the field", () => { - const { verbatim: _verbatim, ...rest } = valid; + it("rejects a missing ruleKind, naming the field", () => { + const { ruleKind: _ruleKind, ...rest } = valid; const result = inputSchema.safeParse(rest); expect(result.success).toBe(false); if (result.success) return; expect(result.error.issues.map((issue) => issue.path[0])).toContain( - "verbatim" + "ruleKind" ); }); @@ -59,10 +64,10 @@ describe("feedback payload schema", () => { // reaches the event; feedback-command.test.ts asserts the event shape. const parsed = inputSchema.parse({ ...valid, - "$survey_response_5feff6a3-6768-4817-92d7-5ae3975c6baa": "smuggled", + "$survey_response_2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2": "smuggled", }); expect(Object.keys(parsed)).not.toContain( - "$survey_response_5feff6a3-6768-4817-92d7-5ae3975c6baa" + "$survey_response_2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2" ); }); }); diff --git a/packages/cli/test/survey-cadence.test.ts b/packages/cli/test/survey-cadence.test.ts index ff810ccd..9bc474ac 100644 --- a/packages/cli/test/survey-cadence.test.ts +++ b/packages/cli/test/survey-cadence.test.ts @@ -15,17 +15,19 @@ import { } from "../src/survey/constants"; describe("survey constants", () => { - // The identifiers are PostHog's, transcribed once. Q3's id changed when the - // question became single choice, which is exactly the kind of drift this - // pins: the value here is what the live survey holds as of 2026-09-17. + // The identifiers are PostHog's, transcribed once. A question's id changes + // whenever the question does, which is exactly the kind of drift this pins: + // the values here are what the 0.11.3 survey holds as of 2026-09-21. it("carries the live survey's question ids in question order", () => { - expect(SURVEY_ID).toBe("01a0b1a0-80fb-0000-5dc1-baa4ec44e619"); + expect(SURVEY_ID).toBe("01a0c7b9-dfe4-0000-d05e-ce253e90a68c"); expect(SURVEY_QUESTIONS.map(({ key, id }) => [key, id])).toEqual([ - ["verbatim", "5feff6a3-6768-4817-92d7-5ae3975c6baa"], - ["goal", "561e87f4-a1b7-4855-b728-29d19421f7e7"], - ["completed", "6ebdfabb-3575-49aa-857c-47b6bbfdebc8"], - ["workedWell", "2316428e-dc3e-4c96-ae67-a6e8c66d7db5"], - ["needsImprovement", "67bedbd9-ca70-4c1c-b1a6-6df830a453dd"], + ["ruleKind", "0874591f-c554-4ac3-8930-e11c436d859e"], + ["verbatim", "2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2"], + ["completed", "605e12a8-82b6-480f-93b2-ab8de0fa08bd"], + ["workedWell", "b5375d87-e295-4833-84ed-fca8140ba992"], + ["needsImprovement", "a8cf706d-3ff7-4845-bea9-501013be958c"], + ["agents", "f85b22df-8e51-4c9c-8219-261b33b71c90"], + ["mostValuableRule", "4f8e938e-22f6-449c-9b8c-43c51d08e214"], ]); });