Skip to content

feat(feedback): move to the 0.11.3 survey and send on skip #382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
theCodeDrift merged 1 commit into from
Sep 22, 2026
Merged

feat(feedback): move to the 0.11.3 survey and send on skip #382

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/feedback-survey-0-11-3.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@taskless/cli": patch
---

The feedback survey is replaced for 0.11.3. Only the kind of rule the user was trying to create is required now; the user's own words, the completion verdict, what worked, what did not, the agent in use, and the most valuable rule so far are all optional. A `skip` at the invite no longer dismisses the survey: the agent sends its own account of the session and leaves the user's words out. `feedback dismiss` is reserved for a user who asks that nothing be sent. Every install is invited once more, since the new survey keeps its own cadence.
71 changes: 71 additions & 0 deletions openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
## Context

The first survey (`01a0b1a0-…`) required `verbatim`, `goal`, and `completed`. Measured against PostHog's definition on 2026-09-21: Q1 and Q3 carried no `optional` field (PostHog's default is required), Q2 was `optional: false`, Q4 and Q5 `optional: true`. The CLI's Zod schema matched that. The consequence in practice: a user who replied `skip` produced a `survey dismissed` and nothing else, even though the agent had four answers ready that needed no one's permission.

The replacement (`01a0c7b9-…`, "Product-market fit (PMF) (0.11.3)", started 2026-09-22) has seven questions. Only Q1 is required.

| # | id | question | optional |
| --- | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | -------- |
| 1 | `0874591f-c554-4ac3-8930-e11c436d859e` | What kind of rule was the user trying to create? | no |
| 2 | `2c3c80dc-dcda-4e29-b52e-a25ef58b5ca2` | Did the user offer any comments? (leave blank if no comments) | yes |
| 3 | `605e12a8-82b6-480f-93b2-ab8de0fa08bd` | Did the user successfully complete the task in your opinion? (`Yes`/`No`/`Unknown`) | yes |
| 4 | `b5375d87-e295-4833-84ed-fca8140ba992` | What steps of the interaction with the Taskless skills & CLI worked well? | yes |
| 5 | `a8cf706d-3ff7-4845-bea9-501013be958c` | What steps of the interaction with Taskless skills & CLI could use improvement? | yes |
| 6 | `f85b22df-8e51-4c9c-8219-261b33b71c90` | What agent(s) or framework(s) is the user using that are open sourced and publicly available? | yes |
| 7 | `4f8e938e-22f6-449c-9b8c-43c51d08e214` | Of the rules created so far, what rule is creating the most value for the team and why? (leave blank if there are no rules) | yes |

Required-ness is enforced by PostHog's popover UI only; the capture API accepts a `survey sent` with any subset of `$survey_response_*` keys. The CLI's schema is therefore the only gate, and it mirrors PostHog's definition so that the responses view and the CLI agree on what a complete answer is.

## Decisions

### Human keys

| key | question | required |
| ------------------ | -------- | -------- |
| `ruleKind` | 1 | yes |
| `verbatim` | 2 | no |
| `completed` | 3 | no |
| `workedWell` | 4 | no |
| `needsImprovement` | 5 | no |
| `agents` | 6 | no |
| `mostValuableRule` | 7 | no |

`ruleKind` rather than `goal`: the question narrowed from "what were they trying to accomplish" to "what kind of rule", and the old key would invite the old answer. `agents` is plural because the question is. `mostValuableRule` is the question's noun phrase.

The map stays the only place ids live, and `buildSurveyResponse` stays a loop over it, so the command does not change.

### `ruleKind` on the onboarding path

`onboard` is surveyed and creates no rule. The recipe tells the agent to answer `none (onboarding)` there, and otherwise to name the engine and what the rule was for in a phrase (`ast-grep, forbid eval in TypeScript`). The value is free text on PostHog's side; the recipe supplies the shape, the schema only requires non-empty.

### `skip` sends; an explicit refusal dismisses

Six of the seven answers are the agent's own account of the session. That is the same category of data as `cli_rule_create` or `cli_check`: telemetry the user opted into by leaving it enabled, and the gate already withholds the invite under `DO_NOT_TRACK`. The only answer that is the user's is the quote, so the invite asks for exactly that and offers exactly two replies:

> Taskless would like to know how this went. Anything you'd like to add in your own words? Reply `skip` if not, and I'll send my own notes on the session.

- words → `verbatim` is those words, unedited, and the agent sends;
- `skip`, silence, or an unrelated reply → `verbatim` is omitted and the agent sends the rest.

There is no third option in the invite, so no reply the invite solicits produces nothing. `feedback dismiss` is kept for the reply it does not solicit: a user who says not to send anything ("no", "don't send that") is asking for the response not to exist, and the CLI honours that and records it as a dismissal. The recipe tells the agent, in that case, to say in one line that the rest of the CLI's telemetry is governed by `DO_NOT_TRACK=1` / `TASKLESS_TELEMETRY_DISABLED=1`, so the user learns where the real switch is rather than being offered a per-survey one that covers one capture in twenty.

### No follow-up question

The old recipe permitted one follow-up, to fill `completed` when the reply left it unclear. `completed` is optional now, and `Unknown` exists for exactly that state, so the follow-up goes. The invite is the only question the flow puts to the user.

### Filling `agents` and `mostValuableRule`

`agents`: the agent names the harness it is running in and any framework it can see the project using, restricted to open-source, publicly available software as the question asks. It omits the key rather than naming an internal or proprietary tool. No PII is involved; this is the name of a product.

`mostValuableRule`: the agent answers from the session and from `.taskless/rules/` when it has a view; it omits the key when it does not, and does not ask the user. The question is aimed at a team that has lived with its rules for a while, and most surveyed sessions are the one that just wrote the first one.

### Cadence

Unchanged. The store is keyed by survey id, so the new id gets a fresh `next_ask` and every install is invited once more on its next surveyed recipe. That is the intended effect of shipping a new survey and the standing spec already states it.

## Alternatives considered

- **Offer `no` in the invite alongside `skip`.** Rejected: it invites a reply that produces nothing from a user who has telemetry enabled, and the agent's six answers are not the user's to withhold any more than the other events are. The refusal path exists for the user who volunteers it, not as a menu item.
- **Remove `feedback dismiss`.** Rejected: an explicit "don't send anything" needs a verb that honours it and records it, and the verb already exists.
- **Keep `goal` as the key and change only its description.** Rejected: the key would be read by an agent that learned the old recipe, and the old answer is not the new question.
- **Send on silence without asking at all.** Rejected: the user's words are the answer the survey exists to collect, and an ask that costs one line is worth it.
33 changes: 33 additions & 0 deletions openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/proposal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## Why

The 0.11.3 release moves the CLI to a new PostHog survey, `01a0c7b9-dfe4-0000-d05e-ce253e90a68c`, which replaces `01a0b1a0-80fb-0000-5dc1-baa4ec44e619`. The new survey was written from what the first one taught us: the questions an agent answers well on its own were required-but-often-guessed, the one question only the user can answer was required and so blocked the whole response when the user said `skip`, and two things we want to know were never asked (which agent harness is in play, and which rule has earned its keep). PostHog assigns a fresh question id per question, so the CLI's key-to-id map must be rebuilt, not edited.

The survey is now agent-first. Only one question is required, the kind of rule the user was trying to create, and the agent knows that without asking anyone. Everything else is optional, including the user's own words. A user who declines to add anything is therefore no longer a dismissal: the agent still sends its own account of the session.

## What Changes

- **`SURVEY_ID` moves to the new survey** and the question map is rebuilt for its seven questions. The cadence store is keyed by survey id, so every install sees the new survey as a fresh ask; that is the behaviour the standing spec already promises.
- **The payload schema changes shape.** `goal` is gone. `ruleKind` (required) replaces it as the one required answer. `verbatim` becomes optional. `completed`, `workedWell`, and `needsImprovement` stay, with `completed` now optional. Two optional keys are new: `agents`, the open-source agent or framework the user is working through, and `mostValuableRule`, the rule that is earning the most for the team and why.
- **A `skip` is no longer a dismissal.** The invite asks for one thing, the user's own words, and offers two replies: words or `skip`. Words go in `verbatim`; `skip`, silence, or an unrelated reply means `verbatim` is omitted, and either way the agent fetches `agent feedback` and sends its own account of the session. The other six answers are the agent's observations, which are telemetry the user has already opted into by leaving it enabled. `feedback dismiss` stays as the explicit opt-out: the invite does not offer it, but a user who says not to send anything gets exactly that, and the recipe tells the agent to point them at the telemetry switch for everything else.
- **The `feedback` and `feedback-invite` recipes are revised** to match: the invite's wording changes, the feedback recipe drops its one permitted follow-up question (it existed only to fill `completed`, which is now optional), and it explains how to fill the two new answers and what `ruleKind` means on the onboarding path.

Nothing here is **BREAKING**. The payload is read by `feedback send` from a file the same agent wrote seconds earlier from the same CLI's recipe; no consumer holds an old-shape payload across a release. `patch`.

## Capabilities

### Modified Capabilities

- `cli-feedback-survey`: the payload contract (keys, which is required), the invite's reply handling (`skip` sends rather than dismisses), and the recipes' instructions.

## Impact

- `packages/cli/src/survey/constants.ts`: new `SURVEY_ID`, new `FeedbackKey` union, new `SURVEY_QUESTIONS` map.
- `packages/cli/src/schemas/feedback.ts`: the Zod schema, which is what the recipe embeds and `feedback send` validates with.
- `packages/cli/src/agent/feedback.md` (topic v2) and `feedback-invite.md` (topic v2).
- `packages/cli/src/commands/feedback.ts`: no logic change; `buildSurveyResponse` iterates the map.
- Tests: `feedback-schema.test.ts`, `feedback-recipes.test.ts`, `feedback-command.test.ts`, `survey-invite.test.ts` where they name keys, ids, or the invite sentence.
- PostHog: `survey shown` / `survey dismissed` / `survey sent` continue under their names with the new `$survey_id`. The old survey keeps its responses; nothing is migrated.

## Delivery shape

**Single PR.** The id, the map, the schema, and the two recipes are only correct together, and the whole diff is a few hundred lines. The change archives on the same PR.
67 changes: 67 additions & 0 deletions ...ges/archive/2026-09-21-feedback-survey-0-11-3/specs/cli-feedback-survey/spec.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
## MODIFIED Requirements

### Requirement: The feedback payload uses human keys mapped to survey questions by the CLI

The feedback payload SHALL be a JSON object with these keys:

| key | required | value |
| ------------------ | -------- | ----------------------------------------------------------------------------------------------------- |
| `ruleKind` | yes | the kind of rule the user was trying to create, non-empty; `none (onboarding)` on the onboarding path |
| `verbatim` | no | the user's own words, unedited |
| `completed` | no | exactly `Yes`, `No`, or `Unknown` |
| `workedWell` | no | what worked well |
| `needsImprovement` | no | what could use improvement |
| `agents` | no | the open-source, publicly available agent(s) or framework(s) in use |
| `mostValuableRule` | no | the rule creating the most value for the team, and why |

The CLI SHALL own the map from these keys to the survey's question identifiers; a payload SHALL NOT contain `$survey_*` keys. An omitted optional key SHALL be omitted from the event rather than sent as an empty string, and an optional key that is present SHALL be non-blank. The schema SHALL be embedded in the `feedback` recipe through the recipe input-schema mechanism.

#### Scenario: Optional answers are omitted, not blanked

- **WHEN** a payload has no `workedWell`
- **THEN** the `survey sent` event SHALL carry no `$survey_response_` key for that question

#### Scenario: Completion is one of three literals

- **WHEN** a payload has `completed: "partially"`
- **THEN** validation SHALL fail naming `completed`

#### Scenario: The rule kind alone is a complete response

- **WHEN** a payload is `{ "ruleKind": "ast-grep, forbid eval in TypeScript" }`
- **THEN** validation SHALL pass
- **AND** the `survey sent` event SHALL carry `$survey_id` and exactly one `$survey_response_<question id>` key

#### Scenario: A missing rule kind is rejected

- **WHEN** a payload carries every optional key and no `ruleKind`
- **THEN** validation SHALL fail naming `ruleKind`

### Requirement: The feedback and feedback-invite recipes

The CLI SHALL embed a `feedback` recipe that tells the agent it is the respondent: it records the user's reply verbatim when there is one, fills the remaining answers from its own observation of the session, writes the payload to `.taskless/.tmp-feedback.json`, runs `feedback send --from` that path, and deletes the file afterwards. The recipe SHALL embed the payload schema, SHALL NOT ask the agent to put the survey's questions to the user one by one, and SHALL NOT permit a follow-up question. It SHALL tell the agent to answer `ruleKind` with `none (onboarding)` when the surveyed recipe was `onboard`, to name only open-source, publicly available software in `agents`, and to omit `mostValuableRule` rather than ask the user for it.

The CLI SHALL embed a `feedback-invite` recipe carrying the text appended to surveyed recipes. Rendered header-less, it SHALL instruct the agent to ask the user exactly once, with the sentence "Taskless would like to know how this went. Anything you'd like to add in your own words? Reply `skip` if not, and I'll send my own notes on the session."; to treat a reply of `skip`, silence, or a reply unrelated to feedback as an omitted `verbatim` and still fetch `agent feedback`; to treat any other reply as the user's words and fetch `agent feedback`; and, only when the user explicitly asks for nothing to be sent, to run `feedback dismiss` and name the telemetry opt-out environment variables in one line. Both recipes SHALL follow the recipe conventions: a `# Topic:` header, the CLI named by its rendered invocation, and commands that exist.

#### Scenario: The appended invite carries no header

- **WHEN** the gate is open and a surveyed recipe is served
- **THEN** the appended text SHALL NOT contain a second `# Topic:` line
- **AND** SHALL name the rendered CLI invocation for both `feedback dismiss` and `agent feedback`

#### Scenario: The feedback recipe embeds the schema

- **WHEN** an agent runs `taskless agent feedback`
- **THEN** stdout SHALL open with `# Topic: feedback` and contain the JSON Schema for the payload

#### Scenario: An explicit refusal is honoured

- **WHEN** the user replies to the invite asking that nothing be sent
- **THEN** the invite SHALL direct the agent to `feedback dismiss`
- **AND** SHALL direct it to name `DO_NOT_TRACK=1` or `TASKLESS_TELEMETRY_DISABLED=1` as the switch for the rest of telemetry

#### Scenario: A skip still sends

- **WHEN** the user replies `skip` to the invite
- **THEN** the invite SHALL direct the agent to `agent feedback` rather than `feedback dismiss`
- **AND** the feedback recipe SHALL direct the agent to omit `verbatim` and send the rest
23 changes: 23 additions & 0 deletions openspec/changes/archive/2026-09-21-feedback-survey-0-11-3/tasks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## 1. Survey contract

- [x] 1.1 `src/survey/constants.ts`: `SURVEY_ID` → `01a0c7b9-dfe4-0000-d05e-ce253e90a68c`; `FeedbackKey` and `SURVEY_QUESTIONS` rebuilt for the seven questions in order; docblock records the date and why the ids changed.
- [x] 1.2 `src/schemas/feedback.ts`: `ruleKind` required; `verbatim`, `completed`, `workedWell`, `needsImprovement`, `agents`, `mostValuableRule` optional and non-blank when present.

## 2. Recipes

- [x] 2.1 `agent/feedback.md` → topic v2: `ruleKind` guidance including the onboarding value, `verbatim` optional, no follow-up question, `agents` and `mostValuableRule` guidance, the required/optional sentence under the schema.
- [x] 2.2 `agent/feedback-invite.md` → topic v2: new ask sentence; `skip`/silence/unrelated → `agent feedback` with `verbatim` omitted; explicit refusal → `feedback dismiss`.

## 3. Tests

- [x] 3.1 `feedback-schema.test.ts`: `ruleKind` alone is valid; missing `ruleKind` names the field; blank optional rejected; the smuggled `$survey_response_` key uses a new-survey id.
- [x] 3.2 `feedback-command.test.ts`: the `survey sent` event carries the new `$survey_id` and the new question ids; a payload without `verbatim` sends with no key for question 2.
- [x] 3.3 `feedback-recipes.test.ts`: the new ask sentence; the invite routes `skip` to `agent feedback` and refusal to `feedback dismiss`; the recipe embeds every key of the new schema.
- [x] 3.4 `survey-invite.test.ts` and `survey-cadence.test.ts`: any literal of the old id updated.

## 4. Spec, changeset, archive

- [x] 4.1 Spec delta for `cli-feedback-survey` (MODIFIED blocks restated in full; dry-run the archive and diff scenario counts).
- [x] 4.2 `.changeset/feedback-survey-0-11-3.md`, `patch`.
- [x] 4.3 `pnpm typecheck && pnpm lint && pnpm --filter @taskless/cli test`.
- [x] 4.4 Archive the change on this PR.
Loading
Loading