Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/check-unverified-remedy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@taskless/cli": patch
---

`taskless check` now says when it ran rules without verifying them, and what would let it verify them. Logged out, `--anonymous`, no usable GitHub `origin`, or a reconcile that cannot complete each print one notice naming the fix: `auth login` or a `TASKLESS_TOKEN` in CI, dropping `--anonymous`, the specific remote problem, or the GitHub App installation steps `rule create` already gives. The notice prints whether or not the project has runtime rules, so an unauthenticated CI job no longer runs its ast-grep and Vale rules unverified without a word. Under `--json` it is an entry in `notices`; the exit code is unchanged.
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
## Why

When `check` cannot have the rule service verify a project's rules, it says
that runtime rules did not run but never says what would let them run. The
three early paths (`--anonymous`, no token, no resolvable GitHub remote)
attached a reason to each skipped runtime rule and printed no notice, so a
project with no runtime rules got no word at all that its ast-grep and Vale
rules ran unverified. The remote failure discarded the specific problem
`resolveRepositoryUrl` had already identified, and the `organization_not_found`
cause named the condition without the remedy `rule create` gives for it.

Server verification of issued rules is new in 0.12.0. A CI job without a token
is the likeliest place to hit the unauthenticated path, and that path was
silent whenever there were no runtime rules.

The standing spec required the opposite for the logged-out case: "SHALL NOT
emit a warning about missing authentication", from a design that kept routine
offline use quiet. That design also allowed "an informational line", and this
change uses that allowance: a `Notice:`, not a warning, with the exit code
unchanged.

## What Changes

- Every path that leaves rules unverified (`--anonymous`, no token, no
resolvable GitHub remote, a reconcile that cannot complete) prints ONE
notice saying the rules were not verified and naming the fix. It prints
whether or not the project has runtime rules.
- The fix named per cause: `--anonymous` names running without it; no token
names `auth login` and `TASKLESS_TOKEN`; a remote failure names which of
not-a-repository, no `origin`, or a non-GitHub `origin` it is; a rejected
token names `auth login`; `organization_not_found` carries the same two
steps as `orgNotFoundMessage()`, now shared through `orgNotFoundRemedy()`.
- Each skipped runtime rule's reason becomes the short cause, since the
notice carries the remedy.
- The `check` recipe goes to topic v6 and describes the notice.

## Capabilities

### New Capabilities

None.

### Modified Capabilities

- `cli-check`: one ADDED requirement for the notice. Two MODIFIED
requirements, restated in full: "Check selects what it runs from auth state"
drops "SHALL NOT emit a warning about missing authentication", and "Check
accepts --anonymous as a no-op" lets the notice name `--anonymous` as the
cause instead of matching the unauthenticated output byte for byte.

## Impact

Additive stderr output on unverified runs, and an entry in the `--json`
`notices` array where there was none. `success`, `results`, `skipped` and the
exit code are unchanged; `skipped[].reason` is reworded. `patch`: the package
is pre-1.0.

## Delivery shape

**Single PR.** Spec, implementation, recipe and tests are one small diff, and
the change is archived in it.
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
## ADDED Requirements

### Requirement: Check names the remedy when rules run unverified

Whenever `taskless check` runs rules without verifying them, because `--anonymous` is set,
no token is available, no GitHub repository URL resolves, or reconciliation cannot complete,
it SHALL emit exactly one notice that says the rules were not verified, states the cause, and
names what would let verification happen. The notice SHALL be emitted whether or not the
project has runtime rules. It SHALL NOT change the exit code. It SHALL be a single line, cause
and remedy together, so that under `--json` it is one entry in the `notices` array; in human
output it SHALL be written to stderr marked as a notice. `--dangerously-run-scripts` is excluded: it carries its own warning.

The remedy SHALL be specific to the cause:

- `--anonymous`: run `check` without `--anonymous`.
- No token: `taskless auth login`, or a `TASKLESS_TOKEN` where `check` runs in CI.
- No repository URL: which of the three problems applies (not a git repository, no `origin`
remote, or an `origin` that is not GitHub) and the step for that problem.
- A rejected token: re-authenticate with `taskless auth login`, or replace the token.
- `404 organization_not_found`: the same steps `rule create` gives for that code, confirming
the Taskless GitHub App installation covers the repository and re-authenticating with
`taskless auth login`.

#### Scenario: Logged out with no runtime rules still says so

- **WHEN** a user runs `taskless check` with no available token in a project with no runtime rules
- **THEN** the CLI SHALL emit one notice that the rules were not verified
- **AND** the notice SHALL name `auth login`
- **AND** the exit code SHALL NOT change because of the notice

#### Scenario: Anonymous names the flag, not authentication

- **WHEN** a user runs `taskless check --anonymous`
- **THEN** the notice SHALL name `--anonymous` as the cause and running without it as the remedy

#### Scenario: A missing origin is named as such

- **WHEN** an authenticated `check` runs in a git repository with no `origin` remote
- **THEN** the notice SHALL say the repository has no `origin` remote and how to add one

#### Scenario: Organization not found carries the installation remedy

- **WHEN** reconciliation returns `404 organization_not_found`
- **THEN** the notice SHALL name confirming the Taskless GitHub App installation and re-authenticating with `auth login`

#### Scenario: The notice is a notices entry under --json

- **WHEN** `taskless check --json` runs unverified
- **THEN** the `notices` array SHALL contain the notice
- **AND** `success`, `results`, and `skipped` SHALL keep their existing meaning

## MODIFIED Requirements

### Requirement: Check accepts --anonymous as a no-op

The `taskless check` command SHALL accept the global `--anonymous` flag (per the `cli`
capability). Because `check` reconciles against the Taskless API when authenticated,
`--anonymous` SHALL force the logged-out path: it SHALL suppress the reconcile network call
and run all local static rules. Aside from forcing the logged-out path, `--anonymous` SHALL NOT
change scan behavior, output shape, or exit codes relative to an unauthenticated `check`. The
not-verified notice SHALL name `--anonymous` as its cause rather than authentication.

#### Scenario: check --anonymous skips reconciliation

- **WHEN** a user runs `taskless check --anonymous`
- **THEN** the CLI SHALL NOT call `POST /cli/api/v2/reconcile`
- **AND** SHALL scan all local static rules

#### Scenario: check --anonymous matches an unauthenticated check

- **WHEN** a user runs `taskless check --anonymous`
- **THEN** its scan behavior, output shape, and exit code SHALL match `taskless check` run with
no available token
- **AND** only the cause and remedy named in the not-verified notice SHALL differ

### Requirement: Check selects what it runs from auth state

`taskless check` SHALL NOT require authentication, and it SHALL choose what it verifies from
the current auth state. When a token is available and `--anonymous` is not set, the CLI SHALL
reconcile **every** rule (ast-grep, Vale, and runtime) and apply the verdict policy of the
`cli-rule-reconciliation` capability. When no token is available, when `--anonymous` is set, or
when reconciliation cannot complete, the CLI SHALL run every static rule (ast-grep and Vale)
unverified and SHALL skip runtime execution unless `--dangerously-run-scripts` is set. The
unauthenticated path SHALL succeed with no network access. It SHALL report that the rules were
not verified as an informational notice ("Check names the remedy when rules run unverified"),
and SHALL NOT fail or change the exit code because no token is available.

#### Scenario: Unauthenticated check runs static rules and skips runtime rules

- **WHEN** a user runs `taskless check` with no available token
- **THEN** the CLI SHALL scan all static rules
- **AND** SHALL NOT call `POST /cli/api/v2/reconcile`
- **AND** SHALL skip runtime rules
- **AND** SHALL emit the not-verified notice naming `auth login`, without changing the exit code

#### Scenario: Authenticated check reconciles runtime rules

- **WHEN** a user runs `taskless check` with an available token and without `--anonymous`
- **THEN** the CLI SHALL reconcile its ast-grep, Vale, and runtime rules in one request
- **AND** SHALL run or execute each rule according to its verdict

#### Scenario: Anonymous forces the logged-out path

- **WHEN** a user runs `taskless check --anonymous` while a token is available
- **THEN** the CLI SHALL behave exactly as an unauthenticated `check` (static rules run, runtime rules skipped, no reconcile call)

#### Scenario: Logged out, an edited static rule still runs

- **WHEN** a user runs `taskless check` with no available token and an issued sg or vale rule has been edited
- **THEN** the edited rule SHALL run with no signature enforcement
- **AND** runtime rules SHALL be skipped
- **AND** the exit code SHALL NOT change because the rule was edited

#### Scenario: Logged in, an edited rule of any engine does not run

- **WHEN** an authenticated `check` reconciles and a rule of any engine is `unsafe`
- **THEN** that rule SHALL NOT run or execute
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
## 1. Spec

- [x] 1.1 ADDED requirement for the not-verified notice; MODIFIED "Check
selects what it runs from auth state" and "Check accepts --anonymous as
a no-op", each restated in full.
- [x] 1.2 Dry-run `openspec archive` and compare the scenario count in
`cli-check` before and after.

## 2. Implementation

- [x] 2.1 `plan-check.ts`: one notice per unverified plan, with cause and
remedy, regardless of runtime rule count.
- [x] 2.2 Name the specific remote problem from the `CLIError` code.
- [x] 2.3 Extract `orgNotFoundRemedy()` from `orgNotFoundMessage()` and use it
for `organization_not_found` in `check`.
- [x] 2.4 `check` recipe topic v6.

## 3. Tests

- [x] 3.1 Logged out with and without runtime rules, human and `--json`.
- [x] 3.2 `--anonymous`, the three remote problems, `organization_not_found`.
- [x] 3.3 Tests that asserted `notices` absent on a clean logged-out `--json`
run now assert the not-verified notice is the only one.
64 changes: 58 additions & 6 deletions openspec/specs/cli-check/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,8 @@ The `taskless check` command SHALL accept the global `--anonymous` flag (per the
capability). Because `check` reconciles against the Taskless API when authenticated,
`--anonymous` SHALL force the logged-out path: it SHALL suppress the reconcile network call
and run all local static rules. Aside from forcing the logged-out path, `--anonymous` SHALL NOT
change scan behavior, output shape, or exit codes relative to an unauthenticated `check`.
change scan behavior, output shape, or exit codes relative to an unauthenticated `check`. The
not-verified notice SHALL name `--anonymous` as its cause rather than authentication.

#### Scenario: check --anonymous skips reconciliation

Expand All @@ -247,8 +248,9 @@ change scan behavior, output shape, or exit codes relative to an unauthenticated
#### Scenario: check --anonymous matches an unauthenticated check

- **WHEN** a user runs `taskless check --anonymous`
- **THEN** its scan behavior, output, and exit code SHALL match `taskless check` run with no
available token
- **THEN** its scan behavior, output shape, and exit code SHALL match `taskless check` run with
no available token
- **AND** only the cause and remedy named in the not-verified notice SHALL differ

### Requirement: Check error output uses standardized error envelope

Expand All @@ -268,16 +270,17 @@ reconcile **every** rule (ast-grep, Vale, and runtime) and apply the verdict pol
`cli-rule-reconciliation` capability. When no token is available, when `--anonymous` is set, or
when reconciliation cannot complete, the CLI SHALL run every static rule (ast-grep and Vale)
unverified and SHALL skip runtime execution unless `--dangerously-run-scripts` is set. The
unauthenticated path SHALL succeed with no network access and SHALL NOT emit a warning about
missing authentication.
unauthenticated path SHALL succeed with no network access. It SHALL report that the rules were
not verified as an informational notice ("Check names the remedy when rules run unverified"),
and SHALL NOT fail or change the exit code because no token is available.

#### Scenario: Unauthenticated check runs static rules and skips runtime rules

- **WHEN** a user runs `taskless check` with no available token
- **THEN** the CLI SHALL scan all static rules
- **AND** SHALL NOT call `POST /cli/api/v2/reconcile`
- **AND** SHALL skip runtime rules
- **AND** SHALL NOT emit a warning about missing authentication
- **AND** SHALL emit the not-verified notice naming `auth login`, without changing the exit code

#### Scenario: Authenticated check reconciles runtime rules

Expand Down Expand Up @@ -734,3 +737,52 @@ nothing to report.
- **WHEN** reconciliation returns vale rule `bar-2` in `unknown` with `copyOf: { ruleId: "foo-1", revisionId: "r1", files: [{ path: ".vale.ini", expected, got }] }` and returns `foo-1` as `missing` with `revisionId` `r2`
- **THEN** `integrity` SHALL include `{ ruleId: "bar-2", engine: "vale", verdict: "unknown", files: [{ path: ".vale.ini", expected, got }], copyOf: { ruleId: "foo-1", revisionId: "r1", sourceMissing: true } }`
- **AND** SHALL include `{ ruleId: "foo-1", engine: "vale", verdict: "missing", revisionId: "r2" }`

### Requirement: Check names the remedy when rules run unverified

Whenever `taskless check` runs rules without verifying them, because `--anonymous` is set,
no token is available, no GitHub repository URL resolves, or reconciliation cannot complete,
it SHALL emit exactly one notice that says the rules were not verified, states the cause, and
names what would let verification happen. The notice SHALL be emitted whether or not the
project has runtime rules. It SHALL NOT change the exit code. It SHALL be a single line, cause
and remedy together, so that under `--json` it is one entry in the `notices` array; in human
output it SHALL be written to stderr marked as a notice. `--dangerously-run-scripts` is excluded: it carries its own warning.

The remedy SHALL be specific to the cause:

- `--anonymous`: run `check` without `--anonymous`.
- No token: `taskless auth login`, or a `TASKLESS_TOKEN` where `check` runs in CI.
- No repository URL: which of the three problems applies (not a git repository, no `origin`
remote, or an `origin` that is not GitHub) and the step for that problem.
- A rejected token: re-authenticate with `taskless auth login`, or replace the token.
- `404 organization_not_found`: the same steps `rule create` gives for that code, confirming
the Taskless GitHub App installation covers the repository and re-authenticating with
`taskless auth login`.

#### Scenario: Logged out with no runtime rules still says so

- **WHEN** a user runs `taskless check` with no available token in a project with no runtime rules
- **THEN** the CLI SHALL emit one notice that the rules were not verified
- **AND** the notice SHALL name `auth login`
- **AND** the exit code SHALL NOT change because of the notice

#### Scenario: Anonymous names the flag, not authentication

- **WHEN** a user runs `taskless check --anonymous`
- **THEN** the notice SHALL name `--anonymous` as the cause and running without it as the remedy

#### Scenario: A missing origin is named as such

- **WHEN** an authenticated `check` runs in a git repository with no `origin` remote
- **THEN** the notice SHALL say the repository has no `origin` remote and how to add one

#### Scenario: Organization not found carries the installation remedy

- **WHEN** reconciliation returns `404 organization_not_found`
- **THEN** the notice SHALL name confirming the Taskless GitHub App installation and re-authenticating with `auth login`

#### Scenario: The notice is a notices entry under --json

- **WHEN** `taskless check --json` runs unverified
- **THEN** the `notices` array SHALL contain the notice
- **AND** `success`, `results`, and `skipped` SHALL keep their existing meaning
12 changes: 9 additions & 3 deletions packages/cli/src/agent/check.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Topic: check (CLI v%(CLI_VERSION)s / topic v5)
# Topic: check (CLI v%(CLI_VERSION)s / topic v6)

## Goal
Run the applicable rules against the codebase and report matches. Two
Expand Down Expand Up @@ -40,7 +40,12 @@ logged in:
- **Logged out, `--anonymous`, no GitHub remote, or service
unavailable**: nothing is verified. ast-grep and Vale rules run as they
are on disk, runtime rules are **skipped** (reported, never run), and
the exit code is unaffected.
the exit code is unaffected. `check` prints ONE notice saying the rules
were not verified and naming the fix: `%(TASKLESS_CLI)s auth login` (or
a `TASKLESS_TOKEN` secret in CI), dropping `--anonymous`, the specific
remote problem, or the GitHub App installation. It prints whether or not
the project has runtime rules, because the static rules ran unverified
either way. Pass the fix on to the user rather than ignoring it.
- **`--dangerously-run-scripts`**: nothing is verified and no network
call is made, logged in or not. Every rule of every engine runs,
runtime included, behind a prominent warning. This is the only way to
Expand All @@ -55,7 +60,8 @@ plan does not include restoring rules, the git steps that do instead
Notices about skipped runtime rules are human-readable stderr only.
Under `--json` they do NOT appear as warnings; instead an additive
optional `skipped: [{ rule, reason }]` array is included alongside the
unchanged `{ success, results }`, and the CI backstop
unchanged `{ success, results }`, the not-verified notice is an entry
in `notices`, and the CI backstop
(`%(TASKLESS_CLI)s agent ci`) is the enforcement point.

## An edited rule
Expand Down
15 changes: 13 additions & 2 deletions packages/cli/src/rules/generate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,22 @@ export function orgNotFoundMessage(): string {
"",
"Most often the organization's Taskless GitHub App installation does not cover this repository. It can also mean your login no longer has access to the organization.",
"",
"- Confirm the Taskless app is installed on this repository's owner and includes this repository.",
`- If access recently changed, re-authenticate with \`${getCliPrefix()} auth login\`.`,
...orgNotFoundRemedy().map((step) => `- ${step}`),
].join("\n");
}

/**
* The steps that resolve `organization_not_found`, one sentence each. Shared
* with `check`, which reports the same condition and must name the same
* remedy.
*/
export function orgNotFoundRemedy(): string[] {
return [
"Confirm the Taskless app is installed on this repository's owner and includes this repository.",
`If access recently changed, re-authenticate with \`${getCliPrefix()} auth login\`.`,
];
}

/** A request's terminal status. */
export type FinishedRequest = RequestStatus;

Expand Down
Loading
Loading