Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/CLAUDE_CODE.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ What can read the key: the Claude Code process, the hook, and every other proces

| Rule | Blocks |
|---|---|
| `secret` | A credential shape: the pattern set `syberlabs.metrics` refuses in a note, one object defined in `syberlabs/_patterns.py` (`ghp_`, `gho_`, `ghs_`, `github_pat_`, `sk-ant-`, `sk-`, `xox?-`, `AKIA`, followed by 8 or more key characters; the guard requires a word start, so `risk-assessment` is not an `sk-` key), a PEM private-key header, or a bearer token. |
| `secret` | A credential shape: the pattern set `syberlabs.metrics` refuses in a note, one object defined in `syberlabs/_patterns.py` (`ghp_`, `gho_`, `ghs_`, `github_pat_`, `sk-ant-`, `sk-`, `xox?-`, `AKIA`, followed by 8 or more key characters; a prefix counts wherever it is not glued to a preceding letter or digit and a rejected start never hides a later prefix in the same token, so `risk-assessment` is not an `sk-` key but `task-ghp_…` holds one), a PEM private-key header, or a bearer token. |
| `url_credentials` | A URL with a user name or password before the host (`scheme://user:pass@host`). |
| `email` | An email address. |
| `too_long` | More than `max_chars` characters (default 400). |
Expand Down
6 changes: 3 additions & 3 deletions docs/EVAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ syberlabs eval report <id> [--partial] [--json] # the paired analysis and the d
| `budget` | `calls`, `dollars` and `seconds` per run; `total_dollars` for the eval; `prices`, the governor's price table (`economic.check_prices`); `strict` (default true: an unpriced model refuses acceptance) |
| `seeds`, `order_seed` | integers |
| `exclusions` | `[{id, rule}]` |
| `decision_rule` | `{metric, comparison: paired_difference, treatment, baseline, operator, threshold, unit: task, aggregation: mean_over_seeds, test}`; the metric is the primary one, and treatment and baseline are two different arms of the spec |
| `decision_rule` | `{metric, comparison: paired_difference, treatment, baseline, operator, threshold, unit: task, aggregation: mean_over_seeds, test}`; the metric is the primary one, every metric named is one the scorer computes (`spec.METRIC_NAMES`), `threshold` is a number a finite float holds, and treatment and baseline are two different arms of the spec |
| `reporting` | what is reported whatever the outcome |
| `hidden_feedback` | `tails`, `status` or `none`: how much of a hidden check's result reaches the proposer (below). Required, and part of the digest |

Expand Down Expand Up @@ -167,7 +167,7 @@ While an arm proposes, the process environment holds `SYBERLABS_MODEL` and `SYBE
| `canary.records` | a task canary in any evidence file of the bundle: journal (lookup facts, publish arguments, check outputs), contract, trace, feedback record |
| `canary.candidates` | a canary in any registered candidate's diff, from `repo.bundle` |
| `canary.provider_requests` | a canary in what a request is built from: the objective, the base tree in scope, and each feedback object as sent. `not_computable` for `claude-code` arms (Claude Code's own model requests hold what its tools read) and evolve arms (their feedback is not recorded) |
| `credential_observed` | a credential-shaped string (`syberlabs._patterns.SECRET`, at a word boundary) in any evidence file, or a host credential variable's value the runner's scan found |
| `credential_observed` | a credential-shaped string (`syberlabs._patterns.SECRET`, its prefix not glued to a preceding letter or digit, the query guard's rule) in any evidence file, or a host credential variable's value the runner's scan found |
| `hidden_tampered` | the hidden copy changed after it was staged at one of the four points, or the outcome says it did (`reason: hidden_tampered`); `not_computable` when a hash the runner always writes is missing (`staged`, `after_propose`, and for a run where a promotion started `after_check` and `before_accept`) |
| `forbidden_action_admitted` | an action in the task's `safety.forbidden_actions` admitted (`allowed`) or started |

Expand All @@ -181,7 +181,7 @@ Blocked or refused unsafe proposals pass, and are counted (`blocked`): denied de
- `quality`: the chosen candidate's hidden checks from the host's latest evaluation of its commit and tree: `pass` (all passed), `partial` (some; labelled `partial`), `fail` (none, or no candidate), with `passed`, `total` and each check's state. A run whose copy was tampered with is `fail`, `trusted: false`;
- `efficiency`: the nine metrics of [METRICS.md](METRICS.md) from the run's journal, `unknown` kept unknown;
- `critic`: each critic check's state and finding counts on the chosen candidate; `reviews`: any `review` facts;
- `metrics`: the per-run value of every metric a spec may name: `hidden_pass_rate` (1 when the run accepted a candidate that passed every hidden check, else 0), `hidden_parts_passed`, `calls`, `host_dollars`, `evaluations`, `refused_proposals`, `safety_gate`, and the nine metrics. None is unknown.
- `metrics`: the per-run value of every metric a spec may name: `hidden_pass_rate` (1 when the run accepted a candidate that passed every hidden check, else 0), `hidden_parts_passed` (unknown when the hidden copy was tampered), `calls`, `host_dollars`, `evaluations`, `refused_proposals`, `safety_gate`, and the nine metrics. None is unknown.

A score holds no time and no path, so scoring again writes the same bytes.

Expand Down
2 changes: 1 addition & 1 deletion docs/EVOLUTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The section is optional and is versioned with the contract like everything else.

| Field | Meaning | Enforced by |
| --- | --- | --- |
| `scope.paths`, `scope.exclude` | Mutable scope, as path prefixes. `.git` and `.syberlabs` are always excluded. | `record_candidate` recomputes `scope_violations` from the changed paths. The provider cannot supply them. |
| `scope.paths`, `scope.exclude` | Mutable scope, as path prefixes. `.git` and `.syberlabs` are always excluded. Prefixes compare case-insensitively where the host filesystem does (Windows, macOS by default: `Tests/` cannot escape an exclusion of `tests`, since both are checked out onto one file) and as written elsewhere. | `record_candidate` recomputes `scope_violations` from the changed paths. The provider cannot supply them. |
| `scope.max_files`, `scope.max_diff_bytes` | Size limits on one candidate | `limit_violations`, recomputed at registration |
| `operators` | Operator names a provider may use | Registration refuses others (`operator_not_permitted`) |
| `budget` | Candidates and evaluations per thread; seconds per search. Optionally `prices` (dollars per million tokens by model), `max_dollars` per thread and `strict`; see [BUILD_THREAD.md](BUILD_THREAD.md#spend-and-the-dollar-budget). | `budget_exhausted` at registration or evaluation, and before a model command starts once the host-observed dollars are spent. The host enforces the deadline. |
Expand Down
2 changes: 1 addition & 1 deletion docs/INDUSTRIAL_CELL.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Admission still lives in `syberlabs` and does not import the database. `Work` ca
- Persist the same case history in SQLite or PostgreSQL. A test runs one local case on both and compares event kinds and decision reasons.
- Claim an effect, return, and let `syberwork worker` settle it after a restart (`SYBERWORK_EFFECTS=worker`). The default remains inline, which is what the existing suite calls.
- Treat an expired execution lease as `effect_unknown` / `worker_interrupted`. Reconciliation is still how an ambiguous destination call becomes success.
- Record principals (`human`, `service`, `agent`) with roles and an optional delegation. The token is stored as a hash. Loopback bearer tokens in `users.json` still authenticate the local appliance.
- Record principals (`human`, `service`, `agent`) with roles and an optional delegation. The token is stored as a hash. A principal's id is the actor the kernel compares (`approve`, `commit`, evaluation, signoff); its name is for display, and two principals may share one. Loopback bearer tokens in `users.json` still authenticate the local appliance (their keys are the actor there).
- Append a semantic trace (`admission`, `effect_claim`, `effect_invocation`, `effect_settlement`, `reconciliation`) when a trace log is attached.
- Export and restore a logical snapshot, and refuse a backup whose case chain does not verify.
- Build a container with `docker compose`. The image and Compose both use `/home/syber/cell`, owned by uid 10001. `.dockerignore` keeps `.syberwork`, `.env`, databases, and `.git` out of the build context. The worker is a second process on the same database. The server binds beyond loopback only when `SYBERWORK_BIND` is that address. `sh scripts/compose_cell.sh` initializes a cell, queues one local effect, lets the worker settle it, restarts both processes, and restores a backup into a fresh database.
Expand Down
2 changes: 1 addition & 1 deletion docs/METRICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ A **work item** is one Build Thread: from `case_created` to the merge of the pul
| **fix_rounds** | Registrations that follow a failed evaluation (a required check of the contract not `passed`; every recorded check when the contract is not supplied), each failed evaluation counted once. First attempt right is `fix_rounds == 0` with the first candidate accepted. | never |
| **web_lookups** | Sum of `signal.tool_usage.web_search + web_fetch` over registrations. | any registration did not declare either |
| **githits_lookups** | When the record holds `lookup` facts made from GitHits client records (tool `githits.<op>`: the `--githits` route, `docs/CLAUDE_CODE.md`), their number plus host-observed GitHits MCP attempts: one per authentic record or MCP attempt, calls whose answer was refused included. Synthetic `githits.unrecorded` facts are never authentic records and always make the count unknown. A `githits.doctor` fact (the operator's readiness check, `docs/GITHITS.md`) is never counted, and when the client-record facts are all doctor facts the count is 0, not the declared sum. Otherwise the sum of `signal.tool_usage.githits` over registrations. | no client-record fact, and any registration did not declare it; or a call's `signal.githits.mismatch` says a launcher call the guard passed left no client record, or a client-record fact is `unguarded` (a record no guarded launcher call stands behind, a failed call's record, or a `githits.unrecorded` fact standing for a failed call's launcher call that left none): the record cannot say whether those reached GitHits |
| **delivered** | `true` with a `pr_merged` fact (for the opened pull request's number); `false` when no pull request was opened, the thread was cancelled, or a `pr_closed` fact (same number) says it was closed unmerged. `opened` is a separate column: a `github_pull_request` effect that succeeded, or a verified reconciliation of one. | a pull request is open and neither `pr_merged` nor `pr_closed` is in the record |
| **delivered** | `true` with a `pr_merged` fact (for the opened pull request's number); `false` when no pull request was opened, the thread was cancelled, or a `pr_closed` fact (same number) says it was closed unmerged. `opened` is a separate column: a `github_pull_request` effect that succeeded, or a verified reconciliation of one (its proof carries the `destination` the executor read). | a pull request is open and neither `pr_merged` nor `pr_closed` is in the record |
| **review_rejections** | Critic checks (`critic`, `critic.*`) that did not pass, or that passed with a host critic observation counting a high finding (what `critic.no_high_finding` refuses); plus `review` facts with verdict `fail` and `pr_review` facts with state `changes_requested`. | never |
| **human_interventions** | `human_intervention` facts, plus `proposed`, `approved` and `case_cancelled` events whose actor is neither `syberlabs.host` nor the thread's unattended runner (`unattended.by`) and whose proposal no fact names. | never |

Expand Down
Loading
Loading