Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
# One organization, one database. Run `syberwork init` before `serve`.
# The home directory is /home/syber/cell, owned by uid 10001, the same path the image creates.
# SYBERWORK_DB_PASSWORD must be set (there is no default password); scripts/compose_cell.sh generates one.
# The service listens on 0.0.0.0 inside its container so Docker can forward to it, and Docker publishes that
# port on the host's loopback only. Widening the mapping is the operator's decision on the host (docs/DEPLOY.md).
services:
db:
image: postgres:16
environment:
POSTGRES_USER: syber
POSTGRES_PASSWORD: syber
POSTGRES_PASSWORD: ${SYBERWORK_DB_PASSWORD:?set SYBERWORK_DB_PASSWORD for the cell database}
POSTGRES_DB: syberwork
volumes:
- syberwork-db:/var/lib/postgresql/data
Expand All @@ -18,13 +21,13 @@ services:
build: .
environment: &cell
SYBERWORK_HOME: /home/syber/cell
SYBERWORK_DATABASE: postgresql://syber:syber@db:5432/syberwork
SYBERWORK_DATABASE: postgresql://syber:${SYBERWORK_DB_PASSWORD:?set SYBERWORK_DB_PASSWORD for the cell database}@db:5432/syberwork
SYBERWORK_ORGANIZATION: example
SYBERWORK_BIND: 0.0.0.0
SYBERWORK_EFFECTS: worker
SYBERWORK_TRACE: "1"
ports:
- "8766:8766"
- "127.0.0.1:8766:8766"
depends_on:
db:
condition: service_healthy
Expand Down
2 changes: 1 addition & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,6 @@ For each past proposal, replay evaluates a selected new contract and policy agai

## Integration

HTTP sources and effects are bound by administrator-installed definitions and do not accept an LLM-provided destination URL. Authorization headers use environment variable names in definitions; secrets never enter the contract. Cross-origin redirects are blocked. Source URLs and the planner URL go through `trusted_origin`. HTTPS must name a public host; loopback HTTP stays `127.0.0.1` with a port. `guard_request` checks the addresses HTTPS resolves to before the call. The signing key for a chain head is not stored in the case database. `python -m syberlabs.witness` holds that seed in a separate process, checks the chain, and appends to the transparency log. `WitnessClient` keeps the public key and opens the log read-only. The log refuses to open a file that already has a `cases` table. A process running as the same OS user can still open the file writable, and an operator who holds the seed can rebuild the log. It is not a public transparency service. Model suggestions are proposals accepted under separate model-bearing credentials, never direct network effects. The included `reference_system.py` is one fully running integration. `translate_stabilize` is a tested translator for one SyberRuntime outcome, not a live adapter. Barn, OmniOS, and customer tools still need adapters matching their authority and event semantics. The other rows in `spec/MAPPINGS.md` stay proposed.
HTTP sources and effects are bound by administrator-installed definitions and do not accept an LLM-provided destination URL. Authorization headers use environment variable names in definitions; secrets never enter the contract. Cross-origin redirects are blocked. Source URLs and the planner URL go through `trusted_origin`. HTTPS must name a public host; loopback HTTP stays `127.0.0.1` with a port. `guard_request` checks the addresses HTTPS resolves to before the call and refuses a name it cannot resolve; a host with percent-escapes or other characters `urllib.request` would rewrite is refused before it is classified. The credentialed openers ignore `http_proxy`/`https_proxy` (`targets.credentialed_opener`), so a bearer token is never handed to an environment proxy, and their HTTPS connections are pinned: each resolves and checks its host again and dials only the addresses it accepted, keeping the name for SNI and Host. IANA special-purpose ranges are refused explicitly rather than by `is_global` alone. Header values (`If-Match` from `version_arg`) are refused if they hold a line break, control character or edge whitespace, before the effect is claimed. A manual attestation (`POST /api/cases/{id}/facts`) may not set a key the case's contract reads from another source or only verified, and an `observer`-only credential reads no case, contract, certificate, metric or builder view. The console document refuses framing (`frame-ancestors 'none'`). The signing key for a chain head is not stored in the case database. `python -m syberlabs.witness` holds that seed in a separate process, checks the chain, and appends to the transparency log. `WitnessClient` keeps the public key and opens the log read-only. The log refuses to open a file that already has a `cases` table. A process running as the same OS user can still open the file writable, and an operator who holds the seed can rebuild the log. It is not a public transparency service. Model suggestions are proposals accepted under separate model-bearing credentials, never direct network effects. The included `reference_system.py` is one fully running integration. `translate_stabilize` is a tested translator for one SyberRuntime outcome, not a live adapter. Barn, OmniOS, and customer tools still need adapters matching their authority and event semantics. The other rows in `spec/MAPPINGS.md` stay proposed.

The [`economic_http` effect](ECONOMIC_ACTIONS.md) adds exact atomic units and a cross-case budget reservation to this same admission boundary. Its trusted settlement service is a separate authority; SyberWork records that service's matching receipt and status assertion, not independent blockchain finality.
13 changes: 7 additions & 6 deletions docs/DEPLOY.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,18 @@ The first deployment of the factory is the SyberWork service, its worker and its

| Service | Image | Role |
| --- | --- | --- |
| `db` | `postgres:16` | The cell's database `syberwork`, user `syber`, on volume `syberwork-db`. Health-checked with `pg_isready`. |
| `syberwork` | built from `Dockerfile` | The operator service, `syberwork serve --port 8766`, published on host port 8766. Binds `0.0.0.0` inside the container because `SYBERWORK_BIND` says so; the container port is reachable only through the published port. |
| `db` | `postgres:16` | The cell's database `syberwork`, user `syber`, password `SYBERWORK_DB_PASSWORD` (required; no default), on volume `syberwork-db`. Health-checked with `pg_isready`. |
| `syberwork` | built from `Dockerfile` | The operator service, `syberwork serve --port 8766`, published on the host's loopback, `127.0.0.1:8766`. Binds `0.0.0.0` inside the container because `SYBERWORK_BIND` says so; the container port is reachable only through that published port. |
| `worker` | the same image | `syberwork worker`: leases queued effect obligations and settles them. Needed because `SYBERWORK_EFFECTS=worker` makes the service claim effects and return. |

Both processes share `SYBERWORK_HOME=/home/syber/cell` on volume `syberwork-home`, which holds `users.json` (credential hashes), `cell.json` and `traces.jsonl`. The environment (`SYBERWORK_DATABASE`, `SYBERWORK_ORGANIZATION=example`, `SYBERWORK_TRACE=1`) is the one in the file; this document changes nothing in it.
Both processes share `SYBERWORK_HOME=/home/syber/cell` on volume `syberwork-home`, which holds `users.json` (credential hashes), `cell.json` and `traces.jsonl`. The environment (`SYBERWORK_DATABASE`, `SYBERWORK_ORGANIZATION=example`, `SYBERWORK_TRACE=1`) is the one in the file; this document changes nothing in it. `users.json` is read again whenever it changes, so removing or rotating a credential there applies without restarting `serve`.

## 2. Running it on one host

The host needs Docker with the Compose plugin and a checkout of this repository. The commands are the ones `scripts/compose_cell.sh` runs, in order, without the smoke steps.

```sh
export SYBERWORK_DB_PASSWORD="<a long random value you keep>" # compose refuses to start without it
docker compose build
docker compose up -d db --wait
docker compose run --rm -T syberwork syberwork init
Expand Down Expand Up @@ -64,7 +65,7 @@ X-Record-Version: 6f1c...e2
| `effects_pending` | Claimed effects not yet settled: queued, running, or `effect_unknown` awaiting reconciliation. |
| `effects_settled_applied_total` | Claimed effects with `effect_succeeded` or a verified `reconciled`. |
| `effects_settled_not_applied_total` | Claimed effects with `effect_rejected` (a no-write refusal or `not_applied`). |
| `queue_depth` | Worker obligations queued or leased. Always 0 when effects settle inline. |
| `queue_depth` | Worker obligations queued, leased, or being recovered after an expired lease. Always 0 when effects settle inline. |
| `uptime_seconds` | Seconds since this server process started. |

Every claimed effect is in exactly one of the three effect counters. `observed_at` is the epoch time of the read, the form the effect's `max_age_seconds` rule understands; the version is a digest of key, value and `observed_at`, so a repeated read of an unchanged value is still a fresh, distinct reading. An unknown key is `404 {"error": "unknown_metric"}`. The route needs a bearer token like every other read on the service; there is no anonymous metrics endpoint.
Expand All @@ -89,8 +90,8 @@ with `SYBERWORK_METRIC_TOKEN` set in the environment of the process that runs `s
## 4. What is not there yet

- **No thread runner service.** Build Threads run from a checkout with `python -m syberlabs`; nothing in the stack starts, schedules or recovers them. That is a later packet.
- **No public exposure and no TLS.** Docker publishes port 8766 on the host's interfaces as `docker-compose.yml` is written; whether the host firewall closes it, or the mapping is narrowed to `127.0.0.1:8766:8766`, is the operator's decision on the host. The service speaks plain HTTP; the intervention effect will only read it over loopback.
- **No secrets broker.** The database password is the literal in `docker-compose.yml`, the credentials are what `init` printed, and the thread's metric token lives in an environment variable. Rotation is `down -v` and a new `init`.
- **No public exposure and no TLS.** Docker publishes port 8766 on the host's loopback only (`127.0.0.1:8766:8766`); widening that mapping, and the firewall that would then guard it, is the operator's decision on the host. The service speaks plain HTTP; the intervention effect will only read it over loopback.
- **No secrets broker.** The database password is whatever `SYBERWORK_DB_PASSWORD` held when the `syberwork-db` volume was first created (a volume created under the old literal `syber` keeps it until `down -v`), the credentials are what `init` printed, and the thread's metric token lives in an environment variable. Rotation is `down -v` and a new `init`.
- **No principal management from the command line.** `Work.register_principal` exists for cell principals with a token hash and a delegation; the CLI exposes only the `init` credentials.
- **Single host, single worker.** One Compose project, one database, one worker process. `syberwork backup` is the only redundancy.
- **Counters are read on request.** `cases_open` reads every case's history on each call; fine for a cell this size, not a design for thousands of cases.
Expand Down
6 changes: 6 additions & 0 deletions scripts/compose_cell.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ set -euo pipefail

cd "$(dirname "$0")/.."

# docker-compose.yml has no default database password. A throwaway cell gets a random one.
if [ -z "${SYBERWORK_DB_PASSWORD:-}" ]; then
SYBERWORK_DB_PASSWORD="$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')"
fi
export SYBERWORK_DB_PASSWORD

docker compose build
docker compose up -d db --wait
docker compose run --rm -T syberwork syberwork init
Expand Down
2 changes: 1 addition & 1 deletion spec/GAPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The schemas describe SyberWork as it behaves. They are not enforced at runtime e

3. **`install_contract` checks the fields admission indexes, including input kinds.** An acceptance `effect` without `action`, a `signoff` without `role`, a `fact` without `key`, or a `required_facts` item without `key` and `source` is rejected at publish. An input kind other than `string` or `integer` is `invalid_contract`. `create_case` rejects a value of the wrong kind with `input_schema`, including a boolean passed where an integer is required.

4. **`trusted_origin` is procedural.** Source install and `SYBERWORK_PLANNER_URL` now call it. The public errors stay `invalid_source` and `planner_unconfigured`. A host that only begins with `http://127.0.0.1:` is rejected. HTTPS rejects non-public addresses, `localhost`, metadata hostnames, and the suffixes `.local`, `.localhost`, `.internal`, and `.localdomain`. `SYBERWORK_HTTPS_HOSTS`, when set, is an extra hostname allowlist. The schemas still do not encode the parser (no userinfo, no fragment, path required, HTTP only for `127.0.0.1` with a port). DNS rebinding is checked at request time by `guard_request`, not at install.
4. **`trusted_origin` is procedural.** Source install and `SYBERWORK_PLANNER_URL` now call it. The public errors stay `invalid_source` and `planner_unconfigured`. A host that only begins with `http://127.0.0.1:` is rejected. HTTPS rejects non-public addresses, `localhost`, metadata hostnames, and the suffixes `.local`, `.localhost`, `.internal`, and `.localdomain`. `SYBERWORK_HTTPS_HOSTS`, when set, is an extra hostname allowlist. The schemas still do not encode the parser (no userinfo, no fragment, path required, HTTP only for `127.0.0.1` with a port). The host must be an IP literal or plain ASCII letter-digit-hyphen labels, so the name checked is the name `urllib.request` dials (a percent-escaped host is refused). DNS rebinding is checked at request time, not at install: `guard_request` refuses a name it cannot resolve, and the credentialed HTTPS connection resolves again and dials only the addresses it accepted. Installed action URLs and the planner URL carry no query, and a source URL carries exactly one `{key}`, in its path.

5. **Proposals are appended before admission.** A non-dict `args` or a missing role list is stored, then denied with `invalid_proposal_shape`. The Proposal schema therefore does not require `args` to be an object.

Expand Down
7 changes: 6 additions & 1 deletion syberlabs/evidence.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,12 @@ def acceptance_results(contract: dict, history: list) -> list[dict]:
and e["body"]["action"] == clause.get("after_action")), 0)
passed = any(e["kind"] == "signed" and e["body"]["role"] == clause["role"] and (not clause.get("after_action") or e["seq"] > completed_seq > 0) for e in history)
elif clause["kind"] == "fact":
passed = any(e["kind"] == "observed" and e["body"]["key"] == clause["key"] and e["body"]["value"] == clause.get("equals") for e in history)
# The newest observation of the key decides, as admission reads facts; an older matching value does
# not. A clause that asks for ``verified`` (or names a ``source``) is held to it.
latest = next((e for e in reversed(history) if e["kind"] == "observed" and e["body"]["key"] == clause["key"]), None)
passed = (latest is not None and latest["body"]["value"] == clause.get("equals")
and (not clause.get("verified") or latest["body"].get("verified") is True)
and ("source" not in clause or latest["body"].get("source") == clause["source"]))
else:
passed = False
results.append({"id": clause["id"], "passed": passed})
Expand Down
2 changes: 1 addition & 1 deletion syberlabs/outcome.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ def _read(self) -> Reading:
headers["Authorization"] = "Bearer " + token
try:
guard_request(self.url)
except Rejected:
except (Rejected, OSError): # OSError: the guard could not resolve the host (URLError)
raise MetricError("metric_unavailable") from None
request = urllib.request.Request(self.url, headers=headers, method="GET")
try:
Expand Down
8 changes: 5 additions & 3 deletions syberlabs/planner.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

from syberlabs.canonical import canonical
from syberlabs.errors import Rejected
from syberlabs.targets import guard_request, trusted_origin
from syberlabs.targets import credentialed_opener, guard_request, trusted_origin


class PlannerRefusal(Rejected):
Expand Down Expand Up @@ -50,7 +50,7 @@ def redirect_request(self, req, fp, code, msg, headers, newurl):
return None


_HTTP = urllib.request.build_opener(_NoRedirect)
_HTTP = credentialed_opener(_NoRedirect)


class HttpPlanner:
Expand All @@ -59,19 +59,21 @@ class HttpPlanner:
def ensure_configured(self) -> str:
target = os.getenv("SYBERWORK_PLANNER_URL", "")
try:
if "?" in target: # a planner URL names an endpoint; a credential in its query would be logged and sent
raise Rejected("invalid_target", "planner URL takes no query string")
trusted_origin(target)
except Rejected:
raise PlannerRefusal("planner_unconfigured", "configure an HTTPS or local planner endpoint") from None
return target

def propose(self, context: Mapping[str, Any]) -> dict[str, Any]:
target = self.ensure_configured()
guard_request(target)
headers = {"Content-Type": "application/json"}
if os.getenv("SYBERWORK_PLANNER_TOKEN"):
headers["Authorization"] = "Bearer " + os.environ["SYBERWORK_PLANNER_TOKEN"]
request = urllib.request.Request(target, data=canonical(dict(context)).encode(), headers=headers, method="POST")
try:
guard_request(target) # a refused address raises Rejected; an unresolved name, URLError
with _HTTP.open(request, timeout=30) as response:
raw = response.read(1024 * 1024 + 1)
if len(raw) > 1024 * 1024:
Expand Down
10 changes: 6 additions & 4 deletions syberlabs/publish.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,10 @@
from urllib.parse import quote

from syberlabs import critic, outcome
from syberlabs.checks import _stop
from syberlabs.errors import Rejected
from syberlabs.session import NoWrite
from syberlabs.targets import guard_request, trusted_origin
from syberlabs.targets import credentialed_opener, guard_request, trusted_origin

EFFECTS = ("git_push", "github_pull_request", "command", "intervention")
_REMOTE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$")
Expand All @@ -52,7 +53,7 @@ def redirect_request(self, req, fp, code, msg, headers, newurl):
return None


HTTP = urllib.request.build_opener(_NoRedirect)
HTTP = credentialed_opener(_NoRedirect)


def _line(value) -> str:
Expand Down Expand Up @@ -309,8 +310,9 @@ def _run(self, argv: list[str], case_id: str, args: dict, key: str) -> tuple[int
deadline = time.monotonic() + self.timeout
while proc.poll() is None:
if time.monotonic() > deadline or os.fstat(out.fileno()).st_size > MAX_OUTPUT:
proc.kill()
proc.wait()
# End the command and everything it started, as a check's timeout does: ``proc.kill`` ends
# one process and leaves a deploy script's children running past the publish.
_stop(proc)
return None, ""
time.sleep(0.02)
out.seek(0)
Expand Down
2 changes: 2 additions & 0 deletions syberlabs/settle.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,8 @@ def _get(target: Published, token: str, path: str, timeout: float):
guard_request(url)
except Rejected as exc:
raise PollFailed(exc.code) from None
except urllib.error.URLError: # the guard could not resolve the host; no request was made
raise PollFailed("network failure (unresolved)") from None
request = urllib.request.Request(url, method="GET", headers={
"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json",
"X-GitHub-Api-Version": "2022-11-28"})
Expand Down
Loading
Loading