Optimize

.github/workflows/rust.yml

@@ -21,8 +21,6 @@ jobs:
     steps:
       - uses: actions/checkout@v4
       - run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}
-      - name: Install Protoc
-        uses: arduino/setup-protoc@v3
       - name: Cache Cargo dependencies
         uses: actions/cache@v4
         with:

AGENTS.md

@@ -7,48 +7,85 @@ Operational guide for coding agents in this repository.
 Rust workspace for a distributed task engine with dynamic mods.
 
 Workspace members:
-- `engine` (binaries: server/client/packer)
-- `enginelib` (shared runtime API, events, tasks, plugin loader)
+- `engine` (binary: `server`)
+- `enginelib` (shared runtime API, events, tasks, plugin loader, task nucleus)
 - `enginelib/macros` (proc macros used by mods and core)
 
+> **Repo is mid-rewrite.** The earlier design (gRPC/proto, `client`/`packer`
+> binaries, and the RFC1002/1003/1004 design docs) was removed in the "rewrite:
+> clean" pass and no longer exists in the tree. Do not cite those RFCs or the
+> proto layer — they are gone. `engine/src/bin/server.rs` is currently a stub
+> (`ServerAPI::init()`, RPC commented out); the transport layer is not yet wired.
+
 ## Start here (fast orientation)
 
 1. `Cargo.toml` (workspace)
-2. `engine/src/bin/server.rs`
-3. `enginelib/src/api.rs`
-4. `enginelib/src/event.rs`
-5. `enginelib/src/plugin.rs`
-6. `engine/proto/engine.proto`
-7. `rfc/rfc1003.md` (client event hooks)
-8. `rfc/rfc1004.md` (server event hooks)
+2. `enginelib/src/api.rs` (`ServerAPI`, task queues, RocksDB, key helpers)
+3. `enginelib/src/nucleus/*` (task lifecycle: submit/lease/renew/complete/cancel/query)
+4. `enginelib/src/event.rs` (event dispatch)
+5. `enginelib/src/plugin.rs` (mod loader)
+6. `enginelib/src/config.rs` (config/token)
+7. `engine/src/bin/server.rs` (entrypoint stub)
 
 ## Build/test commands
 
 - Build all: `cargo build --workspace`
 - Test all: `cargo test --workspace`
 - Run server: `cargo run -p engine --bin server`
-- Run client: `cargo run -p engine --bin client`
 
 ## Key architecture facts
 
-- gRPC service is in `engine/src/bin/server.rs`.
-- Proto definitions in `engine/proto/engine.proto`; generated at build time by `engine/build.rs`.
-- Engine state is centralized in `enginelib::api::EngineAPI`.
-- Persistent queues are sled-backed (`engine_db` / `engine_client_db`).
+- Engine state is centralized in `enginelib::api::ServerAPI`.
+- Persistence is RocksDB-backed (`ServerAPI::db: Arc<rust_rocksdb::DB>`, store dir `engine_db`).
+- Task lifecycle operations live in `enginelib/src/nucleus/*` (`submit`, `lease`, `renew`, `complete`, `cancel`, `query`).
 - Core event system is in `enginelib/src/event.rs`; core events in `enginelib/src/events/*`.
-- Client modding hooks (RFC1003) are wired in `engine/src/bin/client.rs`.
-- Server lifecycle hooks (RFC1004 Tier 1) are wired in `engine/src/bin/server.rs`.
 - Mods are loaded from `./mods/*.rf` by `enginelib/src/plugin.rs`.
+- No network transport is wired yet — `server.rs` only builds `ServerAPI`. There is
+  no proto/gRPC layer and no `client`/`packer` binary in the current tree.
 
 ## Data/persistence model (important)
 
-Sled keys:
-- `tasks` -> `TaskQueue`
-- `executing_tasks` -> `ExecutingTaskQueue`
-- `solved_tasks` -> `SolvedTasks`
-
-`Identifier` = `(namespace, task)`.
-Task instance IDs are separate string IDs.
+RocksDB is the source of truth. Key layout (see helpers in `enginelib/src/api.rs`:
+`task_key`, `task_key_prefix`, `finished_task_key`, `finished_task_key_prefix`):
+- `t:<namespace>:<task_name>:<id>` -> serialized pending task record
+- `f:<namespace>:<task_name>:<id>` -> serialized finished task record
+
+In-memory state on `ServerAPI` (rebuilt from the DB via `load()`), keyed by
+`Identifier`:
+- `task_queue: TaskQueue` — per-type `(async_channel Sender, Receiver, dedup DashSet<id>)`.
+  A task_id is in the dedup set iff it is in-flight (queued in the channel **or** leased).
+- `leased_tasks: LeasedTaskQueue` — per-type `Vec<LeasedTask>` handed to workers,
+  reaped after `LEASE_TTL_SECS`.
+
+`Identifier` = `(namespace, task)`. Task instance IDs are separate string IDs
+(`druid` hex), generated by `submit` and **returned to the caller** so the task
+can later be addressed by `cancel`/`renew`/`complete`/`query`.
+
+### Task lifecycle (`enginelib/src/nucleus/*`)
+
+- `submit(bytes, task_type) -> String` — verifies bytes, persists the pending
+  record, returns the generated task_id. Does not touch in-memory state; `load()`
+  enqueues it.
+- `load(task_type)` — batch-reads pending records from the DB and enqueues any not
+  already in the dedup set.
+- `lease(task_type, user_id) -> StoredTask` — `recv`s from the channel and records a
+  lease (task stays in the dedup set).
+- `renew(task_type, task_id)` — resets the lease's `given_at` to now, extending its
+  TTL before the reaper collects it. Touches nothing else.
+- `complete(task_type, task_id, bytes)` — finalize a leased task: verifies bytes,
+  writes the `f:` record, deletes the `t:` record, then clears the lease + dedup entry.
+- `cancel(task_type, task_id)` — cancels the **lease, not the task**: drops the lease
+  and `try_send`s the record back into the queue (dedup entry and persisted record
+  untouched). If the channel is full/closed it clears the dedup entry so a later
+  `load()` recovers the still-persisted record. Synchronous — no await window in which
+  the task is neither leased nor queued.
+- `query(Query) -> QueryResult` — operator/CRUD surface, all O(seek) (never a
+  full keyspace scan): point `Get { key }` / `Delete { key }`, range
+  `Scan { prefix, after, limit }` / `PurgePrefix { prefix }`, and
+  `EstimateKeys` / `EstimateSize`. Build keys with the `*_key` helpers. Raw
+  `Delete`/`PurgePrefix` bypass dedup/lease state — use `cancel`/`complete` for
+  in-flight tasks. There is intentionally no raw `Put`: create/update go through
+  `submit`/`complete` so verification and in-memory invariants hold.
 
 ## Mod ABI contract (must keep)
 
@@ -57,7 +94,7 @@ Each `.rf` package must contain platform lib at root:
 
 Exported symbols required:
 - `metadata() -> LibraryMetadata`
-- `run(api: &mut EngineAPI)`
+- `run(api: &mut ServerAPI)`
 
 `#[metadata]` and `#[module]` macros are the preferred way to provide these.
 
@@ -67,33 +104,33 @@ Strict compatibility check is enabled when `GE_STRICT_MODS` is set.
 
 - User auth/admin auth are event-driven.
 - Default `AuthEvent` handler currently allows auth (`true`).
-- Admin auth checks `config.toml` token (`cgrpc_token`), with permissive behavior when unset.
+- Admin auth checks the `config.toml` token (`auth_token`, aliased from the old
+  `cgrpc_token`) in `events/admin_auth_event.rs`; **unset token = allow** (permissive).
 
 Do not assume secure defaults without verifying handlers/mods.
 
 ## Change playbooks
 
-### gRPC API change
-1. Edit `engine/proto/engine.proto`
-2. Update server implementation (`engine/src/bin/server.rs`)
-3. Update client calls (`engine/src/bin/client.rs`)
-4. Run `cargo test --workspace`
-
 ### Task lifecycle/queue change
-- Touch: `engine/src/bin/server.rs`, `enginelib/src/task.rs`, `enginelib/src/api.rs`
-- Validate DB sync + task transitions (queued/executing/solved)
+- Touch: `enginelib/src/nucleus/*`, `enginelib/src/api.rs`, `enginelib/src/task.rs`,
+  `engine/src/bin/server.rs`
+- Keep the three representations in sync: RocksDB record, dedup `DashSet`, and
+  `leased_tasks`. A task_id must be in the dedup set for exactly as long as it is
+  queued or leased — never leave it set with the record neither queued nor leased.
+- Validate transitions: submitted -> queued (`load`) -> leased -> completed, plus
+  lease cancel (requeue) and lease reaping (dedup cleared, DB kept for re-`load`).
 
 ### Event/macro change
 - Runtime dispatch: `enginelib/src/event.rs`
 - Macro generation: `enginelib/macros/src/lib.rs`
 - Core event wrappers: `enginelib/src/events/*.rs`
-- Check RFC scope first:
-  - Client hooks: `rfc/rfc1003.md`
-  - Server hooks: `rfc/rfc1004.md`
+- The RFC1003/1004 hook-scope docs no longer exist; derive scope from the
+  `events/*` wrappers and their `#[event(...)]` attributes (namespace/name/ctx).
 
 ### Mod loading issue
 - Inspect `enginelib/src/plugin.rs`
-- Cross-check `rfc/rfc1002.md`
+- The RFC1002 package-format doc no longer exists; the `.rf` ABI is defined by the
+  "Mod ABI contract" section above and enforced in `plugin.rs`.
 
 ## Quality gates before finishing
 
@@ -110,4 +147,7 @@ Report any warnings/errors that indicate risk (panic paths, serialization, auth
 - Existing code has several warnings and `unwrap()` calls in runtime paths.
 - Prefer minimal, targeted edits.
 - Preserve current external behavior unless task explicitly requests behavior change.
-- For server/client modding work, keep cancellation semantics explicit (`Status::aborted` default for cancellable lifecycle hooks unless endpoint policy says otherwise).
+- Event cancellation is in-process, not transport: `#[event(..., cancellable)]` gives
+  an event `cancel()`/`is_cancelled()`, and callers branch on the returned `.cancelled`
+  flag (see `events/*`). The old gRPC `Status::aborted` mapping no longer applies —
+  transport isn't wired yet.