diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index 1d270f4..fb9b931 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -21,8 +21,6 @@ jobs: steps: - uses: actions/checkout@v4 - run: rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }} - - name: Install Protoc - uses: arduino/setup-protoc@v3 - name: Cache Cargo dependencies uses: actions/cache@v4 with: diff --git a/AGENTS.md b/AGENTS.md index ec9fdfb..a2ab551 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,48 +7,85 @@ Operational guide for coding agents in this repository. Rust workspace for a distributed task engine with dynamic mods. Workspace members: -- `engine` (binaries: server/client/packer) -- `enginelib` (shared runtime API, events, tasks, plugin loader) +- `engine` (binary: `server`) +- `enginelib` (shared runtime API, events, tasks, plugin loader, task nucleus) - `enginelib/macros` (proc macros used by mods and core) +> **Repo is mid-rewrite.** The earlier design (gRPC/proto, `client`/`packer` +> binaries, and the RFC1002/1003/1004 design docs) was removed in the "rewrite: +> clean" pass and no longer exists in the tree. Do not cite those RFCs or the +> proto layer — they are gone. `engine/src/bin/server.rs` is currently a stub +> (`ServerAPI::init()`, RPC commented out); the transport layer is not yet wired. + ## Start here (fast orientation) 1. `Cargo.toml` (workspace) -2. `engine/src/bin/server.rs` -3. `enginelib/src/api.rs` -4. `enginelib/src/event.rs` -5. `enginelib/src/plugin.rs` -6. `engine/proto/engine.proto` -7. `rfc/rfc1003.md` (client event hooks) -8. `rfc/rfc1004.md` (server event hooks) +2. `enginelib/src/api.rs` (`ServerAPI`, task queues, RocksDB, key helpers) +3. `enginelib/src/nucleus/*` (task lifecycle: submit/lease/renew/complete/cancel/query) +4. `enginelib/src/event.rs` (event dispatch) +5. `enginelib/src/plugin.rs` (mod loader) +6. `enginelib/src/config.rs` (config/token) +7. `engine/src/bin/server.rs` (entrypoint stub) ## Build/test commands - Build all: `cargo build --workspace` - Test all: `cargo test --workspace` - Run server: `cargo run -p engine --bin server` -- Run client: `cargo run -p engine --bin client` ## Key architecture facts -- gRPC service is in `engine/src/bin/server.rs`. -- Proto definitions in `engine/proto/engine.proto`; generated at build time by `engine/build.rs`. -- Engine state is centralized in `enginelib::api::EngineAPI`. -- Persistent queues are sled-backed (`engine_db` / `engine_client_db`). +- Engine state is centralized in `enginelib::api::ServerAPI`. +- Persistence is RocksDB-backed (`ServerAPI::db: Arc`, store dir `engine_db`). +- Task lifecycle operations live in `enginelib/src/nucleus/*` (`submit`, `lease`, `renew`, `complete`, `cancel`, `query`). - Core event system is in `enginelib/src/event.rs`; core events in `enginelib/src/events/*`. -- Client modding hooks (RFC1003) are wired in `engine/src/bin/client.rs`. -- Server lifecycle hooks (RFC1004 Tier 1) are wired in `engine/src/bin/server.rs`. - Mods are loaded from `./mods/*.rf` by `enginelib/src/plugin.rs`. +- No network transport is wired yet — `server.rs` only builds `ServerAPI`. There is + no proto/gRPC layer and no `client`/`packer` binary in the current tree. ## Data/persistence model (important) -Sled keys: -- `tasks` -> `TaskQueue` -- `executing_tasks` -> `ExecutingTaskQueue` -- `solved_tasks` -> `SolvedTasks` - -`Identifier` = `(namespace, task)`. -Task instance IDs are separate string IDs. +RocksDB is the source of truth. Key layout (see helpers in `enginelib/src/api.rs`: +`task_key`, `task_key_prefix`, `finished_task_key`, `finished_task_key_prefix`): +- `t:::` -> serialized pending task record +- `f:::` -> serialized finished task record + +In-memory state on `ServerAPI` (rebuilt from the DB via `load()`), keyed by +`Identifier`: +- `task_queue: TaskQueue` — per-type `(async_channel Sender, Receiver, dedup DashSet)`. + A task_id is in the dedup set iff it is in-flight (queued in the channel **or** leased). +- `leased_tasks: LeasedTaskQueue` — per-type `Vec` handed to workers, + reaped after `LEASE_TTL_SECS`. + +`Identifier` = `(namespace, task)`. Task instance IDs are separate string IDs +(`druid` hex), generated by `submit` and **returned to the caller** so the task +can later be addressed by `cancel`/`renew`/`complete`/`query`. + +### Task lifecycle (`enginelib/src/nucleus/*`) + +- `submit(bytes, task_type) -> String` — verifies bytes, persists the pending + record, returns the generated task_id. Does not touch in-memory state; `load()` + enqueues it. +- `load(task_type)` — batch-reads pending records from the DB and enqueues any not + already in the dedup set. +- `lease(task_type, user_id) -> StoredTask` — `recv`s from the channel and records a + lease (task stays in the dedup set). +- `renew(task_type, task_id)` — resets the lease's `given_at` to now, extending its + TTL before the reaper collects it. Touches nothing else. +- `complete(task_type, task_id, bytes)` — finalize a leased task: verifies bytes, + writes the `f:` record, deletes the `t:` record, then clears the lease + dedup entry. +- `cancel(task_type, task_id)` — cancels the **lease, not the task**: drops the lease + and `try_send`s the record back into the queue (dedup entry and persisted record + untouched). If the channel is full/closed it clears the dedup entry so a later + `load()` recovers the still-persisted record. Synchronous — no await window in which + the task is neither leased nor queued. +- `query(Query) -> QueryResult` — operator/CRUD surface, all O(seek) (never a + full keyspace scan): point `Get { key }` / `Delete { key }`, range + `Scan { prefix, after, limit }` / `PurgePrefix { prefix }`, and + `EstimateKeys` / `EstimateSize`. Build keys with the `*_key` helpers. Raw + `Delete`/`PurgePrefix` bypass dedup/lease state — use `cancel`/`complete` for + in-flight tasks. There is intentionally no raw `Put`: create/update go through + `submit`/`complete` so verification and in-memory invariants hold. ## Mod ABI contract (must keep) @@ -57,7 +94,7 @@ Each `.rf` package must contain platform lib at root: Exported symbols required: - `metadata() -> LibraryMetadata` -- `run(api: &mut EngineAPI)` +- `run(api: &mut ServerAPI)` `#[metadata]` and `#[module]` macros are the preferred way to provide these. @@ -67,33 +104,33 @@ Strict compatibility check is enabled when `GE_STRICT_MODS` is set. - User auth/admin auth are event-driven. - Default `AuthEvent` handler currently allows auth (`true`). -- Admin auth checks `config.toml` token (`cgrpc_token`), with permissive behavior when unset. +- Admin auth checks the `config.toml` token (`auth_token`, aliased from the old + `cgrpc_token`) in `events/admin_auth_event.rs`; **unset token = allow** (permissive). Do not assume secure defaults without verifying handlers/mods. ## Change playbooks -### gRPC API change -1. Edit `engine/proto/engine.proto` -2. Update server implementation (`engine/src/bin/server.rs`) -3. Update client calls (`engine/src/bin/client.rs`) -4. Run `cargo test --workspace` - ### Task lifecycle/queue change -- Touch: `engine/src/bin/server.rs`, `enginelib/src/task.rs`, `enginelib/src/api.rs` -- Validate DB sync + task transitions (queued/executing/solved) +- Touch: `enginelib/src/nucleus/*`, `enginelib/src/api.rs`, `enginelib/src/task.rs`, + `engine/src/bin/server.rs` +- Keep the three representations in sync: RocksDB record, dedup `DashSet`, and + `leased_tasks`. A task_id must be in the dedup set for exactly as long as it is + queued or leased — never leave it set with the record neither queued nor leased. +- Validate transitions: submitted -> queued (`load`) -> leased -> completed, plus + lease cancel (requeue) and lease reaping (dedup cleared, DB kept for re-`load`). ### Event/macro change - Runtime dispatch: `enginelib/src/event.rs` - Macro generation: `enginelib/macros/src/lib.rs` - Core event wrappers: `enginelib/src/events/*.rs` -- Check RFC scope first: - - Client hooks: `rfc/rfc1003.md` - - Server hooks: `rfc/rfc1004.md` +- The RFC1003/1004 hook-scope docs no longer exist; derive scope from the + `events/*` wrappers and their `#[event(...)]` attributes (namespace/name/ctx). ### Mod loading issue - Inspect `enginelib/src/plugin.rs` -- Cross-check `rfc/rfc1002.md` +- The RFC1002 package-format doc no longer exists; the `.rf` ABI is defined by the + "Mod ABI contract" section above and enforced in `plugin.rs`. ## Quality gates before finishing @@ -110,4 +147,7 @@ Report any warnings/errors that indicate risk (panic paths, serialization, auth - Existing code has several warnings and `unwrap()` calls in runtime paths. - Prefer minimal, targeted edits. - Preserve current external behavior unless task explicitly requests behavior change. -- For server/client modding work, keep cancellation semantics explicit (`Status::aborted` default for cancellable lifecycle hooks unless endpoint policy says otherwise). +- Event cancellation is in-process, not transport: `#[event(..., cancellable)]` gives + an event `cancel()`/`is_cancelled()`, and callers branch on the returned `.cancelled` + flag (see `events/*`). The old gRPC `Status::aborted` mapping no longer applies — + transport isn't wired yet. diff --git a/Cargo.lock b/Cargo.lock index 287b6d6..5959e87 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -26,56 +26,6 @@ dependencies = [ "libc", ] -[[package]] -name = "anstream" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" -dependencies = [ - "anstyle", - "anstyle-parse", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - -[[package]] -name = "anstyle" -version = "1.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" - -[[package]] -name = "anstyle-parse" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ce7f38b242319f7cabaa6813055467063ecdc9d355bbb4ce0c68908cd8130e" -dependencies = [ - "utf8parse", -] - -[[package]] -name = "anstyle-query" -version = "1.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc" -dependencies = [ - "windows-sys", -] - -[[package]] -name = "anstyle-wincon" -version = "3.0.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d" -dependencies = [ - "anstyle", - "once_cell_polyfill", - "windows-sys", -] - [[package]] name = "anyhow" version = "1.0.102" @@ -92,14 +42,15 @@ dependencies = [ ] [[package]] -name = "async-trait" -version = "0.1.89" +name = "async-channel" +version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +checksum = "924ed96dd52d1b75e9c1a3e6275715fd320f5f9439fb5a4a11fa51f4221158d2" dependencies = [ - "proc-macro2", - "quote", - "syn", + "concurrent-queue", + "event-listener-strategy", + "futures-core", + "pin-project-lite", ] [[package]] @@ -111,12 +62,6 @@ dependencies = [ "critical-section", ] -[[package]] -name = "atomic-waker" -version = "1.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" - [[package]] name = "autocfg" version = "1.5.0" @@ -124,60 +69,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] -name = "axum" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" -dependencies = [ - "axum-core", - "bytes", - "futures-util", - "http", - "http-body", - "http-body-util", - "itoa", - "matchit", - "memchr", - "mime", - "percent-encoding", - "pin-project-lite", - "serde_core", - "sync_wrapper", - "tower", - "tower-layer", - "tower-service", -] - -[[package]] -name = "axum-core" -version = "0.5.6" +name = "bindgen" +version = "0.72.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1" +checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895" dependencies = [ - "bytes", - "futures-core", - "http", - "http-body", - "http-body-util", - "mime", - "pin-project-lite", - "sync_wrapper", - "tower-layer", - "tower-service", + "bitflags", + "cexpr", + "clang-sys", + "itertools", + "proc-macro2", + "quote", + "regex", + "rustc-hash", + "shlex", + "syn", ] -[[package]] -name = "base64" -version = "0.22.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" - -[[package]] -name = "bitflags" -version = "1.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" - [[package]] name = "bitflags" version = "2.11.1" @@ -222,6 +130,16 @@ version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" +[[package]] +name = "bzip2-sys" +version = "0.1.13+1.0.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "225bff33b2141874fe80d71e07d6eec4f85c5c216453dd96388240f96e1acc14" +dependencies = [ + "cc", + "pkg-config", +] + [[package]] name = "camino" version = "1.2.2" @@ -262,9 +180,20 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "43c5703da9466b66a946814e1adf53ea2c90f10063b86290cc9eb67ce3478a20" dependencies = [ "find-msvc-tools", + "jobserver", + "libc", "shlex", ] +[[package]] +name = "cexpr" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766" +dependencies = [ + "nom", +] + [[package]] name = "cfg-if" version = "1.0.4" @@ -273,9 +202,9 @@ checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" [[package]] name = "chacha20" -version = "0.10.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" dependencies = [ "cfg-if", "cpufeatures 0.3.0", @@ -297,54 +226,16 @@ dependencies = [ ] [[package]] -name = "clap" -version = "4.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b193af5b67834b676abd72466a96c1024e6a6ad978a1f484bd90b85c94041351" -dependencies = [ - "clap_builder", - "clap_derive", -] - -[[package]] -name = "clap_builder" -version = "4.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" -dependencies = [ - "anstream", - "anstyle", - "clap_lex", - "strsim", -] - -[[package]] -name = "clap_complete" -version = "4.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ff7a1dccbdd8b078c2bdebff47e404615151534d5043da397ec50286816f9cb" -dependencies = [ - "clap", -] - -[[package]] -name = "clap_derive" -version = "4.6.0" +name = "clang-sys" +version = "1.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1110bd8a634a1ab8cb04345d8d878267d57c3cf1b38d91b71af6686408bbca6a" +checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" dependencies = [ - "heck", - "proc-macro2", - "quote", - "syn", + "glob", + "libc", + "libloading 0.8.9", ] -[[package]] -name = "clap_lex" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9" - [[package]] name = "clru" version = "0.6.3" @@ -364,18 +255,12 @@ dependencies = [ ] [[package]] -name = "colorchoice" -version = "1.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" - -[[package]] -name = "colored" -version = "3.1.1" +name = "concurrent-queue" +version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34" +checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" dependencies = [ - "windows-sys", + "crossbeam-utils", ] [[package]] @@ -520,16 +405,16 @@ dependencies = [ [[package]] name = "dashmap" -version = "6.1.0" +version = "6.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" +checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c" dependencies = [ "cfg-if", "crossbeam-utils", "hashbrown 0.14.5", "lock_api", "once_cell", - "parking_lot_core 0.9.12", + "parking_lot_core", ] [[package]] @@ -606,7 +491,7 @@ dependencies = [ [[package]] name = "druid" version = "0.1.0" -source = "git+https://github.com/GrandEngineering/druid.git#dc2e5dcab0568079ffb4bad9a58e489c51c7fbad" +source = "git+https://github.com/voltaero/druid.git#dc2e5dcab0568079ffb4bad9a58e489c51c7fbad" dependencies = [ "rand", "uuid", @@ -620,9 +505,9 @@ checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" [[package]] name = "either" -version = "1.15.0" +version = "1.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" [[package]] name = "embedded-io" @@ -647,38 +532,28 @@ dependencies = [ [[package]] name = "engine" -version = "0.1.0" +version = "0.3.0" dependencies = [ - "clap", - "clap_complete", - "colored", - "druid", "enginelib", - "prost", - "serde", - "tokio", - "toml", - "tonic", - "tonic-build", - "tonic-prost", - "tonic-prost-build", - "tonic-reflection", ] [[package]] name = "enginelib" -version = "0.2.0" +version = "0.3.0" dependencies = [ + "async-channel", "chrono", "crossbeam", + "dashmap", "directories", + "druid", "inventory", - "libloading", + "libloading 0.9.0", "macros", "oxifs", "postcard", + "rust-rocksdb", "serde", - "sled", "tokio", "toml", "tracing", @@ -703,6 +578,27 @@ dependencies = [ "windows-sys", ] +[[package]] +name = "event-listener" +version = "5.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13b66accf52311f30a0db42147dadea9850cb48cd070028831ae5f5d4b856ab" +dependencies = [ + "concurrent-queue", + "parking", + "pin-project-lite", +] + +[[package]] +name = "event-listener-strategy" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" +dependencies = [ + "event-listener", + "pin-project-lite", +] + [[package]] name = "faster-hex" version = "0.10.0" @@ -736,12 +632,6 @@ version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" -[[package]] -name = "fixedbitset" -version = "0.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" - [[package]] name = "fnv" version = "1.0.7" @@ -760,63 +650,11 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" -[[package]] -name = "fs2" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9564fc758e15025b46aa6643b1b77d047d1a56a1aea6e01002ac0c7026876213" -dependencies = [ - "libc", - "winapi", -] - -[[package]] -name = "futures-channel" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" -dependencies = [ - "futures-core", -] - [[package]] name = "futures-core" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" - -[[package]] -name = "futures-sink" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" - -[[package]] -name = "futures-task" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" - -[[package]] -name = "futures-util" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" -dependencies = [ - "futures-core", - "futures-task", - "pin-project-lite", - "slab", -] - -[[package]] -name = "fxhash" -version = "0.2.1" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c31b6d751ae2c7f11320402d34e41349dd1016f8d5d45e48c4312bc8625af50c" -dependencies = [ - "byteorder", -] +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" [[package]] name = "generic-array" @@ -898,7 +736,7 @@ dependencies = [ "gix-utils", "gix-validate", "gix-worktree", - "parking_lot 0.12.5", + "parking_lot", "signal-hook 0.3.18", "smallvec", "thiserror", @@ -1005,7 +843,7 @@ version = "0.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2409cffa4fe8b303847d5b6ba8df9da9ba65d302fc5ee474ea0cac5afde79840" dependencies = [ - "bitflags 2.11.1", + "bitflags", "bstr", "gix-path", "libc", @@ -1144,7 +982,7 @@ version = "0.23.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e8546300aee4c65c5862c22a3e321124a69b654a61a8b60de546a9284812b7e2" dependencies = [ - "bitflags 2.11.1", + "bitflags", "bstr", "gix-features", "gix-path", @@ -1170,7 +1008,7 @@ checksum = "222f7428636020bef272a87ed833ea48bf5fb3193f99852ae16fbb5a602bd2f0" dependencies = [ "gix-hash", "hashbrown 0.16.1", - "parking_lot 0.12.5", + "parking_lot", ] [[package]] @@ -1192,7 +1030,7 @@ version = "0.45.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ea6d3e9e11647ba49f441dea0782494cc6d2875ff43fa4ad9094e6957f42051" dependencies = [ - "bitflags 2.11.1", + "bitflags", "bstr", "filetime", "fnv", @@ -1262,7 +1100,7 @@ dependencies = [ "gix-pack", "gix-path", "gix-quote", - "parking_lot 0.12.5", + "parking_lot", "tempfile", "thiserror", ] @@ -1315,7 +1153,7 @@ version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed9e0c881933c37a7ef45288d6c5779c4a7b3ad240b4c37657e1d9829eb90085" dependencies = [ - "bitflags 2.11.1", + "bitflags", "bstr", "gix-attributes", "gix-config-value", @@ -1396,7 +1234,7 @@ version = "0.39.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "91898c83b18c635696f7355d171cfa74a52f38022ff89581f567768935ebc4c8" dependencies = [ - "bitflags 2.11.1", + "bitflags", "bstr", "gix-commitgraph", "gix-date", @@ -1429,7 +1267,7 @@ version = "0.12.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ea9962ed6d9114f7f100efe038752f41283c225bb507a2888903ac593dffa6be" dependencies = [ - "bitflags 2.11.1", + "bitflags", "gix-path", "libc", "windows-sys", @@ -1494,7 +1332,7 @@ dependencies = [ "dashmap", "gix-fs", "libc", - "parking_lot 0.12.5", + "parking_lot", "signal-hook 0.4.4", "signal-hook-registry", "tempfile", @@ -1528,7 +1366,7 @@ version = "0.51.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d052b83d1d1744be95ac6448ac02f95f370a8f6720e466be9ce57146e39f5280" dependencies = [ - "bitflags 2.11.1", + "bitflags", "gix-commitgraph", "gix-date", "gix-hash", @@ -1593,23 +1431,10 @@ dependencies = [ ] [[package]] -name = "h2" -version = "0.4.13" +name = "glob" +version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54" -dependencies = [ - "atomic-waker", - "bytes", - "fnv", - "futures-core", - "futures-sink", - "http", - "indexmap", - "slab", - "tokio", - "tokio-util", - "tracing", -] +checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" [[package]] name = "hash32" @@ -1691,106 +1516,6 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" -[[package]] -name = "http" -version = "1.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a" -dependencies = [ - "bytes", - "itoa", -] - -[[package]] -name = "http-body" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" -dependencies = [ - "bytes", - "http", -] - -[[package]] -name = "http-body-util" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" -dependencies = [ - "bytes", - "futures-core", - "http", - "http-body", - "pin-project-lite", -] - -[[package]] -name = "httparse" -version = "1.10.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" - -[[package]] -name = "httpdate" -version = "1.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9" - -[[package]] -name = "hyper" -version = "1.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6299f016b246a94207e63da54dbe807655bf9e00044f73ded42c3ac5305fbcca" -dependencies = [ - "atomic-waker", - "bytes", - "futures-channel", - "futures-core", - "h2", - "http", - "http-body", - "httparse", - "httpdate", - "itoa", - "pin-project-lite", - "smallvec", - "tokio", - "want", -] - -[[package]] -name = "hyper-timeout" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0" -dependencies = [ - "hyper", - "hyper-util", - "pin-project-lite", - "tokio", - "tower-service", -] - -[[package]] -name = "hyper-util" -version = "0.1.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" -dependencies = [ - "bytes", - "futures-channel", - "futures-util", - "http", - "http-body", - "hyper", - "libc", - "pin-project-lite", - "socket2", - "tokio", - "tower-service", - "tracing", -] - [[package]] name = "iana-time-zone" version = "0.1.65" @@ -1848,15 +1573,6 @@ dependencies = [ "serde_core", ] -[[package]] -name = "instant" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" -dependencies = [ - "cfg-if", -] - [[package]] name = "inventory" version = "0.3.24" @@ -1866,17 +1582,11 @@ dependencies = [ "rustversion", ] -[[package]] -name = "is_terminal_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695" - [[package]] name = "itertools" -version = "0.14.0" +version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" dependencies = [ "either", ] @@ -1928,6 +1638,16 @@ dependencies = [ "jiff-tzdb", ] +[[package]] +name = "jobserver" +version = "0.1.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" +dependencies = [ + "getrandom 0.4.2", + "libc", +] + [[package]] name = "js-sys" version = "0.3.95" @@ -1967,26 +1687,47 @@ checksum = "52ff2c0fe9bc6cb6b14a0592c2ff4fa9ceb83eea9db979b0487cd054946a2b8f" [[package]] name = "libloading" -version = "0.9.0" +version = "0.8.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" dependencies = [ "cfg-if", "windows-link", ] [[package]] -name = "libredox" -version = "0.1.16" +name = "libloading" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "libredox" +version = "0.1.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e02f3bb43d335493c96bf3fd3a321600bf6bd07ed34bc64118e9293bdffea46c" dependencies = [ - "bitflags 2.11.1", + "bitflags", "libc", "plain", "redox_syscall 0.7.4", ] +[[package]] +name = "libz-sys" +version = "1.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85bc9657773828b90eeb625adff10eeac83cc21bbfd8e23a03eaa8a33c9e28d9" +dependencies = [ + "cc", + "pkg-config", + "vcpkg", +] + [[package]] name = "linux-raw-sys" version = "0.12.1" @@ -2008,6 +1749,16 @@ version = "0.4.29" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" +[[package]] +name = "lz4-sys" +version = "1.11.1+lz4-1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6bd8c0d6c6ed0cd30b3652886bb8711dc4bb01d637a68105a3d5158039b418e6" +dependencies = [ + "cc", + "libc", +] + [[package]] name = "macros" version = "0.1.0" @@ -2026,12 +1777,6 @@ dependencies = [ "regex-automata", ] -[[package]] -name = "matchit" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" - [[package]] name = "maybe-async" version = "0.2.10" @@ -2059,10 +1804,10 @@ dependencies = [ ] [[package]] -name = "mime" -version = "0.3.17" +name = "minimal-lexical" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" +checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" [[package]] name = "mio" @@ -2076,10 +1821,14 @@ dependencies = [ ] [[package]] -name = "multimap" -version = "0.10.1" +name = "nom" +version = "7.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" +checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" +dependencies = [ + "memchr", + "minimal-lexical", +] [[package]] name = "nu-ansi-term" @@ -2120,12 +1869,6 @@ version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" -[[package]] -name = "once_cell_polyfill" -version = "1.70.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" - [[package]] name = "option-ext" version = "0.2.0" @@ -2143,15 +1886,10 @@ dependencies = [ ] [[package]] -name = "parking_lot" -version = "0.11.2" +name = "parking" +version = "2.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99" -dependencies = [ - "instant", - "lock_api", - "parking_lot_core 0.8.6", -] +checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" [[package]] name = "parking_lot" @@ -2160,21 +1898,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" dependencies = [ "lock_api", - "parking_lot_core 0.9.12", -] - -[[package]] -name = "parking_lot_core" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60a2cfe6f0ad2bfc16aefa463b497d5c7a5ecd44a23efa72aa342d90177356dc" -dependencies = [ - "cfg-if", - "instant", - "libc", - "redox_syscall 0.2.16", - "smallvec", - "winapi", + "parking_lot_core", ] [[package]] @@ -2196,43 +1920,18 @@ version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" -[[package]] -name = "petgraph" -version = "0.8.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" -dependencies = [ - "fixedbitset", - "hashbrown 0.15.5", - "indexmap", -] - -[[package]] -name = "pin-project" -version = "1.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1749c7ed4bcaf4c3d0a3efc28538844fb29bcdd7d2b67b2be7e20ba861ff517" -dependencies = [ - "pin-project-internal", -] - -[[package]] -name = "pin-project-internal" -version = "1.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9b20ed30f105399776b9c883e68e536ef602a16ae6f596d2c473591d6ad64c6" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - [[package]] name = "pin-project-lite" version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + [[package]] name = "plain" version = "0.2.3" @@ -2298,80 +1997,7 @@ version = "30.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5a6efc566849d3d9d737c5cb06cc50e48950ebe3d3f9d70631490fff3a07b139" dependencies = [ - "parking_lot 0.12.5", -] - -[[package]] -name = "prost" -version = "0.14.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2ea70524a2f82d518bce41317d0fae74151505651af45faf1ffbd6fd33f0568" -dependencies = [ - "bytes", - "prost-derive", -] - -[[package]] -name = "prost-build" -version = "0.14.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "343d3bd7056eda839b03204e68deff7d1b13aba7af2b2fd16890697274262ee7" -dependencies = [ - "heck", - "itertools", - "log", - "multimap", - "petgraph", - "prettyplease", - "prost", - "prost-types", - "pulldown-cmark", - "pulldown-cmark-to-cmark", - "regex", - "syn", - "tempfile", -] - -[[package]] -name = "prost-derive" -version = "0.14.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" -dependencies = [ - "anyhow", - "itertools", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "prost-types" -version = "0.14.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8991c4cbdb8bc5b11f0b074ffe286c30e523de90fee5ba8132f1399f23cb3dd7" -dependencies = [ - "prost", -] - -[[package]] -name = "pulldown-cmark" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c3a14896dfa883796f1cb410461aef38810ea05f2b2c33c5aded3649095fdad" -dependencies = [ - "bitflags 2.11.1", - "memchr", - "unicase", -] - -[[package]] -name = "pulldown-cmark-to-cmark" -version = "22.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50793def1b900256624a709439404384204a5dc3a6ec580281bfaac35e882e90" -dependencies = [ - "pulldown-cmark", + "parking_lot", ] [[package]] @@ -2391,9 +2017,9 @@ checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] name = "rand" -version = "0.10.1" +version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" dependencies = [ "chacha20", "getrandom 0.4.2", @@ -2406,22 +2032,13 @@ version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" -[[package]] -name = "redox_syscall" -version = "0.2.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a" -dependencies = [ - "bitflags 1.3.2", -] - [[package]] name = "redox_syscall" version = "0.5.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" dependencies = [ - "bitflags 2.11.1", + "bitflags", ] [[package]] @@ -2430,7 +2047,7 @@ version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f450ad9c3b1da563fb6948a8e0fb0fb9269711c9c73d9ea1de5058c79c8d643a" dependencies = [ - "bitflags 2.11.1", + "bitflags", ] [[package]] @@ -2473,6 +2090,42 @@ version = "0.8.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" +[[package]] +name = "rust-librocksdb-sys" +version = "0.47.0+11.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "040b883ae99f5f0e40bd93cd9ae7bdc9b2e53e58cc19a16a8cb503685f79b834" +dependencies = [ + "bindgen", + "bzip2-sys", + "cc", + "glob", + "libc", + "libz-sys", + "lz4-sys", + "pkg-config", + "rustflags", + "tikv-jemalloc-sys", + "zstd-sys", +] + +[[package]] +name = "rust-rocksdb" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2a7b79a9e8015b4cd08caa31aae5208beffd4a851853df1abd30a8f35b311309" +dependencies = [ + "libc", + "parking_lot", + "rust-librocksdb-sys", +] + +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + [[package]] name = "rustc_version" version = "0.4.1" @@ -2482,13 +2135,19 @@ dependencies = [ "semver", ] +[[package]] +name = "rustflags" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a39e0e9135d7a7208ee80aa4e3e4b88f0f5ad7be92153ed70686c38a03db2e63" + [[package]] name = "rustix" version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" dependencies = [ - "bitflags 2.11.1", + "bitflags", "errno", "libc", "linux-raw-sys", @@ -2650,28 +2309,6 @@ dependencies = [ "libc", ] -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "sled" -version = "0.34.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f96b4737c2ce5987354855aed3797279def4ebf734436c6aa4552cf8e169935" -dependencies = [ - "crc32fast", - "crossbeam-epoch", - "crossbeam-utils", - "fs2", - "fxhash", - "libc", - "log", - "parking_lot 0.11.2", -] - [[package]] name = "smallvec" version = "1.15.1" @@ -2726,12 +2363,6 @@ dependencies = [ "unicode-ident", ] -[[package]] -name = "sync_wrapper" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" - [[package]] name = "tar" version = "0.4.45" @@ -2785,6 +2416,16 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "tikv-jemalloc-sys" +version = "0.6.1+5.3.0-1-ge13ca993e8ccb9ba9847cc330696e02839f328f7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd8aa5b2ab86a2cefa406d889139c162cbb230092f7d1d7cbc1716405d852a3b" +dependencies = [ + "cc", + "libc", +] + [[package]] name = "time" version = "0.3.47" @@ -2842,7 +2483,7 @@ dependencies = [ "bytes", "libc", "mio", - "parking_lot 0.12.5", + "parking_lot", "pin-project-lite", "signal-hook-registry", "socket2", @@ -2861,30 +2502,6 @@ dependencies = [ "syn", ] -[[package]] -name = "tokio-stream" -version = "0.1.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" -dependencies = [ - "futures-core", - "pin-project-lite", - "tokio", -] - -[[package]] -name = "tokio-util" -version = "0.7.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" -dependencies = [ - "bytes", - "futures-core", - "futures-sink", - "pin-project-lite", - "tokio", -] - [[package]] name = "toml" version = "1.1.2+spec-1.1.0" @@ -2924,119 +2541,6 @@ version = "1.1.1+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "756daf9b1013ebe47a8776667b466417e2d4c5679d441c26230efd9ef78692db" -[[package]] -name = "tonic" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fec7c61a0695dc1887c1b53952990f3ad2e3a31453e1f49f10e75424943a93ec" -dependencies = [ - "async-trait", - "axum", - "base64", - "bytes", - "h2", - "http", - "http-body", - "http-body-util", - "hyper", - "hyper-timeout", - "hyper-util", - "percent-encoding", - "pin-project", - "socket2", - "sync_wrapper", - "tokio", - "tokio-stream", - "tower", - "tower-layer", - "tower-service", - "tracing", -] - -[[package]] -name = "tonic-build" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1882ac3bf5ef12877d7ed57aad87e75154c11931c2ba7e6cde5e22d63522c734" -dependencies = [ - "prettyplease", - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "tonic-prost" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a55376a0bbaa4975a3f10d009ad763d8f4108f067c7c2e74f3001fb49778d309" -dependencies = [ - "bytes", - "prost", - "tonic", -] - -[[package]] -name = "tonic-prost-build" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3144df636917574672e93d0f56d7edec49f90305749c668df5101751bb8f95a" -dependencies = [ - "prettyplease", - "proc-macro2", - "prost-build", - "prost-types", - "quote", - "syn", - "tempfile", - "tonic-build", -] - -[[package]] -name = "tonic-reflection" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aaf0685a51e6d02b502ba0764002e766b7f3042aed13d9234925b6ffbfa3fca7" -dependencies = [ - "prost", - "prost-types", - "tokio", - "tokio-stream", - "tonic", - "tonic-prost", -] - -[[package]] -name = "tower" -version = "0.5.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" -dependencies = [ - "futures-core", - "futures-util", - "indexmap", - "pin-project-lite", - "slab", - "sync_wrapper", - "tokio", - "tokio-util", - "tower-layer", - "tower-service", - "tracing", -] - -[[package]] -name = "tower-layer" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" - -[[package]] -name = "tower-service" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" - [[package]] name = "tracing" version = "0.1.44" @@ -3119,24 +2623,12 @@ dependencies = [ "syn", ] -[[package]] -name = "try-lock" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" - [[package]] name = "typenum" version = "1.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" -[[package]] -name = "unicase" -version = "2.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" - [[package]] name = "unicode-bom" version = "2.0.3" @@ -3164,17 +2656,11 @@ version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" -[[package]] -name = "utf8parse" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" - [[package]] name = "uuid" -version = "1.23.0" +version = "1.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ac8b6f42ead25368cf5b098aeb3dc8a1a2c05a3eee8a9a1a68c640edbfc79d9" +checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" dependencies = [ "getrandom 0.4.2", "js-sys", @@ -3187,6 +2673,12 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + [[package]] name = "vergen" version = "9.1.0" @@ -3245,15 +2737,6 @@ dependencies = [ "winapi-util", ] -[[package]] -name = "want" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" -dependencies = [ - "try-lock", -] - [[package]] name = "wasi" version = "0.11.1+wasi-snapshot-preview1" @@ -3351,28 +2834,12 @@ version = "0.244.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" dependencies = [ - "bitflags 2.11.1", + "bitflags", "hashbrown 0.15.5", "indexmap", "semver", ] -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - [[package]] name = "winapi-util" version = "0.1.11" @@ -3382,12 +2849,6 @@ dependencies = [ "windows-sys", ] -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - [[package]] name = "windows-core" version = "0.62.2" @@ -3529,7 +2990,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" dependencies = [ "anyhow", - "bitflags 2.11.1", + "bitflags", "indexmap", "log", "serde", @@ -3580,3 +3041,13 @@ name = "zmij" version = "1.0.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/Cargo.toml b/Cargo.toml index 028582f..6e9ee15 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,4 +3,4 @@ resolver = "3" members = ["engine", "enginelib", "enginelib/macros"] [workspace.dependencies] serde = { version = "1.0.228", features = ["derive", "rc"] } -toml = "1.1.2+spec-1.1.0" +toml = "1.1.2" diff --git a/engine/Cargo.toml b/engine/Cargo.toml index 9e82df7..40b1d9e 100644 --- a/engine/Cargo.toml +++ b/engine/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "engine" -version = "0.1.0" +version = "0.3.0" edition = "2024" license-file = "LICENSE.md" description = "A Blazingly fast distributed task system" @@ -9,23 +9,5 @@ description = "A Blazingly fast distributed task system" default = [] dev = [] [dependencies] -clap = { version = "4.5.60", features = ["derive"] } -clap_complete = "4.5.66" -colored = "3.1.1" -# directories = "5.0.1" -druid = { git = "https://github.com/GrandEngineering/druid.git" } -enginelib = { path = "../enginelib" } -# libloading = "0.8.6" -prost = "0.14" -serde = { workspace = true } -# serde = "1.0.219" -tokio = { version = "1.50.0", features = ["rt-multi-thread", "macros"] } -toml = { workspace = true } -# toml = "0.8.19" -tonic = "0.14" -tonic-prost = "0.14.5" -tonic-reflection = "0.14" -[build-dependencies] -tonic-build = "0.14" -tonic-prost-build = "0.14" +enginelib = { path = "../enginelib" } diff --git a/engine/build.rs b/engine/build.rs deleted file mode 100644 index 85f99ac..0000000 --- a/engine/build.rs +++ /dev/null @@ -1,11 +0,0 @@ -use std::error::Error; -use std::{env, path::PathBuf}; -fn main() -> Result<(), Box> { - let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); - - tonic_prost_build::configure() - .file_descriptor_set_path(out_dir.join("engine_descriptor.bin")) - .compile_protos(&["proto/engine.proto"], &["proto"])?; - - Ok(()) -} diff --git a/engine/proto/engine.proto b/engine/proto/engine.proto deleted file mode 100644 index 14a429d..0000000 --- a/engine/proto/engine.proto +++ /dev/null @@ -1,71 +0,0 @@ -syntax = "proto3"; -package engine; -service Engine { - rpc AquireTask(TaskRequest) returns (Task); - rpc AquireTaskReg(empty) returns (TaskRegistry); - rpc PublishTask(Task) returns (empty); - rpc cgrpc(cgrpcmsg) returns (cgrpcmsg); - rpc CreateTask(Task) returns (Task); - rpc DeleteTask(TaskSelector) returns (empty); - rpc GetTasks(TaskPageRequest) returns (TaskPage); - rpc CheckAuth(empty) returns (empty); - rpc GetMetadata(empty) returns (ServerMetadata); -} -message ServerMetadata { - string engine_api = 1; - repeated ModuleInfo mods = 2; -} -message ModuleInfo { - string mod_id = 1; - string api_version = 2; - string rustc_version = 3; - string mod_version = 4; -} -message TaskSelector { - TaskState state = 1; - string namespace = 2; - string task = 3; - string id = 4; -} -message empty {} -enum TaskState { - QUEUED = 0; - PROCESSING = 1; - SOLVED = 2; -} -message TaskPageRequest { - string namespace = 1; - string task = 2; - uint64 page = 3; - uint32 pageSize = 4; - TaskState state = 5; -} -message TaskPage { - string namespace = 1; - string task = 2; - uint64 page = 3; - uint32 pageSize = 4; - TaskState state = 5; - repeated Task tasks = 6; -} -message cgrpcmsg { - string handler_mod_id = 1; - string handler_id = 2; - bytes event_payload = 3; - string token = 4; -} - -message TaskRegistry { - repeated string tasks = 1; // namespace:task -} - -message TaskRequest { - string task_id = 1; // namespace:task - // bytes payload = 2; -} -message Task { - string id = 4; // the task unique identifier - bytes task_payload = 1; - string task_id = 2; // namespace:task - bytes payload = 3; -} diff --git a/engine/src/bin/client.rs b/engine/src/bin/client.rs deleted file mode 100644 index 546d42e..0000000 --- a/engine/src/bin/client.rs +++ /dev/null @@ -1,244 +0,0 @@ -use enginelib::{ - Registry, - api::EngineAPI, - events::Events, - event::info, - plugin::LibraryInstance, - prelude::debug, -}; -use proto::engine_client; -use std::{collections::HashMap, error::Error, sync::Arc}; -use tonic::{ - Request, - metadata::{MetadataKey, MetadataValue}, - transport::Endpoint, -}; - -pub mod proto { - tonic::include_proto!("engine"); -} - -#[tokio::main] -async fn main() -> Result<(), Box> { - let mut api = EngineAPI::default_client(); - EngineAPI::init_client(&mut api); - Events::ClientStart(&api); - - compute_module(Arc::new(api)).await; - Ok(()) -} - -fn make_interceptor( - api_for_interceptor: Arc, -) -> impl FnMut(Request<()>) -> Result, tonic::Status> + Clone { - move |mut req: Request<()>| { - let headers = Arc::new(std::sync::RwLock::new(HashMap::::new())); - Events::ClientAuthPrepare(api_for_interceptor.as_ref(), headers.clone()); - - if let Ok(headers) = headers.read() { - for (key, value) in headers.iter() { - if let (Ok(key), Ok(value)) = ( - MetadataKey::from_bytes(key.as_bytes()), - MetadataValue::try_from(value.as_str()), - ) { - req.metadata_mut().insert(key, value); - } - } - } - - Ok(req) - } -} - -async fn worker_loop( - worker_id: usize, - api: Arc, - channel: tonic::transport::Channel, - task_ids: Arc>, -) { - let interceptor = make_interceptor(api.clone()); - let mut client = engine_client::EngineClient::with_interceptor(channel, interceptor); - - loop { - let mut got_any = false; - - for task_id in task_ids.iter() { - if Events::BeforeTaskAcquire(api.as_ref(), task_id.clone()) { - continue; - } - - let task_resp = client - .aquire_task(Request::new(proto::TaskRequest { - task_id: task_id.clone(), - })) - .await; - - let mut task_req = match task_resp { - Ok(resp) => { - got_any = true; - resp - } - Err(status) if status.code() == tonic::Code::NotFound => continue, - Err(status) if status.code() == tonic::Code::PermissionDenied => { - debug!( - "worker {}: auth failed during acquire for {}", - worker_id, task_id - ); - tokio::time::sleep(std::time::Duration::from_millis(200)).await; - continue; - } - Err(status) => { - debug!("worker {}: acquire failed for {}: {:?}", worker_id, task_id, status); - continue; - } - }; - - let task_payload = task_req.get_mut(); - - let acquired_payload = Arc::new(std::sync::RwLock::new(task_payload.task_payload.clone())); - Events::TaskAcquired( - api.as_ref(), - task_id.clone(), - task_payload.id.clone(), - acquired_payload.clone(), - ); - if let Ok(payload) = acquired_payload.read() { - task_payload.task_payload = payload.clone(); - } - - let exec_payload = Arc::new(std::sync::RwLock::new(task_payload.task_payload.clone())); - if Events::BeforeTaskExecute( - api.as_ref(), - task_id.clone(), - task_payload.id.clone(), - exec_payload.clone(), - ) { - continue; - } - if let Ok(payload) = exec_payload.read() { - task_payload.task_payload = payload.clone(); - } - - let identifier = match task_id.split_once(':') { - Some(v) => v, - None => continue, - }; - let task = match api - .task_registry - .get(&(identifier.0.to_string(), identifier.1.to_string())) - { - Some(t) => t, - None => continue, - }; - let mut task = task.from_bytes(&task_payload.task_payload); - - task.run_hip(); - - let mut solv_task = proto::Task { - id: task_payload.id.clone(), - payload: Vec::new(), - task_id: task_id.clone(), - task_payload: task.to_bytes(), - }; - - let publish_payload = Arc::new(std::sync::RwLock::new(solv_task.task_payload.clone())); - if Events::BeforeTaskPublish( - api.as_ref(), - task_id.clone(), - task_payload.id.clone(), - publish_payload.clone(), - ) { - continue; - } - if let Ok(payload) = publish_payload.read() { - solv_task.task_payload = payload.clone(); - } - - if let Err(status) = client.publish_task(Request::new(solv_task)).await { - debug!( - "worker {}: publish failed for {}: {:?}", - worker_id, task_id, status - ); - } - } - - if !got_any { - tokio::time::sleep(std::time::Duration::from_millis(2)).await; - } - } -} - -// Compute Module -// Verifies server and also is -// Responsible for getting task, executing and publishing it. -async fn compute_module(api: Arc) { - let url = "http://[::1]:50051"; - let endpoint = Endpoint::from_static(url) - .tcp_nodelay(true) - .http2_adaptive_window(true) - .keep_alive_while_idle(true) - .tcp_keepalive(Some(std::time::Duration::from_secs(30))); - - let channel = endpoint.connect().await.unwrap(); - let interceptor = make_interceptor(api.clone()); - let mut client = engine_client::EngineClient::with_interceptor(channel.clone(), interceptor); - - // Get server metadata - let server_meta = client - .get_metadata(Request::new(proto::Empty {})) - .await - .unwrap() - .into_inner(); - - // validate server - assert!(server_meta.engine_api == enginelib::GIT_VERSION); - for x in &server_meta.mods { - assert!(x.api_version == enginelib::GIT_VERSION); - #[cfg(not(debug_assertions))] - assert!(x.rustc_version == enginelib::RUSTC_VERSION); - assert!(api.lib_manager.libraries.contains_key(&x.mod_id)); - let module: &LibraryInstance = api - .lib_manager - .libraries - .get(&x.mod_id) - .expect("Client Missing Mod"); - assert!(module.metadata.mod_version == x.mod_version) - } - - let task_reg = client - .aquire_task_reg(Request::new(proto::Empty {})) - .await - .unwrap() - .into_inner(); - - let task_ids = Arc::new(task_reg.tasks); - let worker_count = std::env::var("GE_CLIENT_WORKERS") - .ok() - .and_then(|v| v.parse::().ok()) - .filter(|v| *v > 0) - .unwrap_or_else(|| { - std::thread::available_parallelism() - .map(|n| n.get()) - .unwrap_or(4) - }); - - info!( - "Starting {} client workers for {} task types", - worker_count, - task_ids.len() - ); - - let mut handles = Vec::with_capacity(worker_count); - for worker_id in 0..worker_count { - let api_i = api.clone(); - let channel_i = channel.clone(); - let task_ids_i = task_ids.clone(); - handles.push(tokio::spawn(async move { - worker_loop(worker_id, api_i, channel_i, task_ids_i).await; - })); - } - - for handle in handles { - let _ = handle.await; - } -} diff --git a/engine/src/bin/packer.rs b/engine/src/bin/packer.rs deleted file mode 100644 index 46cf05c..0000000 --- a/engine/src/bin/packer.rs +++ /dev/null @@ -1,1057 +0,0 @@ -use clap::{Args, CommandFactory, Subcommand, ValueEnum, ValueHint}; -use clap::{Command, Parser}; -use clap_complete::{Generator, Shell, generate}; -use colored::*; -use enginelib::events::{Events, ID}; -// For coloring the output -use enginelib::Registry; -use enginelib::api::postcard; -use enginelib::prelude::error; -use enginelib::task::{StoredTask, Task, TaskQueue}; -use enginelib::{api::EngineAPI, config::Config, event::info}; -use serde::Deserialize; -use std::collections::BTreeMap; -use std::collections::HashMap; -use std::ffi::OsString; -use std::fs::File; -use std::io::Write; -use std::io::{self, BufReader, ErrorKind, Read}; -use std::path::PathBuf; -use toml::Value; -use tonic::{ - Request, - metadata::{MetadataKey, MetadataValue}, - transport::Endpoint, -}; - -pub mod proto { - tonic::include_proto!("engine"); -} - -#[derive(Debug)] -struct Entry { - namespace: String, - id: String, - data: BTreeMap, -} - -#[derive(Debug, Deserialize)] -#[serde(transparent)] -struct RawDoc( - std::collections::BTreeMap>>, -); - -fn parse_entries(raw: RawDoc) -> Vec { - let mut result = Vec::new(); - - for (compound_key, records) in raw.0 { - // split on colon: "widget:button" -> ("widget", "button") - let mut parts = compound_key.splitn(2, ':'); - let namespace = parts.next().unwrap_or("").to_string(); - let id = parts.next().unwrap_or("").to_string(); - - for data in records { - result.push(Entry { - namespace: namespace.clone(), - id: id.clone(), - data, - }); - } - } - - result -} - -/// A simple CLI application -#[derive(Parser, Debug)] -#[command(name = "packer")] -#[command(version = "1.0")] -#[command(author = "GrandEngineering")] -#[command(about = "A simple CLI app to pack tasks")] -struct Cli { - #[command(subcommand)] - command: Option, - #[arg(long = "generate", value_enum)] - generator: Option, -} - -#[derive(Subcommand, Debug, PartialEq)] -enum Commands { - #[command()] - Pack(PackArgs), - #[command()] - Unpack(PackArgs), - #[command()] - Upload(PackArgs), - #[command()] - AdminCheck, - #[command()] - AdminList(ListArgs), - #[command()] - AdminExport(ExportArgs), - #[command()] - AdminDelete(DeleteArgs), - #[command()] - Schema, -} - -#[derive(Args, Debug, PartialEq)] -struct PackArgs { - #[arg(short, required = true, value_hint = ValueHint::FilePath)] - input: PathBuf, - #[arg(long)] - stream: bool, -} - -#[derive(Clone, Debug, ValueEnum, PartialEq)] -enum StateArg { - Queued, - Processing, - Solved, -} - -impl StateArg { - fn to_proto(&self) -> i32 { - match self { - StateArg::Queued => proto::TaskState::Queued as i32, - StateArg::Processing => proto::TaskState::Processing as i32, - StateArg::Solved => proto::TaskState::Solved as i32, - } - } - - fn as_str(&self) -> &'static str { - match self { - StateArg::Queued => "queued", - StateArg::Processing => "processing", - StateArg::Solved => "solved", - } - } -} - -#[derive(Args, Debug, PartialEq)] -struct ListArgs { - #[arg(long)] - task_id: Option, // namespace:task - #[arg(long, value_enum, default_value = "queued")] - state: StateArg, - #[arg(long)] - all_states: bool, - #[arg(long, default_value_t = 1000)] - page_size: u32, -} - -#[derive(Args, Debug, PartialEq)] -struct ExportArgs { - #[arg(short = 'o', long, value_hint = ValueHint::FilePath, default_value = "output.rustforge.bin")] - output: PathBuf, - #[arg(long)] - task_id: Option, // namespace:task - #[arg(long, value_enum, default_value = "queued")] - state: StateArg, - #[arg(long)] - all_states: bool, - #[arg(long, default_value_t = 1000)] - page_size: u32, -} - -#[derive(Args, Debug, PartialEq)] -struct DeleteArgs { - #[arg(long)] - namespace: String, - #[arg(long)] - task: String, - #[arg(long)] - id: String, - #[arg(long, value_enum)] - state: StateArg, -} -fn print_completions(generator: G, cmd: &mut Command) { - generate( - generator, - cmd, - cmd.get_name().to_string(), - &mut io::stdout(), - ); -} - -fn build_headers(api: &EngineAPI, admin: bool) -> HashMap { - let headers = std::sync::Arc::new(std::sync::RwLock::new(HashMap::::new())); - Events::ClientAuthPrepare(api, headers.clone()); - let mut prepared_headers = headers.read().map(|h| h.clone()).unwrap_or_default(); - - if admin { - if let Some(token) = api.cfg.config_toml.cgrpc_token.clone() { - prepared_headers.insert("authorization".to_string(), token); - } - } - - prepared_headers -} - -const CHUNKED_MAGIC: [u8; 4] = *b"RFPK"; -const CHUNKED_VERSION: u8 = 2; -const FRAME_TASK: u8 = 1; -const FRAME_END: u8 = 255; - -#[derive(Debug)] -struct ChunkedTaskRecord { - namespace: String, - task: String, - id: String, - payload: Vec, -} - -fn write_chunked_header(writer: &mut W) -> io::Result<()> { - writer.write_all(&CHUNKED_MAGIC)?; - writer.write_all(&[CHUNKED_VERSION])?; - Ok(()) -} - -fn write_chunked_task_record( - writer: &mut W, - record: &ChunkedTaskRecord, -) -> io::Result<()> { - let ns = record.namespace.as_bytes(); - let task = record.task.as_bytes(); - let id = record.id.as_bytes(); - - let ns_len: u16 = ns - .len() - .try_into() - .map_err(|_| io::Error::new(ErrorKind::InvalidInput, "namespace too large"))?; - let task_len: u16 = task - .len() - .try_into() - .map_err(|_| io::Error::new(ErrorKind::InvalidInput, "task too large"))?; - let id_len: u32 = id - .len() - .try_into() - .map_err(|_| io::Error::new(ErrorKind::InvalidInput, "id too large"))?; - let payload_len: u64 = record - .payload - .len() - .try_into() - .map_err(|_| io::Error::new(ErrorKind::InvalidInput, "payload too large"))?; - - let mut meta = [0u8; 1 + 2 + 2 + 4 + 8]; - meta[0] = FRAME_TASK; - meta[1..3].copy_from_slice(&ns_len.to_le_bytes()); - meta[3..5].copy_from_slice(&task_len.to_le_bytes()); - meta[5..9].copy_from_slice(&id_len.to_le_bytes()); - meta[9..17].copy_from_slice(&payload_len.to_le_bytes()); - - writer.write_all(&meta)?; - writer.write_all(ns)?; - writer.write_all(task)?; - writer.write_all(id)?; - writer.write_all(&record.payload)?; - - Ok(()) -} - -fn write_chunked_end(writer: &mut W) -> io::Result<()> { - writer.write_all(&[FRAME_END])?; - Ok(()) -} - -fn read_exact_or_eof(reader: &mut R, buf: &mut [u8]) -> io::Result { - match reader.read_exact(buf) { - Ok(()) => Ok(true), - Err(e) if e.kind() == ErrorKind::UnexpectedEof => Ok(false), - Err(e) => Err(e), - } -} - -fn read_chunked_header(reader: &mut R) -> io::Result<()> { - let mut magic = [0u8; 4]; - reader.read_exact(&mut magic)?; - if magic != CHUNKED_MAGIC { - return Err(io::Error::new( - ErrorKind::InvalidData, - "invalid chunked magic", - )); - } - - let mut version = [0u8; 1]; - reader.read_exact(&mut version)?; - if version[0] != CHUNKED_VERSION { - return Err(io::Error::new( - ErrorKind::InvalidData, - format!("unsupported chunked version {}", version[0]), - )); - } - - Ok(()) -} - -fn read_chunked_next(reader: &mut R) -> io::Result> { - let mut frame = [0u8; 1]; - if !read_exact_or_eof(reader, &mut frame)? { - return Ok(None); - } - - match frame[0] { - FRAME_END => Ok(None), - FRAME_TASK => { - let mut ns_len = [0u8; 2]; - let mut task_len = [0u8; 2]; - let mut id_len = [0u8; 4]; - let mut payload_len = [0u8; 8]; - reader.read_exact(&mut ns_len)?; - reader.read_exact(&mut task_len)?; - reader.read_exact(&mut id_len)?; - reader.read_exact(&mut payload_len)?; - - let ns_len = u16::from_le_bytes(ns_len) as usize; - let task_len = u16::from_le_bytes(task_len) as usize; - let id_len = u32::from_le_bytes(id_len) as usize; - let payload_len_u64 = u64::from_le_bytes(payload_len); - if payload_len_u64 > usize::MAX as u64 { - return Err(io::Error::new( - ErrorKind::InvalidData, - "payload length too large for this platform", - )); - } - let payload_len = payload_len_u64 as usize; - - let mut ns = vec![0u8; ns_len]; - let mut task = vec![0u8; task_len]; - let mut id = vec![0u8; id_len]; - let mut payload = vec![0u8; payload_len]; - - reader.read_exact(&mut ns)?; - reader.read_exact(&mut task)?; - reader.read_exact(&mut id)?; - reader.read_exact(&mut payload)?; - - let namespace = String::from_utf8(ns) - .map_err(|_| io::Error::new(ErrorKind::InvalidData, "invalid namespace utf8"))?; - let task = String::from_utf8(task) - .map_err(|_| io::Error::new(ErrorKind::InvalidData, "invalid task utf8"))?; - let id = String::from_utf8(id) - .map_err(|_| io::Error::new(ErrorKind::InvalidData, "invalid id utf8"))?; - - Ok(Some(ChunkedTaskRecord { - namespace, - task, - id, - payload, - })) - } - other => Err(io::Error::new( - ErrorKind::InvalidData, - format!("unknown frame type {}", other), - )), - } -} - -#[tokio::main] -async fn main() { - let cli = Cli::parse(); - if let Some(generator) = cli.generator { - let mut cmd = Cli::command(); - eprintln!("Generating completion file for {generator:?}..."); - print_completions(generator, &mut cmd); - } - let mut api = EngineAPI::default_client(); - EngineAPI::init_client(&mut api); - // packer intentionally uses client init path; load config explicitly for host/admin token - api.cfg = Config::new(); - for (id, tsk) in api.task_registry.tasks.iter() { - api.task_queue.tasks.entry(id.clone()).or_default(); - } - if let Some(command) = cli.command { - match command { - Commands::Schema => { - let mut buf: Vec = Vec::new(); - for tsk in api.task_registry.tasks { - let unw = tsk.1.to_toml(); - buf.push(format![r#"[["{}:{}"]]"#, tsk.0.0, tsk.0.1]); - buf.push(unw); - } - let ns = buf.join("\n"); - match File::create("schema.rustforge.toml") { - Ok(mut file) => { - if let Err(e) = file.write_all(ns.as_bytes()) { - error!("Failed to write schema file: {}", e); - } else { - info!("Wrote schema.rustforge.toml"); - } - } - Err(e) => { - error!("Failed to create schema file: {}", e); - } - } - } - Commands::Unpack(input) => { - if input.input.exists() { - info!("Unpacking File: {}", input.input.to_string_lossy()); - let mut buf = Vec::new(); - - // Attempt to open and read the input file. If either step fails, - // we do not proceed to deserialization or writing the output file. - match File::open(&input.input) { - Ok(mut f) => { - if let Err(e) = f.read_to_end(&mut buf) { - error!( - "Failed to read input file {}: {}", - input.input.display(), - e - ); - // reading failed -> do not proceed to deserialize or write - return; - } - } - Err(e) => { - error!("Failed to open input file {}: {}", input.input.display(), e); - // opening failed -> do not proceed to deserialize or write - return; - } - } - - // Try to deserialize. Only on successful deserialization do we - // process entries and write the output TOML file. - let maybe_queue: Option = - match postcard::from_bytes::(&buf) { - Ok(k) => Some(k), - Err(e) => { - error!("Failed to deserialize task queue: {}", e); - None - } - }; - - if let Some(k) = maybe_queue { - let mut final_out: Vec = Vec::new(); - - for tasks in k.tasks { - match api.task_registry.tasks.get(&tasks.0.clone()) { - Some(tt) => { - for task in tasks.1 { - if tt.verify(task.bytes.clone()) { - let tmp_nt = tt.from_bytes(&task.bytes); - final_out.push(format![ - r#"[["{}:{}"]]"#, - tasks.0.0.clone(), - tasks.0.1.clone() - ]); - final_out.push(tmp_nt.to_toml()); - info!("{:?}", tmp_nt); - } - } - } - None => { - error!("Unknown template for {}:{}", tasks.0.0, tasks.0.1); - } - } - } - - let ns = final_out.join("\n"); - match File::create("output.rustforge.toml") { - Ok(mut file) => { - if let Err(e) = file.write_all(ns.as_bytes()) { - error!("Failed to write output.rustforge.toml: {}", e); - } else { - info!("Wrote output.rustforge.toml"); - } - } - Err(e) => { - error!("Failed to create output.rustforge.toml: {}", e); - } - } - } else { - // Deserialization failed; we logged the error above and intentionally do not - // create/write the output file to avoid producing an empty output. - } - } - } - Commands::Upload(input) => { - if !input.input.exists() { - error!("File does not exist: {}", input.input.to_string_lossy()); - return; - } - - info!( - "Uploading File: {}{}", - input.input.to_string_lossy(), - if input.stream { " (stream mode)" } else { "" } - ); - - let prepared_headers = build_headers(&api, false); - - let interceptor = move |mut req: Request<()>| { - for (key, value) in prepared_headers.iter() { - if let (Ok(key), Ok(value)) = ( - MetadataKey::from_bytes(key.as_bytes()), - MetadataValue::try_from(value.as_str()), - ) { - req.metadata_mut().insert(key, value); - } - } - Ok(req) - }; - - let endpoint = format!("http://{}", api.cfg.config_toml.host); - let channel = match Endpoint::from_shared(endpoint.clone()) { - Ok(ep) => match ep.connect().await { - Ok(ch) => ch, - Err(e) => { - error!("Failed to connect to server {}: {}", endpoint, e); - return; - } - }, - Err(e) => { - error!("Invalid server endpoint {}: {}", endpoint, e); - return; - } - }; - - let mut client = - proto::engine_client::EngineClient::with_interceptor(channel, interceptor); - - let mut uploaded = 0usize; - let mut failed = 0usize; - - if input.stream { - let file = match File::open(&input.input) { - Ok(f) => f, - Err(e) => { - error!("Failed to open input file {}: {}", input.input.display(), e); - return; - } - }; - let mut reader = BufReader::new(file); - if let Err(e) = read_chunked_header(&mut reader) { - error!("Failed to read chunked stream header: {}", e); - return; - } - - loop { - let record = match read_chunked_next(&mut reader) { - Ok(v) => v, - Err(e) => { - error!("Failed to read chunked stream record: {}", e); - return; - } - }; - - let Some(record) = record else { - break; - }; - - let task_id = format!("{}:{}", record.namespace, record.task); - let req = proto::Task { - id: record.id, - task_id: task_id.clone(), - task_payload: record.payload, - payload: Vec::new(), - }; - - match client.create_task(Request::new(req)).await { - Ok(_) => uploaded += 1, - Err(e) => { - failed += 1; - error!("Failed to upload task {}: {}", task_id, e); - } - } - } - } else { - let mut buf = Vec::new(); - match File::open(&input.input) { - Ok(mut f) => { - if let Err(e) = f.read_to_end(&mut buf) { - error!( - "Failed to read input file {}: {}", - input.input.display(), - e - ); - return; - } - } - Err(e) => { - error!("Failed to open input file {}: {}", input.input.display(), e); - return; - } - } - - let queue: TaskQueue = match postcard::from_bytes::(&buf) { - Ok(q) => q, - Err(e) => { - error!("Failed to deserialize task queue: {}", e); - return; - } - }; - - for ((namespace, task), tasks) in queue.tasks { - let task_id = format!("{}:{}", namespace, task); - for stored in tasks { - let req = proto::Task { - id: stored.id, - task_id: task_id.clone(), - task_payload: stored.bytes, - payload: Vec::new(), - }; - match client.create_task(Request::new(req)).await { - Ok(_) => uploaded += 1, - Err(e) => { - failed += 1; - error!("Failed to upload task {}: {}", task_id, e); - } - } - } - } - } - - info!( - "Upload complete. uploaded={}, failed={}, mode={}", - uploaded, - failed, - if input.stream { "stream" } else { "legacy" } - ); - } - Commands::AdminCheck => { - let prepared_headers = build_headers(&api, true); - let interceptor = move |mut req: Request<()>| { - for (key, value) in prepared_headers.iter() { - if let (Ok(key), Ok(value)) = ( - MetadataKey::from_bytes(key.as_bytes()), - MetadataValue::try_from(value.as_str()), - ) { - req.metadata_mut().insert(key, value); - } - } - Ok(req) - }; - - let endpoint = format!("http://{}", api.cfg.config_toml.host); - let channel = match Endpoint::from_shared(endpoint.clone()) { - Ok(ep) => match ep.connect().await { - Ok(ch) => ch, - Err(e) => { - error!("Failed to connect to server {}: {}", endpoint, e); - return; - } - }, - Err(e) => { - error!("Invalid server endpoint {}: {}", endpoint, e); - return; - } - }; - - let mut client = - proto::engine_client::EngineClient::with_interceptor(channel, interceptor); - match client.check_auth(Request::new(proto::Empty {})).await { - Ok(_) => info!("Admin auth check: OK"), - Err(e) => error!("Admin auth check failed: {}", e), - } - } - Commands::AdminList(args) => { - let prepared_headers = build_headers(&api, true); - let interceptor = move |mut req: Request<()>| { - for (key, value) in prepared_headers.iter() { - if let (Ok(key), Ok(value)) = ( - MetadataKey::from_bytes(key.as_bytes()), - MetadataValue::try_from(value.as_str()), - ) { - req.metadata_mut().insert(key, value); - } - } - Ok(req) - }; - - let endpoint = format!("http://{}", api.cfg.config_toml.host); - let channel = match Endpoint::from_shared(endpoint.clone()) { - Ok(ep) => match ep.connect().await { - Ok(ch) => ch, - Err(e) => { - error!("Failed to connect to server {}: {}", endpoint, e); - return; - } - }, - Err(e) => { - error!("Invalid server endpoint {}: {}", endpoint, e); - return; - } - }; - - let mut client = - proto::engine_client::EngineClient::with_interceptor(channel, interceptor); - - let task_ids: Vec = if let Some(task_id) = args.task_id.clone() { - vec![task_id] - } else { - match client.aquire_task_reg(Request::new(proto::Empty {})).await { - Ok(res) => res.into_inner().tasks, - Err(e) => { - error!("Failed to fetch task registry: {}", e); - return; - } - } - }; - - let states: Vec = if args.all_states { - vec![StateArg::Queued, StateArg::Processing, StateArg::Solved] - } else { - vec![args.state.clone()] - }; - - let mut listed = 0usize; - println!("state\ttask\tid"); - for task_id in task_ids { - let Some((namespace, task)) = task_id.split_once(':') else { - error!("Invalid task id '{}' (expected namespace:task)", task_id); - continue; - }; - - for state in &states { - let mut page = 0u64; - loop { - let req = proto::TaskPageRequest { - namespace: namespace.to_string(), - task: task.to_string(), - page, - page_size: args.page_size, - state: state.to_proto(), - }; - - let resp = match client.get_tasks(Request::new(req)).await { - Ok(r) => r.into_inner(), - Err(e) => { - error!( - "GetTasks failed for {} state {:?}: {}", - task_id, state, e - ); - break; - } - }; - - if resp.tasks.is_empty() { - break; - } - - for t in resp.tasks { - println!("{}\t{}:{}\t{}", state.as_str(), namespace, task, t.id); - listed += 1; - } - - page += 1; - } - } - } - - info!("Listed {} task(s)", listed); - } - Commands::AdminExport(args) => { - let prepared_headers = build_headers(&api, true); - let interceptor = move |mut req: Request<()>| { - for (key, value) in prepared_headers.iter() { - if let (Ok(key), Ok(value)) = ( - MetadataKey::from_bytes(key.as_bytes()), - MetadataValue::try_from(value.as_str()), - ) { - req.metadata_mut().insert(key, value); - } - } - Ok(req) - }; - - let endpoint = format!("http://{}", api.cfg.config_toml.host); - let channel = match Endpoint::from_shared(endpoint.clone()) { - Ok(ep) => match ep.connect().await { - Ok(ch) => ch, - Err(e) => { - error!("Failed to connect to server {}: {}", endpoint, e); - return; - } - }, - Err(e) => { - error!("Invalid server endpoint {}: {}", endpoint, e); - return; - } - }; - - let mut client = - proto::engine_client::EngineClient::with_interceptor(channel, interceptor); - - let task_ids: Vec = if let Some(task_id) = args.task_id.clone() { - vec![task_id] - } else { - match client.aquire_task_reg(Request::new(proto::Empty {})).await { - Ok(res) => res.into_inner().tasks, - Err(e) => { - error!("Failed to fetch task registry: {}", e); - return; - } - } - }; - - let states: Vec = if args.all_states { - vec![StateArg::Queued, StateArg::Processing, StateArg::Solved] - } else { - vec![args.state.clone()] - }; - - let mut out_queue = TaskQueue::default(); - let mut fetched = 0usize; - - for task_id in task_ids { - let Some((namespace, task)) = task_id.split_once(':') else { - error!("Invalid task id '{}' (expected namespace:task)", task_id); - continue; - }; - - for state in &states { - let mut page = 0u64; - loop { - let req = proto::TaskPageRequest { - namespace: namespace.to_string(), - task: task.to_string(), - page, - page_size: args.page_size, - state: state.to_proto(), - }; - - let resp = match client.get_tasks(Request::new(req)).await { - Ok(r) => r.into_inner(), - Err(e) => { - error!( - "GetTasks failed for {} state {:?}: {}", - task_id, state, e - ); - break; - } - }; - - if resp.tasks.is_empty() { - break; - } - - let key = ID(namespace, task); - let bucket = out_queue.tasks.entry(key).or_default(); - for t in resp.tasks { - bucket.push(StoredTask { - bytes: t.task_payload, - id: t.id, - }); - fetched += 1; - } - - page += 1; - } - } - } - - match postcard::to_allocvec(&out_queue) { - Ok(data) => match File::create(&args.output) { - Ok(mut file) => { - if let Err(e) = file.write_all(&data) { - error!("Failed to write {}: {}", args.output.display(), e); - } else { - info!( - "Export complete. wrote {} task(s) to {}", - fetched, - args.output.display() - ); - } - } - Err(e) => { - error!("Failed to create {}: {}", args.output.display(), e); - } - }, - Err(e) => error!("Failed to serialize export: {}", e), - } - } - Commands::AdminDelete(args) => { - let prepared_headers = build_headers(&api, true); - let interceptor = move |mut req: Request<()>| { - for (key, value) in prepared_headers.iter() { - if let (Ok(key), Ok(value)) = ( - MetadataKey::from_bytes(key.as_bytes()), - MetadataValue::try_from(value.as_str()), - ) { - req.metadata_mut().insert(key, value); - } - } - Ok(req) - }; - - let endpoint = format!("http://{}", api.cfg.config_toml.host); - let channel = match Endpoint::from_shared(endpoint.clone()) { - Ok(ep) => match ep.connect().await { - Ok(ch) => ch, - Err(e) => { - error!("Failed to connect to server {}: {}", endpoint, e); - return; - } - }, - Err(e) => { - error!("Invalid server endpoint {}: {}", endpoint, e); - return; - } - }; - - let mut client = - proto::engine_client::EngineClient::with_interceptor(channel, interceptor); - - let req = proto::TaskSelector { - state: args.state.to_proto(), - namespace: args.namespace.clone(), - task: args.task.clone(), - id: args.id.clone(), - }; - - match client.delete_task(Request::new(req)).await { - Ok(_) => info!( - "Deleted task {} from {}:{} ({:?})", - args.id, args.namespace, args.task, args.state - ), - Err(e) => error!("DeleteTask failed: {}", e), - } - } - Commands::Pack(input) => { - if input.input.exists() { - info!("Packing File: {}", input.input.to_string_lossy()); - match std::fs::read_to_string(&input.input) { - Ok(toml_str) => { - match toml::from_str::(&toml_str) { - Ok(raw) => { - let entries = parse_entries(raw); - for entry in entries { - match api - .task_registry - .get(&ID(entry.namespace.as_str(), entry.id.as_str())) - { - Some(template) => { - match toml::to_string(&entry.data) { - Ok(toml_string) => { - let t = template.from_toml(toml_string); - let key = ID( - entry.namespace.as_str(), - entry.id.as_str(), - ); - let mut vec = api - .task_queue - .tasks - .get(&key) - .cloned() - .unwrap_or_default(); - vec.push(StoredTask { - id: "".into(), //ids are minted on the server - bytes: t.to_bytes(), - }); - api.task_queue.tasks.insert(key, vec); - } - Err(e) => { - error!( - "Failed to convert entry data to TOML string: {}", - e - ); - } - } - } - None => { - error!( - "Template not found for {}:{}", - entry.namespace, entry.id - ); - } - } - } - if input.stream { - match File::create("output.rustforge.bin") { - Ok(mut file) => { - if let Err(e) = write_chunked_header(&mut file) { - error!( - "Failed to write chunked output header: {}", - e - ); - return; - } - - let mut wrote = 0usize; - for ((namespace, task), tasks) in - &api.task_queue.tasks - { - for stored in tasks { - let record = ChunkedTaskRecord { - namespace: namespace.clone(), - task: task.clone(), - id: stored.id.clone(), - payload: stored.bytes.clone(), - }; - if let Err(e) = write_chunked_task_record( - &mut file, &record, - ) { - error!( - "Failed to write chunked output record: {}", - e - ); - return; - } - wrote += 1; - } - } - - if let Err(e) = write_chunked_end(&mut file) { - error!( - "Failed to finalize chunked output: {}", - e - ); - return; - } - - info!( - "Wrote output.rustforge.bin (chunked stream, {} task(s))", - wrote - ); - } - Err(e) => { - error!( - "Failed to create output.rustforge.bin: {}", - e - ); - } - } - } else { - match postcard::to_allocvec(&api.task_queue) { - Ok(data) => { - match File::create("output.rustforge.bin") { - Ok(mut file) => { - if let Err(e) = file.write_all(&data) { - error!( - "Failed to write output.rustforge.bin: {}", - e - ); - } else { - info!("Wrote output.rustforge.bin"); - } - } - Err(e) => { - error!( - "Failed to create output.rustforge.bin: {}", - e - ); - } - } - } - Err(e) => { - error!("Failed to serialize task queue: {}", e); - } - } - } - } - Err(e) => { - error!("Failed to parse input TOML: {}", e); - } - } - } - Err(e) => { - error!("Failed to read input file {}: {}", input.input.display(), e); - } - } - } else { - error!("File does not exist: {}", input.input.to_string_lossy()) - } - } - } - } -} diff --git a/engine/src/bin/server.rs b/engine/src/bin/server.rs index 7b70ff8..4a78f83 100644 --- a/engine/src/bin/server.rs +++ b/engine/src/bin/server.rs @@ -1,770 +1,7 @@ -use engine::{get_auth, get_uid}; -use enginelib::plugin::LibraryMetadata; -use enginelib::{ - Identifier, RawIdentifier, Registry, - api::EngineAPI, - chrono::Utc, - event::{debug, info, warn}, - events::{self, Events, ID}, - plugin::LibraryManager, - task::{SolvedTasks, StoredExecutingTask, StoredTask, Task, TaskQueue}, -}; -use proto::{ - TaskState, - engine_server::{Engine, EngineServer}, -}; -use std::{ - collections::HashMap, - env::consts::OS, - io::Read, - net::{Ipv4Addr, Ipv6Addr, SocketAddr, SocketAddrV4}, - sync::{Arc, RwLock as RS_RwLock}, -}; -use tokio::sync::RwLock; -use tonic::{Request, Response, Status, metadata::MetadataValue, transport::Server}; +use enginelib::api::ServerAPI; -use crate::proto::ModuleInfo; - -mod proto { - tonic::include_proto!("engine"); - pub(crate) const FILE_DESCRIPTOR_SET: &[u8] = - tonic::include_file_descriptor_set!("engine_descriptor"); -} -#[allow(non_snake_case)] -struct EngineService { - pub EngineAPI: Arc>, -} -#[tonic::async_trait] -impl Engine for EngineService { - async fn get_metadata( - &self, - request: tonic::Request, - ) -> Result, Status> { - let api = self.EngineAPI.read().await; - - let modules: Vec = api - .lib_manager - .libraries - .values() - .map(|lib| lib.metadata.clone()) - //Dont show server mods - .filter(|lib| !lib.mod_server) - // vec> --> vec - .map(|f| ModuleInfo { - mod_id: f.mod_id.clone(), - api_version: f.api_version.clone(), - rustc_version: f.rustc_version.clone(), - mod_version: f.mod_version.clone(), - }) - .collect(); - - let res = proto::ServerMetadata { - engine_api: enginelib::GIT_VERSION.to_string(), - mods: modules, - }; - return Ok(Response::new(res)); - } - - async fn check_auth( - &self, - request: tonic::Request, - ) -> Result, Status> { - let challenge = get_auth(&request); - let mut api = self.EngineAPI.write().await; - let db = api.db.clone(); - let output = Events::CheckAdminAuth(&mut api, challenge, ("".into(), "".into()), db); - if !output { - warn!("Auth check failed - permission denied"); - return Err(tonic::Status::permission_denied("Invalid Auth")); - }; - return Ok(tonic::Response::new(proto::Empty {})); - } - async fn delete_task( - &self, - request: tonic::Request, - ) -> Result, Status> { - let mut api = self.EngineAPI.write().await; - let data = request.get_ref(); - let challenge = get_auth(&request); - let db = api.db.clone(); - let id = ID(&data.namespace, &data.task); - - let output = Events::CheckAdminAuth(&mut api, challenge, ("".into(), "".into()), db); - if !output { - warn!("Auth check failed - permission denied"); - return Err(tonic::Status::permission_denied("Invalid Auth")); - }; - // Generic helper for removing a task by id from a collection, using an id extractor closure - fn delete_task_from_collection( - collection: &mut HashMap<(String, String), Vec>, - id: &(String, String), - task_id: &str, - state_name: &str, - namespace: &str, - task: &str, - id_extractor: F, - ) -> Result<(), Status> - where - F: Fn(&T) -> &str, - { - match collection.get_mut(id) { - Some(query) => { - let orig_len = query.len(); - query.retain(|f| id_extractor(f) != task_id); - if query.len() == orig_len { - info!( - "DeleteTask: Task with id {} not found in {} state for namespace: {}, task: {}", - task_id, state_name, namespace, task - ); - return Err(Status::not_found(format!( - "Task with id {} not found in {} state", - task_id, state_name - ))); - } - Ok(()) - } - None => { - info!( - "DeleteTask: No tasks found in {} state for namespace: {}, task: {}", - state_name, namespace, task - ); - Err(Status::not_found(format!( - "No tasks found in {} state for given namespace and task", - state_name - ))) - } - } - } - - // Use the helper for each state - let result = match data.state() { - TaskState::Processing => delete_task_from_collection( - &mut api.executing_tasks.tasks, - &id, - &data.id, - "Processing", - &data.namespace, - &data.task, - |f| &f.id, - ), - TaskState::Solved => delete_task_from_collection( - &mut api.solved_tasks.tasks, - &id, - &data.id, - "Solved", - &data.namespace, - &data.task, - |f| &f.id, - ), - TaskState::Queued => delete_task_from_collection( - &mut api.task_queue.tasks, - &id, - &data.id, - "Queued", - &data.namespace, - &data.task, - |f| &f.id, - ), - }; - - if let Err(e) = result { - return Err(e); - } - - // Sync running memory into DB - EngineAPI::sync_db(&mut api); - info!( - "DeleteTask: Successfully deleted task with id {} in state {:?} for namespace: {}, task: {}", - data.id, - data.state(), - data.namespace, - data.task - ); - Ok(tonic::Response::new(proto::Empty {})) - } - /// Retrieves a paginated list of tasks filtered by namespace, task name, and state. - /// - /// Authenticates the request and, if authorized, returns tasks in the specified state - /// (`Processing`, `Queued`, or `Solved`) for the given namespace and task name. The results - /// are sorted by task ID and paginated according to the requested page and page size. - /// - /// Returns a `TaskPage` containing the filtered tasks and pagination metadata, or a - /// permission denied error if authentication fails. - /// - /// # Examples - /// - /// ``` - /// // Example usage within a tonic gRPC client context: - /// let request = proto::TaskPageRequest { - /// namespace: "example_ns".to_string(), - /// task: "example_task".to_string(), - /// state: proto::TaskState::Queued as i32, - /// page: 0, - /// page_size: 10, - /// }; - /// let response = engine_client.get_tasks(request).await?; - /// assert!(response.get_ref().tasks.len() <= 10); - /// ``` - async fn get_tasks( - &self, - request: tonic::Request, - ) -> std::result::Result, tonic::Status> { - let mut api = self.EngineAPI.write().await; - let challenge = get_auth(&request); - - let db = api.db.clone(); - if !Events::CheckAdminAuth(&mut api, challenge, ("".into(), "".into()), db) { - info!("GetTask denied due to Invalid Auth"); - return Err(Status::permission_denied("Invalid authentication")); - }; - let data = request.get_ref(); - - let q: Vec = match data.clone().state() { - TaskState::Processing => { - match api - .executing_tasks - .tasks - .get(&(data.namespace.clone(), data.task.clone())) - { - Some(tasks) => { - let mut task_refs: Vec<_> = tasks.iter().collect(); - task_refs.sort_by_key(|f| &f.id); - task_refs - .iter() - .map(|f| proto::Task { - id: f.id.clone(), - task_id: format!("{}:{}", data.namespace, data.task), - task_payload: f.bytes.clone(), - payload: Vec::new(), - }) - .collect() - } - None => { - info!( - "Namespace {:?} and task {:?} not found in Processing state", - data.namespace, data.task - ); - Vec::new() - } - } - } - TaskState::Queued => { - match api - .task_queue - .tasks - .get(&(data.namespace.clone(), data.task.clone())) - { - Some(tasks) => { - let mut d = tasks.clone(); - d.sort_by_key(|f| f.id.clone()); - d.iter() - .map(|f| proto::Task { - id: f.id.clone(), - task_id: format!("{}:{}", data.namespace, data.task), - task_payload: f.bytes.clone(), - payload: Vec::new(), - }) - .collect() - } - None => { - info!( - "Namespace {:?} and task {:?} not found in Queued state", - data.namespace, data.task - ); - Vec::new() - } - } - } - TaskState::Solved => { - match api - .solved_tasks - .tasks - .get(&(data.namespace.clone(), data.task.clone())) - { - Some(tasks) => { - let mut d = tasks.clone(); - d.sort_by_key(|f| f.id.clone()); - d.iter() - .map(|f| proto::Task { - id: f.id.clone(), - task_id: format!("{}:{}", data.namespace, data.task), - task_payload: f.bytes.clone(), - payload: Vec::new(), - }) - .collect() - } - None => { - info!( - "Namespace {:?} and task {:?} not found in Solved state", - data.namespace, data.task - ); - Vec::new() - } - } - } - }; - let index = data.page * data.page_size as u64; - let end = index + (api.cfg.config_toml.pagination_limit.min(data.page_size) as u64); - let final_vec: Vec<_> = q - .iter() - .skip(index as usize) - .take(data.page_size as usize) - .cloned() - .collect(); - return Ok(tonic::Response::new(proto::TaskPage { - namespace: data.namespace.clone(), - task: data.task.clone(), - page: data.page, - page_size: data.page_size, - state: data.state, - tasks: final_vec, - })); - } - /// Handles custom gRPC messages with admin-level authentication. - /// - /// Processes a CGRPC request by verifying admin credentials and dispatching the event payload to the appropriate handler. Returns the processed event payload in the response. If authentication fails, returns a permission denied error. - /// - /// # Returns - /// A `Cgrpcmsg` response containing the processed event payload, or a permission denied gRPC status on failed authentication. - /// - /// # Examples - /// - /// ``` - /// // Example usage within a gRPC client context: - /// let request = proto::Cgrpcmsg { - /// handler_mod_id: "mod".to_string(), - /// handler_id: "handler".to_string(), - /// event_payload: vec![1, 2, 3], - /// // ... other fields ... - /// }; - /// let response = engine_service.cgrpc(tonic::Request::new(request)).await?; - /// assert_eq!(response.get_ref().handler_mod_id, "mod"); - /// ``` - async fn cgrpc( - &self, - request: tonic::Request, - ) -> std::result::Result, tonic::Status> { - info!( - "CGRPC request received for handler: {}:{}", - request.get_ref().handler_mod_id, - request.get_ref().handler_id - ); - let mut api = self.EngineAPI.write().await; - let challenge = get_auth(&request); - let db = api.db.clone(); - debug!("Checking admin authentication for CGRPC request"); - let output = Events::CheckAdminAuth( - &mut api, - challenge, - ( - request.get_ref().handler_mod_id.clone(), - request.get_ref().handler_id.clone(), - ), - db, - ); - if !output { - warn!("CGRPC auth check failed - permission denied"); - return Err(tonic::Status::permission_denied("Invalid CGRPC Auth")); - }; - let out = Arc::new(std::sync::RwLock::new(Vec::new())); - debug!("Dispatching CGRPC event to handler"); - Events::CgrpcEvent( - &mut api, - ID("engine_core", "grpc"), - request.get_ref().event_payload.clone(), - out.clone(), - ); - let mut res = request.get_ref().clone(); - res.event_payload = match out.read() { - Ok(g) => g.clone(), - Err(_) => { - warn!("CGRPC response lock poisoned, returning empty payload"); - Vec::new() - } - }; - info!("CGRPC request processed successfully"); - return Ok(tonic::Response::new(res)); - } - async fn aquire_task_reg( - &self, - request: tonic::Request, - ) -> Result, tonic::Status> { - let uid = get_uid(&request); - let challenge = get_auth(&request); - info!("Task registry request received from user: {}", uid); - let mut api = self.EngineAPI.write().await; - let db = api.db.clone(); - - debug!("Validating authentication for task registry request"); - if !Events::CheckAuth(&mut api, uid.clone(), challenge, db) { - info!( - "Task registry request denied - invalid authentication for user: {}", - uid - ); - return Err(Status::permission_denied("Invalid authentication")); - }; - let mut tasks: Vec = Vec::new(); - for (k, v) in &api.task_registry.tasks { - let js: Vec = vec![k.0.clone(), k.1.clone()]; - let jstr = js.join(":"); - tasks.push(jstr); - } - info!("Returning task registry with {} tasks", tasks.len()); - let response = proto::TaskRegistry { tasks }; - Ok(tonic::Response::new(response)) - } - - async fn aquire_task( - &self, - request: tonic::Request, - ) -> Result, tonic::Status> { - let challenge = get_auth(&request); - let task_id = request.get_ref().task_id.clone(); - let uid = get_uid(&request); - info!( - "Task acquisition request received from user: {} for task: {}", - uid, task_id - ); - - { - let mut api = self.EngineAPI.write().await; - let db = api.db.clone(); - debug!("Validating authentication for task acquisition"); - if !Events::CheckAuth(&mut api, uid.clone(), challenge, db) { - info!( - "Task acquisition denied - invalid authentication for user: {}", - uid - ); - return Err(Status::permission_denied("Invalid authentication")); - }; - } - - let (namespace, task_name) = task_id.split_once(':').ok_or_else(|| { - info!("Invalid task ID format: {}", task_id); - Status::invalid_argument("Invalid task ID format, expected 'namespace:task'") - })?; - - debug!("Looking up task definition for {}:{}", namespace, task_name); - let key = ID(namespace, task_name); - - { - let api = self.EngineAPI.read().await; - if api.task_registry.get(&key).is_none() { - warn!( - "Task acquisition failed - task does not exist: {}:{}", - namespace, task_name - ); - return Err(Status::invalid_argument("Task Does not Exist")); - } - if Events::ServerBeforeTaskAcquire(&api, uid.clone(), task_id.clone()) { - info!( - "ServerBeforeTaskAcquire cancelled for user: {} task: {}", - uid, task_id - ); - return Err(Status::aborted( - "Task acquire cancelled by server event handler", - )); - } - } - - let (ttask, tasks_key_state, exec_key_state, db) = { - let mut api = self.EngineAPI.write().await; - - let queue = api - .task_queue - .tasks - .get_mut(&key) - .ok_or_else(|| Status::not_found("No queued tasks available"))?; - - if queue.is_empty() { - info!("No queued tasks for {}:{}", namespace, task_name); - return Err(Status::not_found("No queued tasks available")); - } - - let ttask = queue.remove(0); - let task_payload = ttask.bytes.clone(); - - api.executing_tasks - .tasks - .entry(key.clone()) - .or_default() - .push(enginelib::task::StoredExecutingTask { - bytes: task_payload, - user_id: uid.clone(), - given_at: Utc::now(), - id: ttask.id.clone(), - }); - - let tasks_key_state = api.task_queue.tasks.get(&key).cloned().unwrap_or_default(); - let exec_key_state = api - .executing_tasks - .tasks - .get(&key) - .cloned() - .unwrap_or_default(); - let db = api.db.clone(); - (ttask, tasks_key_state, exec_key_state, db) - }; - - let tasks_op = EngineAPI::state_op_tasks(&key, &tasks_key_state) - .map_err(|e| Status::internal(format!("Serialization error: {}", e)))?; - let exec_op = EngineAPI::state_op_executing(&key, &exec_key_state) - .map_err(|e| Status::internal(format!("Serialization error: {}", e)))?; - - EngineAPI::apply_batch_ops(&db, vec![tasks_op, exec_op]) - .map_err(|e| Status::internal(format!("DB insert error: {}", e)))?; - - { - let api = self.EngineAPI.read().await; - Events::ServerTaskAcquired(&api, uid.clone(), task_id.clone(), ttask.id.clone()); - } - - Ok(tonic::Response::new(proto::Task { - id: ttask.id, - task_id, - task_payload: ttask.bytes, - payload: Vec::new(), - })) - } - async fn publish_task( - &self, - request: tonic::Request, - ) -> Result, tonic::Status> { - let challenge = get_auth(&request); - let uid = get_uid(&request); - - let task_id = request.get_ref().task_id.clone(); - let instance_id = request.get_ref().id.clone(); - let payload_for_event = Arc::new(std::sync::RwLock::new( - request.get_ref().task_payload.clone(), - )); - - { - let mut api = self.EngineAPI.write().await; - let db = api.db.clone(); - if !Events::CheckAuth(&mut api, uid.clone(), challenge, db) { - info!("Aquire Task denied due to Invalid Auth"); - return Err(Status::permission_denied("Invalid authentication")); - }; - } - - { - let api = self.EngineAPI.read().await; - if Events::ServerBeforeTaskPublish( - &api, - uid.clone(), - task_id.clone(), - instance_id.clone(), - payload_for_event.clone(), - ) { - info!( - "ServerBeforeTaskPublish cancelled for user: {} task: {}", - uid, task_id - ); - return Err(Status::aborted( - "Task publish cancelled by server event handler", - )); - } - } - - let publish_payload = payload_for_event - .read() - .map(|p| p.clone()) - .map_err(|_| Status::internal("Task publish payload lock poisoned"))?; - - let (namespace, task_name) = task_id - .split_once(':') - .ok_or_else(|| Status::invalid_argument("Invalid Params"))?; - let key = ID(namespace, task_name); - - let reg_tsk = { - let api = self.EngineAPI.read().await; - if !api.task_registry.tasks.contains_key(&key) { - warn!( - "Task acquisition failed - task does not exist: {}:{}", - namespace, task_name - ); - return Err(Status::invalid_argument("Task Does not Exist")); - } - - match api.task_registry.get(&key) { - Some(r) => r, - None => { - warn!("Task registry missing for {}:{}", namespace, task_name); - return Err(Status::invalid_argument("Task Does not Exist")); - } - } - }; - - if !reg_tsk.verify(publish_payload.clone()) { - info!("Failed to parse task"); - return Err(Status::invalid_argument("Failed to parse given task bytes")); - } - - let (solved_id, exec_key_state, solved_key_state, db) = { - let mut api = self.EngineAPI.write().await; - - let solved_id = { - let exec_tasks = api - .executing_tasks - .tasks - .get_mut(&key) - .ok_or_else(|| tonic::Status::not_found("Invalid taskid or userid"))?; - - let idx = exec_tasks - .iter() - .position(|f| f.id == instance_id && f.user_id == uid) - .ok_or_else(|| tonic::Status::not_found("Invalid taskid or userid"))?; - - exec_tasks.remove(idx).id - }; - - api.solved_tasks.tasks.entry(key.clone()).or_default().push( - enginelib::task::StoredTask { - bytes: publish_payload.clone(), - id: solved_id.clone(), - }, - ); - - let exec_key_state = api - .executing_tasks - .tasks - .get(&key) - .cloned() - .unwrap_or_default(); - let solved_key_state = api - .solved_tasks - .tasks - .get(&key) - .cloned() - .unwrap_or_default(); - let db = api.db.clone(); - - (solved_id, exec_key_state, solved_key_state, db) - }; - - let exec_op = EngineAPI::state_op_executing(&key, &exec_key_state) - .map_err(|e| Status::internal(format!("Serialization error: {}", e)))?; - let solved_op = EngineAPI::state_op_solved(&key, &solved_key_state) - .map_err(|e| Status::internal(format!("Serialization error: {}", e)))?; - - EngineAPI::apply_batch_ops(&db, vec![exec_op, solved_op]) - .map_err(|e| Status::internal(format!("DB insert error: {}", e)))?; - - { - let api = self.EngineAPI.read().await; - Events::ServerTaskPublished(&api, uid.clone(), task_id.clone(), solved_id); - } - - info!("Task published successfully: {} by user: {}", task_id, uid); - Ok(tonic::Response::new(proto::Empty {})) - } - async fn create_task( - &self, - request: tonic::Request, - ) -> Result, tonic::Status> { - let mut api = self.EngineAPI.write().await; - let challenge = get_auth(&request); - let uid = get_uid(&request); - let db = api.db.clone(); - if !Events::CheckAuth(&mut api, uid, challenge, db) { - //TODO: change to AdminSpecific Auth - info!("Create Task denied due to Invalid Auth"); - return Err(Status::permission_denied("Invalid authentication")); - }; - let task = request.get_ref(); - let task_id = task.task_id.clone(); - let payload_for_event = Arc::new(std::sync::RwLock::new(task.task_payload.clone())); - if Events::ServerBeforeTaskCreate(&api, task_id.clone(), payload_for_event.clone()) { - info!("ServerBeforeTaskCreate cancelled for task: {}", task_id); - return Err(Status::aborted( - "Task create cancelled by server event handler", - )); - } - let task_payload = payload_for_event - .read() - .map(|p| p.clone()) - .map_err(|_| Status::internal("Task create payload lock poisoned"))?; - - let parts: Vec<&str> = task_id.splitn(2, ':').collect(); - if parts.len() != 2 || parts[0].is_empty() || parts[1].is_empty() { - return Err(Status::invalid_argument( - "Invalid task ID format, expected 'namespace:task'", - )); - } - let id: Identifier = (parts[0].to_string(), parts[1].to_string()); - let tsk_reg = api.task_registry.get(&id); - if let Some(tsk_reg) = tsk_reg { - if !tsk_reg.clone().verify(task_payload.clone()) { - warn!("Failed to parse given task bytes"); - return Err(Status::invalid_argument("Failed to parse given task bytes")); - } - let tbp_tsk = StoredTask { - bytes: task_payload.clone(), - id: druid::Druid::default().to_hex(), - }; - api.task_queue - .tasks - .entry(id.clone()) - .or_default() - .push(tbp_tsk.clone()); - - let tasks_key_state = api.task_queue.tasks.get(&id).cloned().unwrap_or_default(); - let task_op = EngineAPI::state_op_tasks(&id, &tasks_key_state) - .map_err(|e| Status::internal(format!("Serialization error: {}", e)))?; - if let Err(e) = EngineAPI::apply_batch_ops(&api.db, vec![task_op]) { - return Err(Status::internal(format!("DB insert error: {}", e))); - } - Events::ServerTaskCreated( - &api, - task_id.clone(), - tbp_tsk.id.clone(), - Arc::new(std::sync::RwLock::new(tbp_tsk.bytes.clone())), - ); - return Ok(tonic::Response::new(proto::Task { - id: tbp_tsk.id.clone(), - task_id: task_id.clone(), - payload: Vec::new(), - task_payload: tbp_tsk.bytes.clone(), - })); - } - Err(tonic::Status::aborted("Error")) - } -} - -#[tokio::main(flavor = "multi_thread", worker_threads = 8)] -async fn main() -> Result<(), Box> { - let mut api = EngineAPI::default(); - EngineAPI::init(&mut api); - Events::init_auth(&mut api); - Events::StartEvent(&mut api); - Events::ServerStart(&api); - let addr = api - .cfg - .config_toml - .host - .parse() - .unwrap_or(SocketAddr::V4(SocketAddrV4::new( - Ipv4Addr::new(127, 0, 0, 1), - 50051, - ))); - let apii = Arc::new(RwLock::new(api)); - EngineAPI::init_chron(apii.clone()); - let engine = EngineService { EngineAPI: apii }; - - // Build reflection service, mapping its concrete error into Box - let reflection_service = tonic_reflection::server::Builder::configure() - .register_encoded_file_descriptor_set(proto::FILE_DESCRIPTOR_SET) - .build_v1() - .map_err(|e| Box::new(e) as Box)?; - - // Start server and map transport errors into Box so `?` works with our return type. - Server::builder() - .add_service(reflection_service) - .add_service(EngineServer::new(engine)) - .serve(addr) - .await - .map_err(|e| Box::new(e) as Box)?; - - Ok(()) +fn main() { + let mut api = ServerAPI::init(); + // let server = enginelib::server::RPC::new(&api); + // server.run(); } diff --git a/engine/src/lib.rs b/engine/src/lib.rs deleted file mode 100644 index 5b529f0..0000000 --- a/engine/src/lib.rs +++ /dev/null @@ -1,17 +0,0 @@ -use tonic::Request; - -pub fn get_uid(req: &Request) -> String { - req.metadata() - .get("uid") - .and_then(|v| v.to_str().ok()) - .map(|s| s.to_string()) - .unwrap_or_default() -} - -pub fn get_auth(req: &Request) -> String { - req.metadata() - .get("authorization") - .and_then(|v| v.to_str().ok()) - .map(|s| s.to_string()) - .unwrap_or_default() -} diff --git a/enginelib/Cargo.toml b/enginelib/Cargo.toml index 0481fa3..2d97185 100644 --- a/enginelib/Cargo.toml +++ b/enginelib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "enginelib" -version = "0.2.0" +version = "0.3.0" edition = "2024" description = "A library for the GE engine allowing for modding and removing boilerplate." license-file = "LICENSE.md" @@ -16,16 +16,19 @@ toml = { workspace = true } tracing = "0.1.44" tracing-subscriber = "0.3.22" chrono = { version = "0.4.44", features = ["serde"] } -sled = "0.34.7" tokio = { version = "1.50.0", features = ["full"] } postcard = { version = "1.1.3", features = ["use-std"] } inventory = "0.3.22" crossbeam = "0.8.4" +dashmap = "6.2.1" +async-channel = "2.5.0" +druid = {git="https://github.com/voltaero/druid.git"} [build-dependencies] vergen-gix = { version = "9.1.0", features = ["build", "cargo", "rustc"] } -[profile.release] -codegen-units = 1 # Make builds deterministic -[profile.dev] -codegen-units = 1 # Make builds deterministic +# Profiles moved to the workspace root Cargo.toml — Cargo ignores profile +# tables in non-root member manifests. [dev-dependencies] tracing-test = "0.2.6" +[dependencies.rust-rocksdb] +features = ["jemalloc"] +version = "0.51.0" diff --git a/enginelib/src/api.rs b/enginelib/src/api.rs index aab43f8..0220a5c 100644 --- a/enginelib/src/api.rs +++ b/enginelib/src/api.rs @@ -1,280 +1,58 @@ -use chrono::Utc; -use crossbeam::queue::ArrayQueue; -use tokio::{spawn, sync::RwLock, time::interval}; -use tracing::{Level, debug, error, info, instrument}; - -use crate::{ - Identifier, Registry, - config::Config, - event::{EngineEventHandlerRegistry, EventBus}, - plugin::LibraryManager, - task::{LeasedTaskQueue, StoredTask, Task, TaskQueue}, -}; +use crate::error::Error; +use crate::task::Task; +use crate::{Identifier, Registry, config::Config, event::EventBus, plugin::LibraryManager}; +use chrono::{DateTime, Utc}; +use dashmap::{DashMap, DashSet}; pub use postcard; pub use postcard::from_bytes; pub use postcard::to_allocvec; -use std::{ - collections::{HashMap, HashSet}, - sync::Arc, - time::Duration, -}; +use serde::{Deserialize, Serialize}; +use std::sync::Arc; +use tracing::{Level, debug, instrument}; -pub struct ServerAPI { - pub cfg: Config, // RW - pub task_queue: TaskQueue, // RW - pub leased_tasks: LeasedTaskQueue, // RW - pub task_registry: EngineTaskRegistry, // RW - pub event_bus: EventBus, // RW - pub db: sled::Db, // R - pub lib_manager: LibraryManager, // RW - pub client: bool, // RW +// t:namespace:task_name: -> Serialized Task Record +pub fn task_key_prefix(task_type: &Identifier) -> String { + format!("t:{}:{}:", task_type.0, task_type.1) } -impl Default for ServerAPI { - fn default() -> Self { - Self { - cfg: Config::default(), - task_queue: TaskQueue::default(), - db: sled::open("engine_db").unwrap(), - lib_manager: LibraryManager::default(), - task_registry: EngineTaskRegistry::default(), - event_bus: EventBus { - event_handler_registry: EngineEventHandlerRegistry { - event_handlers: HashMap::new(), - }, - }, - leased_tasks: LeasedTaskQueue::default(), - client: false, - } - } +pub fn task_key(task_type: &Identifier, task_id: &str) -> String { + format!("t:{}:{}:{}", task_type.0, task_type.1, task_id) } -impl ServerAPI { - // pub fn default_client() -> Self { - // Self { - // cfg: Config::default(), - // task_queue: TaskQueue::default(), - // db: sled::open("engine_client_db").unwrap(), - // lib_manager: LibraryManager::default(), - // task_registry: EngineTaskRegistry::default(), - // event_bus: EventBus { - // event_handler_registry: EngineEventHandlerRegistry { - // event_handlers: HashMap::new(), - // }, - // }, - // solved_tasks: SolvedTasks::default(), - // executing_tasks: ExecutingTaskQueue::default(), - // client: true, - // } - //} - pub fn test_default() -> Self { - // `sled::Config::temporary(true)` defaults to `/dev/shm` on Linux when no path is set. - // Some environments deny writes there, so force a unique temp path. - use std::sync::atomic::{AtomicUsize, Ordering}; - - static DB_COUNTER: AtomicUsize = AtomicUsize::new(0); - let db_id = DB_COUNTER.fetch_add(1, Ordering::Relaxed); - let db_path = std::env::temp_dir().join(format!( - "enginelib-test-db-{}-{}", - std::process::id(), - db_id - )); - - Self { - client: false, - cfg: Config::new(), - leased_tasks: LeasedTaskQueue::default(), - task_queue: TaskQueue::default(), - db: sled::Config::new() - .path(db_path) - .temporary(true) - .flush_every_ms(None) - .open() - .unwrap(), - lib_manager: LibraryManager::default(), - task_registry: EngineTaskRegistry::default(), - event_bus: EventBus { - event_handler_registry: EngineEventHandlerRegistry { - event_handlers: HashMap::new(), - }, - }, - } - } - pub fn init(api: &mut Self) { - Self::setup_logger(); - api.cfg = Config::new(); - Self::init_db(api); - let mut new_lib_manager = LibraryManager::default(); - new_lib_manager.load_modules(api); - api.lib_manager = new_lib_manager; - for (id, _tsk) in api.task_registry.tasks.iter() { - api.task_queue - .tasks - .entry(id.clone()) - .insert_entry(ArrayQueue::new( - api.cfg.config_toml.task_block_size as usize, - )); - api.leased_tasks.tasks.entry(id.clone()).or_default(); - } - - Self::init_events(api); - } - - fn init_events(api: &mut Self) { - crate::event::register_inventory_handlers(api); - } - pub fn init_chron(api: Arc>) { - let t = api.try_read().unwrap().cfg.config_toml.clean_tasks; - spawn(clear_sled_periodically(api, t)); - } - const TASKS_PREFIX: &'static str = "tasks:"; - const LEASING_PREFIX: &'static str = "leasing:"; - const SOLVED_PREFIX: &'static str = "solved:"; - fn state_key(prefix: &str, task_id: &Identifier, id: String) -> Vec { - format!("{}{}\u{1f}{}:{}", prefix, task_id.0, task_id.1, id).into_bytes() - } - - fn parse_state_key(prefix: &str, key: &[u8]) -> Option { - let key = std::str::from_utf8(key).ok()?; - let rest = key.strip_prefix(prefix)?; - let (task_id, id) = rest.split_once(":")?; - let (namespace, task) = task_id.split_once('\u{1f}')?; - Some((namespace.to_string(), task.to_string())) - } - - // pub fn apply_batch_entries( - // db: &sled::Db, - // entries: Vec<(&'static str, Vec)>, - // ) -> sled::Result<()> { - // let mut batch = sled::Batch::default(); - // for (key, value) in entries { - // batch.insert(key, value); - // } - // db.apply_batch(batch) - // } - - // pub fn apply_batch_ops( - // db: &sled::Db, - // ops: Vec<(Vec, Option>)>, - // ) -> sled::Result<()> { - // let mut batch = sled::Batch::default(); - // for (key, value) in ops { - // match value { - // Some(v) => batch.insert(key, v), - // None => batch.remove(key), - // } - // } - // db.apply_batch(batch) - // } - - // pub fn state_op_tasks( - // id: &Identifier, - // value: &Vec, - // ) -> Result<(Vec, Option>), postcard::Error> { - // if value.is_empty() { - // Ok((Self::state_key(Self::TASKS_PREFIX, id), None)) - // } else { - // Ok(( - // Self::state_key(Self::TASKS_PREFIX, id), - // Some(postcard::to_allocvec(value)?), - // )) - // } - // } - - // pub fn state_op_executing( - // id: &Identifier, - // value: &Vec, - // ) -> Result<(Vec, Option>), postcard::Error> { - // if value.is_empty() { - // Ok((Self::state_key(Self::LEASING_PREFIX, id), None)) - // } else { - // Ok(( - // Self::state_key(Self::LEASING_PREFIX, id), - // Some(postcard::to_allocvec(value)?), - // )) - // } - // } - - // pub fn state_op_solved( - // id: &Identifier, - // value: &Vec, - // ) -> Result<(Vec, Option>), postcard::Error> { - // if value.is_empty() { - // Ok((Self::state_key(Self::SOLVED_PREFIX, id), None)) - // } else { - // Ok(( - // Self::state_key(Self::SOLVED_PREFIX, id), - // Some(postcard::to_allocvec(value)?), - // )) - // } - // } - - // pub fn sync_db(api: &mut ServerAPI) { - // // IF THIS FN CAUSES PANIC SOMETHING IS VERY BROKEN - // let mut ops: Vec<(Vec, Option>)> = Vec::new(); - - // for prefix in [ - // Self::TASKS_PREFIX, - // Self::LEASING_PREFIX, - // Self::SOLVED_PREFIX, - // ] { - // for item in api.db.scan_prefix(prefix.as_bytes()) { - // if let Ok((key, _)) = item { - // ops.push((key.to_vec(), None)); - // } - // } - // } - - // for (id, tasks) in &api.task_queue.tasks { - // ops.push(Self::state_op_tasks(id, tasks).unwrap()); - // } - // for (id, tasks) in &api.executing_tasks.tasks { - // ops.push(Self::state_op_executing(id, tasks).unwrap()); - // } - // for (id, tasks) in &api.solved_tasks.tasks { - // ops.push(Self::state_op_solved(id, tasks).unwrap()); - // } +// f:namespace:task_name: -> Finished Task Record +pub fn finished_task_key(task_type: &Identifier, task_id: &str) -> String { + format!("f:{}:{}:{}", task_type.0, task_type.1, task_id) +} - // Self::apply_batch_ops(&api.db, ops).unwrap(); - // debug!("Synced in-memory state to keyed sled storage"); - // } +pub fn finished_task_key_prefix(task_type: &Identifier) -> String { + format!("f:{}:{}:", task_type.0, task_type.1) +} - fn init_db(api: &mut ServerAPI) { - api.task_queue = TaskQueue::default(); - api.leased_tasks = LeasedTaskQueue::default(); +pub fn task_id_from_key(key: &str) -> Option<&str> { + key.splitn(4, ':').nth(3) +} - for item in api.db.scan_prefix(Self::TASKS_PREFIX.as_bytes()) { - if let Ok((key, value)) = item { - if let Some(id) = Self::parse_state_key(Self::TASKS_PREFIX, &key) { - if let Ok(tasks) = postcard::from_bytes::(&value) { - api.task_queue.tasks.get(&id).unwrap().push(Arc::new(tasks)); - } - } - } - } +pub struct ServerAPI { + pub cfg: Config, // RW + pub event_bus: EventBus, // RW + pub lib_manager: LibraryManager, // RW + pub db: Arc, + pub task_queue: TaskQueue, // RW + pub leased_tasks: LeasedTaskQueue, // RW + pub task_registry: EngineTaskRegistry, // RW +} - for item in api.db.scan_prefix(Self::LEASING_PREFIX.as_bytes()) { - if let Ok((key, value)) = item { - if let Some(id) = Self::parse_state_key(Self::LEASING_PREFIX, &key) { - if let Ok(tasks) = postcard::from_bytes::>(&value) { - api.executing_tasks.tasks.insert(id, tasks); - } - } - } - } +impl Default for ServerAPI { + fn default() -> Self { + let path = "./engine_db"; + let mut opts = rust_rocksdb::Options::default(); + opts.create_if_missing(true); - for item in api.db.scan_prefix(Self::SOLVED_PREFIX.as_bytes()) { - if let Ok((key, value)) = item { - if let Some(id) = Self::parse_state_key(Self::SOLVED_PREFIX, &key) { - if let Ok(tasks) = postcard::from_bytes::>(&value) { - api.solved_tasks.tasks.insert(id, tasks); - } - } - } - } - } + let db = Arc::new( + rust_rocksdb::DB::open(&opts, path) + .expect("Failed to open RocksDB — check that the path is writable and no other process holds the lock"), + ); - pub fn setup_logger() { use std::sync::OnceLock; static INIT: OnceLock<()> = OnceLock::new(); @@ -294,14 +72,116 @@ impl ServerAPI { // builds the subscriber. .try_init(); }); + + let mut k = Self { + cfg: Config::default(), + event_bus: EventBus::default(), + lib_manager: LibraryManager::default(), + db, + task_registry: EngineTaskRegistry::default(), + task_queue: TaskQueue::default(), + leased_tasks: LeasedTaskQueue::default(), + }; + LibraryManager::load_modules(&mut k); + crate::event::register_inventory_handlers(&mut k); + return k; + } +} +impl ServerAPI { + pub async fn load(api: &Arc, task_type: Identifier) -> Result<(), Error> { + const LOAD_BATCH_SIZE: usize = 4096; + let prefix = task_key_prefix(&task_type); + + // Collect the batch while holding the map guard, but never await under it: + // send() on a full channel would otherwise block all writers to this shard. + let (sender, batch) = { + let k = api + .task_queue + .tasks + .get(&task_type) + .ok_or(Error::not_found("TaskTypeNotFound"))?; + + let mut batch = Vec::new(); + for result in api.db.prefix_iterator(prefix.as_bytes()) { + if batch.len() >= LOAD_BATCH_SIZE { + break; + } + let (key, value) = match result { + Ok(kv) => kv, + Err(err) => { + eprintln!("RocksDB read error: {err}"); + continue; + } + }; + if !key.starts_with(prefix.as_bytes()) { + break; + } + let Ok(key) = String::from_utf8(key.to_vec()) else { + continue; + }; + let Some(task_id) = task_id_from_key(&key) else { + continue; + }; + if k.2.insert(task_id.to_string()) { + batch.push(StoredTask { + bytes: value.into(), + task_id: task_id.to_string(), + task_type: task_type.clone(), + }); + } + } + (k.0.clone(), batch) + }; + + let mut batch = batch.into_iter(); + while let Some(task) = batch.next() { + if let Err(err) = sender.send(task).await { + // Un-mark everything that never made it into the queue so a + // later load() can retry those tasks. + if let Some(k) = api.task_queue.tasks.get(&task_type) { + k.2.remove(&err.0.task_id); + for task in batch { + k.2.remove(&task.task_id); + } + } + return Err(Error::new(format!("Failed to send stored task: {err}"))); + } + } + + Ok(()) + } + pub fn populate(api: &Arc) { + api.task_registry.tasks.iter().for_each(|f| { + let key = f.key(); + let (tx, rx) = async_channel::bounded(8192); // Add to config or make unbound ? + api.task_queue + .tasks + .entry(key.clone()) + .or_insert((tx, rx, DashSet::default())); + api.leased_tasks.tasks.entry(key.clone()).or_default(); + // task reg should be populated by mods + }); + } + pub fn init() -> Arc { + let api = Arc::new(Self::default()); + Self::populate(&api); + let dapi = api.clone(); + std::thread::spawn(move || { + loop { + std::thread::sleep(std::time::Duration::from_secs(3600)); + LeasedTaskQueue::reap_expired(&dapi); + } + }); + api } } + #[derive(Default, Clone, Debug)] pub struct EngineTaskRegistry { - pub tasks: HashMap>, + pub tasks: DashMap>, } impl Registry for EngineTaskRegistry { - #[instrument] + #[instrument(skip_all, fields(namespace = %identifier.0, name = %identifier.1))] fn register(&mut self, task: Arc, identifier: Identifier) { // Insert the task into the hashmap with (mod_id, identifier) as the key debug!( @@ -310,94 +190,67 @@ impl Registry for EngineTaskRegistry { ); self.tasks.insert(identifier, task); } - + #[instrument(skip_all, fields(namespace = %identifier.0, name = %identifier.1))] fn get(&self, identifier: &Identifier) -> Option> { self.tasks.get(identifier).map(|obj| obj.clone_box()) } } +#[derive(Debug, Default, Clone)] +pub struct TaskQueue { + pub tasks: DashMap< + Identifier, + ( + async_channel::Sender, + async_channel::Receiver, + dashmap::DashSet, + ), + >, +} +#[derive(Debug, Default, Clone)] +pub struct LeasedTaskQueue { + pub tasks: DashMap>, +} +#[derive(Debug, Default, Clone, Serialize, Deserialize)] +pub struct StoredTask { + pub bytes: Vec, + pub task_id: String, + pub task_type: Identifier, +} +#[derive(Debug, Default, Clone, Serialize, Deserialize)] +pub struct LeasedTask { + pub stored_task: Arc, + pub user_id: String, + pub given_at: DateTime, +} -pub async fn clear_sled_periodically(api: Arc>, n_minutes: u64) { - info!("Sled Cron Job Started"); - let mut interval = interval(Duration::from_secs(n_minutes * 60)); - loop { - interval.tick().await; - info!("Purging Unsolved Tasks"); - - let now = Utc::now().timestamp(); - let mut rw_api = api.write().await; +const LEASE_TTL_SECS: i64 = 3600; - let mut moved_tasks: Vec<(Identifier, StoredTask)> = Vec::new(); - let mut touched_exec: HashSet = HashSet::new(); +impl LeasedTask { + fn expired(&self) -> bool { + self.given_at.timestamp() + LEASE_TTL_SECS < Utc::now().timestamp() + } +} - for (id, task_list) in rw_api.executing_tasks.tasks.iter_mut() { - let before_len = task_list.len(); - task_list.retain(|info| { - let age = now - info.given_at.timestamp(); - if age > 3600 { - info!("Task {:?} is older than an hour! Moving...", info); - moved_tasks.push(( - id.clone(), - StoredTask { - id: info.id.clone(), - bytes: info.bytes.clone(), - }, - )); +impl LeasedTaskQueue { + fn reap_expired(api: &Arc) { + let mut reaped: Vec<(Identifier, String)> = Vec::new(); + api.leased_tasks.tasks.retain(|task_type, tasks| { + tasks.retain(|task| { + if task.expired() { + reaped.push((task_type.clone(), task.stored_task.task_id.clone())); false } else { true } }); - - if task_list.len() != before_len { - touched_exec.insert(id.clone()); - } - } - - let mut touched_tasks: HashSet = HashSet::new(); - for (id, task) in moved_tasks { - rw_api - .task_queue - .tasks - .entry(id.clone()) - .or_default() - .push(task); - touched_tasks.insert(id); - } - - if touched_exec.is_empty() && touched_tasks.is_empty() { - continue; - } - - let mut ops: Vec<(Vec, Option>)> = Vec::new(); - - for id in touched_exec { - let value = rw_api - .executing_tasks - .tasks - .get(&id) - .cloned() - .unwrap_or_default(); - match ServerAPI::state_op_executing(&id, &value) { - Ok(op) => ops.push(op), - Err(e) => error!("Failed to serialize executing_tasks entry: {:?}", e), - } - } - - for id in touched_tasks { - let value = rw_api - .task_queue - .tasks - .get(&id) - .cloned() - .unwrap_or_default(); - match ServerAPI::state_op_tasks(&id, &value) { - Ok(op) => ops.push(op), - Err(e) => error!("Failed to serialize tasks entry: {:?}", e), + !tasks.is_empty() + }); + // Clear reaped ids from the dedup set outside the leased_tasks locks so + // load() can re-enqueue the tasks from their still-persisted records. + for (task_type, task_id) in reaped { + if let Some(queue) = api.task_queue.tasks.get(&task_type) { + queue.2.remove(&task_id); } } - - if let Err(e) = ServerAPI::apply_batch_ops(&rw_api.db, ops) { - error!("Failed to update task state in Sled batch: {:?}", e); - } } } diff --git a/enginelib/src/config.rs b/enginelib/src/config.rs index 2d46cb5..b779bfd 100644 --- a/enginelib/src/config.rs +++ b/enginelib/src/config.rs @@ -1,4 +1,4 @@ -use std::{fs, io::Error, u32}; +use std::{fs, io::Error}; use serde::{Deserialize, Serialize}; use tracing::{error, instrument}; @@ -7,37 +7,20 @@ fn default_host() -> String { "[::1]:50051".into() } -fn default_clean_tasks() -> u64 { - 60 -} -fn default_task_block_size() -> u32 { - 256 -} -fn default_pagination_limit() -> u32 { - u32::MAX -} - #[derive(Debug, Serialize, Deserialize, Clone)] pub struct ConfigTomlServer { - #[serde(default)] - pub cgrpc_token: Option, // Administrator Token, used to invoke cgrpc reqs. If not preset will default to no protection. #[serde(default = "default_host")] pub host: String, - #[serde(default = "default_clean_tasks")] - pub clean_tasks: u64, - #[serde(default = "default_pagination_limit")] - pub pagination_limit: u32, - #[serde(default = "default_task_block_size")] - pub task_block_size: u32, + + // Renamed from cgrpc_token; keep the alias so existing configs still authenticate. + #[serde(alias = "cgrpc_token")] + pub auth_token: Option, } impl Default for ConfigTomlServer { fn default() -> Self { Self { - cgrpc_token: None, host: default_host(), - clean_tasks: default_clean_tasks(), - pagination_limit: default_pagination_limit(), - task_block_size: default_task_block_size(), + auth_token: Option::None, } } } @@ -52,9 +35,9 @@ impl Config { pub fn new() -> Self { let mut content: String = "".to_owned(); let result: Result = fs::read_to_string("config.toml"); - if result.is_ok() { - content = result.unwrap(); - }; + if let Ok(file_content) = result { + content = file_content; + } let config_toml: ConfigTomlServer = toml::from_str(&content).unwrap_or_else(|err| { error!("Failed to parse config file."); error!("{:#?}", err); diff --git a/enginelib/src/error.rs b/enginelib/src/error.rs new file mode 100644 index 0000000..7fae89c --- /dev/null +++ b/enginelib/src/error.rs @@ -0,0 +1,68 @@ +use std::fmt; + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ErrorKind { + NotFound, + NotSupported, + InvalidArgument, + IOError, + Unknown, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Error { + kind: ErrorKind, + message: String, +} + +impl Error { + pub fn new(message: String) -> Error { + Error { + kind: ErrorKind::Unknown, + message, + } + } + + pub fn with_kind(kind: ErrorKind, message: impl Into) -> Error { + Error { + kind, + message: message.into(), + } + } + + pub fn not_found(message: impl Into) -> Error { + Self::with_kind(ErrorKind::NotFound, message) + } + + pub fn invalid_argument(message: impl Into) -> Error { + Self::with_kind(ErrorKind::InvalidArgument, message) + } + + pub fn io_error(message: impl Into) -> Error { + Self::with_kind(ErrorKind::IOError, message) + } + + pub fn kind(&self) -> ErrorKind { + self.kind + } +} + +impl AsRef for Error { + fn as_ref(&self) -> &str { + &self.message + } +} + +impl From for String { + fn from(e: Error) -> String { + e.message + } +} + +impl std::error::Error for Error {} + +impl fmt::Display for Error { + fn fmt(&self, formatter: &mut fmt::Formatter) -> Result<(), fmt::Error> { + self.message.fmt(formatter) + } +} diff --git a/enginelib/src/event.rs b/enginelib/src/event.rs index b8de754..0be2e93 100644 --- a/enginelib/src/event.rs +++ b/enginelib/src/event.rs @@ -1,6 +1,6 @@ use crate::{Identifier, api::ServerAPI}; +use dashmap::DashMap; use std::any::Any; -use std::collections::HashMap; use std::fmt::Debug; use std::sync::Arc; use tracing::instrument; @@ -22,10 +22,10 @@ pub fn register_inventory_handlers_for_origin(api: &mut ServerAPI, origin: &'sta fn register_inventory_handlers_inner(api: &mut ServerAPI, origin: Option<&'static str>) { for item in inventory::iter:: { - if let Some(origin) = origin { - if item.origin != origin { - continue; - } + if let Some(origin) = origin + && item.origin != origin + { + continue; } (item.func)(api); } @@ -51,9 +51,10 @@ pub trait EventCTX: EventHandler { self.handleCTX(event); } + #[allow(non_snake_case)] fn handleCTX(&self, event: &mut C); } - +#[derive(Default, Clone)] pub struct EventBus { pub event_handler_registry: EngineEventHandlerRegistry, } @@ -80,7 +81,7 @@ pub trait EventHandler: Any + Send + Sync { #[derive(Clone, Default)] pub struct EngineEventHandlerRegistry { - pub event_handlers: HashMap>>, + pub event_handlers: DashMap>>, } impl EngineEventHandlerRegistry { @@ -90,7 +91,7 @@ impl EngineEventHandlerRegistry { identifier: Identifier, ) { let handler = Arc::new(handler); - let handlers = self.event_handlers.entry(identifier.clone()).or_default(); + let mut handlers = self.event_handlers.entry(identifier.clone()).or_default(); handlers.push(handler); debug!( "EventBus: Registered handler for event {}.{}", @@ -100,6 +101,14 @@ impl EngineEventHandlerRegistry { } impl EventBus { + pub fn has_handlers(&self, identifier: &Identifier) -> bool { + self.event_handler_registry + .event_handlers + .get(identifier) + .map(|handlers| !handlers.is_empty()) + .unwrap_or(false) + } + pub fn register_handler( &mut self, handler: H, @@ -115,7 +124,7 @@ impl EventBus { debug!("EventBus: Firing event {}.{}", id.0, id.1); if let Some(handlers) = self.event_handler_registry.event_handlers.get(&id) { - for handler in handlers { + for handler in handlers.iter() { if event.is_cancelled() && !handler.receive_cancelled() { continue; } diff --git a/enginelib/src/events/admin_auth_event.rs b/enginelib/src/events/admin_auth_event.rs index c13533d..6f0a156 100644 --- a/enginelib/src/events/admin_auth_event.rs +++ b/enginelib/src/events/admin_auth_event.rs @@ -1,7 +1,7 @@ use std::sync::{Arc, RwLock}; use macros::{Event, event_handler}; -use sled::Db; +use rust_rocksdb::DB; use crate::{Identifier, api::ServerAPI}; @@ -12,17 +12,17 @@ pub struct AdminAuthEvent { pub id: Identifier, pub payload: String, pub target: Identifier, - pub db: Db, + pub db: Arc, pub output: Arc>, } // Event trait auto-implemented by derive macro impl AdminAuthEvent { pub fn fire( - api: &mut ServerAPI, + api: &ServerAPI, payload: String, target: Identifier, - db: Db, + db: Arc, output: Arc>, ) { api.event_bus.fire(&mut AdminAuthEvent { @@ -35,7 +35,7 @@ impl AdminAuthEvent { }); } - pub fn check(api: &mut ServerAPI, payload: String, target: Identifier, db: Db) -> bool { + pub fn check(api: &ServerAPI, payload: String, target: Identifier, db: Arc) -> bool { let output = Arc::new(RwLock::new(false)); Self::fire(api, payload, target, db, output.clone()); *output.read().unwrap() @@ -45,7 +45,7 @@ impl AdminAuthEvent { #[event_handler( namespace = "core", name = "admin_auth_event", - ctx = api.cfg.config_toml.cgrpc_token.clone() + ctx = api.cfg.config_toml.auth_token.clone() )] fn admin_auth_handler(event: &mut AdminAuthEvent, token: &Option) { match token.as_deref() { diff --git a/enginelib/src/events/auth_event.rs b/enginelib/src/events/auth_event.rs index 1056a99..2a3352b 100644 --- a/enginelib/src/events/auth_event.rs +++ b/enginelib/src/events/auth_event.rs @@ -2,7 +2,6 @@ use std::sync::{Arc, RwLock}; use crate::{Identifier, api::ServerAPI}; use macros::{Event, event_handler}; -use sled::Db; #[derive(Clone, Debug, Event)] #[event(namespace = "core", name = "auth_event", cancellable)] @@ -11,15 +10,15 @@ pub struct AuthEvent { pub id: Identifier, pub uid: String, pub challenge: String, - pub db: Db, + pub db: Arc, pub output: Arc>, } impl AuthEvent { pub fn fire( - api: &mut ServerAPI, + api: &ServerAPI, uid: String, challenge: String, - db: Db, + db: Arc, output: Arc>, ) { api.event_bus.fire(&mut AuthEvent { @@ -32,7 +31,12 @@ impl AuthEvent { }); } - pub fn check(api: &mut ServerAPI, uid: String, challenge: String, db: Db) -> bool { + pub fn check( + api: &ServerAPI, + uid: String, + challenge: String, + db: Arc, + ) -> bool { let output = Arc::new(RwLock::new(false)); Self::fire(api, uid, challenge, db, output.clone()); *output.read().unwrap() diff --git a/enginelib/src/events/before_task_acquire_event.rs b/enginelib/src/events/before_task_acquire_event.rs deleted file mode 100644 index 9b85683..0000000 --- a/enginelib/src/events/before_task_acquire_event.rs +++ /dev/null @@ -1,27 +0,0 @@ -use macros::Event; - -use crate::{Identifier, api::ServerAPI}; - -#[derive(Clone, Debug, Event)] -#[event(namespace = "client", name = "before_task_acquire", cancellable)] -pub struct BeforeTaskAcquireEvent { - pub cancelled: bool, - pub id: Identifier, - pub task_id: String, -} - -impl BeforeTaskAcquireEvent { - pub fn fire(api: &ServerAPI, task_id: String) -> Self { - let mut event = BeforeTaskAcquireEvent { - cancelled: false, - id: ("client".to_string(), "before_task_acquire".to_string()), - task_id, - }; - api.event_bus.fire(&mut event); - event - } - - pub fn check(api: &ServerAPI, task_id: String) -> bool { - Self::fire(api, task_id).cancelled - } -} diff --git a/enginelib/src/events/before_task_block_acquire_event.rs b/enginelib/src/events/before_task_block_acquire_event.rs new file mode 100644 index 0000000..13a79f1 --- /dev/null +++ b/enginelib/src/events/before_task_block_acquire_event.rs @@ -0,0 +1,30 @@ +use macros::Event; + +use crate::{Identifier, api::ServerAPI}; + +#[derive(Clone, Debug, Event)] +#[event(namespace = "client", name = "before_task_block_acquire", cancellable)] +pub struct BeforeTaskBlockAcquireEvent { + pub cancelled: bool, + pub id: Identifier, + pub task_ids: Vec, +} + +impl BeforeTaskBlockAcquireEvent { + pub fn fire(api: &ServerAPI, task_ids: Vec) -> Self { + let mut event = BeforeTaskBlockAcquireEvent { + cancelled: false, + id: ( + "client".to_string(), + "before_task_block_acquire".to_string(), + ), + task_ids, + }; + api.event_bus.fire(&mut event); + event + } + + pub fn check(api: &ServerAPI, task_ids: Vec) -> bool { + Self::fire(api, task_ids).cancelled + } +} diff --git a/enginelib/src/events/before_task_block_execute_event.rs b/enginelib/src/events/before_task_block_execute_event.rs new file mode 100644 index 0000000..5da0ccc --- /dev/null +++ b/enginelib/src/events/before_task_block_execute_event.rs @@ -0,0 +1,46 @@ +use std::sync::{Arc, RwLock}; + +use macros::Event; + +use crate::{Identifier, api::ServerAPI}; + +#[derive(Clone, Debug, Event)] +#[event(namespace = "client", name = "before_task_block_execute", cancellable)] +pub struct BeforeTaskBlockExecuteEvent { + pub cancelled: bool, + pub id: Identifier, + pub task_id: String, + pub instance_ids: Vec, + pub payloads: Vec>>>, +} + +impl BeforeTaskBlockExecuteEvent { + pub fn fire( + api: &ServerAPI, + task_id: String, + instance_ids: Vec, + payloads: Vec>>>, + ) -> Self { + let mut event = BeforeTaskBlockExecuteEvent { + cancelled: false, + id: ( + "client".to_string(), + "before_task_block_execute".to_string(), + ), + task_id, + instance_ids, + payloads, + }; + api.event_bus.fire(&mut event); + event + } + + pub fn check( + api: &ServerAPI, + task_id: String, + instance_ids: Vec, + payloads: Vec>>>, + ) -> bool { + Self::fire(api, task_id, instance_ids, payloads).cancelled + } +} diff --git a/enginelib/src/events/before_task_block_publish_event.rs b/enginelib/src/events/before_task_block_publish_event.rs new file mode 100644 index 0000000..c97cb64 --- /dev/null +++ b/enginelib/src/events/before_task_block_publish_event.rs @@ -0,0 +1,46 @@ +use std::sync::{Arc, RwLock}; + +use macros::Event; + +use crate::{Identifier, api::ServerAPI}; + +#[derive(Clone, Debug, Event)] +#[event(namespace = "client", name = "before_task_block_publish", cancellable)] +pub struct BeforeTaskBlockPublishEvent { + pub cancelled: bool, + pub id: Identifier, + pub task_id: String, + pub instance_ids: Vec, + pub payloads: Vec>>>, +} + +impl BeforeTaskBlockPublishEvent { + pub fn fire( + api: &ServerAPI, + task_id: String, + instance_ids: Vec, + payloads: Vec>>>, + ) -> Self { + let mut event = BeforeTaskBlockPublishEvent { + cancelled: false, + id: ( + "client".to_string(), + "before_task_block_publish".to_string(), + ), + task_id, + instance_ids, + payloads, + }; + api.event_bus.fire(&mut event); + event + } + + pub fn check( + api: &ServerAPI, + task_id: String, + instance_ids: Vec, + payloads: Vec>>>, + ) -> bool { + Self::fire(api, task_id, instance_ids, payloads).cancelled + } +} diff --git a/enginelib/src/events/before_task_execute_event.rs b/enginelib/src/events/before_task_execute_event.rs deleted file mode 100644 index 07e4d63..0000000 --- a/enginelib/src/events/before_task_execute_event.rs +++ /dev/null @@ -1,43 +0,0 @@ -use std::sync::{Arc, RwLock}; - -use macros::Event; - -use crate::{Identifier, api::ServerAPI}; - -#[derive(Clone, Debug, Event)] -#[event(namespace = "client", name = "before_task_execute", cancellable)] -pub struct BeforeTaskExecuteEvent { - pub cancelled: bool, - pub id: Identifier, - pub task_id: String, - pub instance_id: String, - pub payload: Arc>>, -} - -impl BeforeTaskExecuteEvent { - pub fn fire( - api: &ServerAPI, - task_id: String, - instance_id: String, - payload: Arc>>, - ) -> Self { - let mut event = BeforeTaskExecuteEvent { - cancelled: false, - id: ("client".to_string(), "before_task_execute".to_string()), - task_id, - instance_id, - payload, - }; - api.event_bus.fire(&mut event); - event - } - - pub fn check( - api: &ServerAPI, - task_id: String, - instance_id: String, - payload: Arc>>, - ) -> bool { - Self::fire(api, task_id, instance_id, payload).cancelled - } -} diff --git a/enginelib/src/events/before_task_publish_event.rs b/enginelib/src/events/before_task_publish_event.rs deleted file mode 100644 index f3c31dd..0000000 --- a/enginelib/src/events/before_task_publish_event.rs +++ /dev/null @@ -1,43 +0,0 @@ -use std::sync::{Arc, RwLock}; - -use macros::Event; - -use crate::{Identifier, api::ServerAPI}; - -#[derive(Clone, Debug, Event)] -#[event(namespace = "client", name = "before_task_publish", cancellable)] -pub struct BeforeTaskPublishEvent { - pub cancelled: bool, - pub id: Identifier, - pub task_id: String, - pub instance_id: String, - pub payload: Arc>>, -} - -impl BeforeTaskPublishEvent { - pub fn fire( - api: &ServerAPI, - task_id: String, - instance_id: String, - payload: Arc>>, - ) -> Self { - let mut event = BeforeTaskPublishEvent { - cancelled: false, - id: ("client".to_string(), "before_task_publish".to_string()), - task_id, - instance_id, - payload, - }; - api.event_bus.fire(&mut event); - event - } - - pub fn check( - api: &ServerAPI, - task_id: String, - instance_id: String, - payload: Arc>>, - ) -> bool { - Self::fire(api, task_id, instance_id, payload).cancelled - } -} diff --git a/enginelib/src/events/cgrpc_event.rs b/enginelib/src/events/cgrpc_event.rs index ef821a0..110768b 100644 --- a/enginelib/src/events/cgrpc_event.rs +++ b/enginelib/src/events/cgrpc_event.rs @@ -16,7 +16,7 @@ pub struct CgrpcEvent { impl CgrpcEvent { pub fn fire( - api: &mut ServerAPI, + api: &ServerAPI, handler_id: Identifier, payload: Vec, output: Arc>>, diff --git a/enginelib/src/events/mod.rs b/enginelib/src/events/mod.rs index 48ee7a5..8383011 100644 --- a/enginelib/src/events/mod.rs +++ b/enginelib/src/events/mod.rs @@ -1,8 +1,8 @@ pub mod admin_auth_event; pub mod auth_event; -pub mod before_task_acquire_event; -pub mod before_task_execute_event; -pub mod before_task_publish_event; +pub mod before_task_block_acquire_event; +pub mod before_task_block_execute_event; +pub mod before_task_block_publish_event; pub mod cgrpc_event; pub mod client_auth_prepare_event; pub mod client_start_event; @@ -10,43 +10,43 @@ pub mod server_before_task_acquire_event; pub mod server_before_task_create_event; pub mod server_before_task_publish_event; pub mod server_start_event; -pub mod server_task_acquired_event; -pub mod server_task_created_event; -pub mod server_task_published_event; +pub mod server_task_block_acquired_event; +pub mod server_task_block_created_event; +pub mod server_task_block_published_event; pub mod start_event; -pub mod task_acquired_event; +pub mod task_block_acquired_event; use std::collections::HashMap; use std::sync::{Arc, RwLock}; -use sled::Db; +use rust_rocksdb::DB; use crate::{Identifier, api::ServerAPI}; +#[allow(non_snake_case)] pub fn ID(namespace: &str, id: &str) -> Identifier { (namespace.to_string(), id.to_string()) } - pub struct Events; #[allow(non_snake_case)] impl Events { pub fn init_auth(_api: &mut ServerAPI) {} - pub fn CheckAuth(api: &mut ServerAPI, uid: String, challenge: String, db: Db) -> bool { + pub fn CheckAuth(api: &ServerAPI, uid: String, challenge: String, db: Arc) -> bool { auth_event::AuthEvent::check(api, uid, challenge, db) } pub fn CheckAdminAuth( - api: &mut ServerAPI, + api: &ServerAPI, payload: String, target: Identifier, - db: Db, + db: Arc, ) -> bool { admin_auth_event::AdminAuthEvent::check(api, payload, target, db) } pub fn CgrpcEvent( - api: &mut ServerAPI, + api: &ServerAPI, handler_id: Identifier, payload: Vec, output: Arc>>, @@ -66,35 +66,50 @@ impl Events { client_auth_prepare_event::ClientAuthPrepareEvent::fire(api, headers) } - pub fn BeforeTaskAcquire(api: &ServerAPI, task_id: String) -> bool { - before_task_acquire_event::BeforeTaskAcquireEvent::check(api, task_id) + pub fn BeforeTaskBlockAcquire(api: &ServerAPI, task_ids: Vec) -> bool { + before_task_block_acquire_event::BeforeTaskBlockAcquireEvent::check(api, task_ids) } - pub fn TaskAcquired( + pub fn TaskBlockAcquired( api: &ServerAPI, task_id: String, - instance_id: String, - payload: Arc>>, + instance_ids: Vec, + payloads: Vec>>>, ) { - task_acquired_event::TaskAcquiredEvent::fire(api, task_id, instance_id, payload) + task_block_acquired_event::TaskBlockAcquiredEvent::fire( + api, + task_id, + instance_ids, + payloads, + ) } - pub fn BeforeTaskExecute( + pub fn BeforeTaskBlockExecute( api: &ServerAPI, task_id: String, - instance_id: String, - payload: Arc>>, + instance_ids: Vec, + payloads: Vec>>>, ) -> bool { - before_task_execute_event::BeforeTaskExecuteEvent::check(api, task_id, instance_id, payload) + before_task_block_execute_event::BeforeTaskBlockExecuteEvent::check( + api, + task_id, + instance_ids, + payloads, + ) } - pub fn BeforeTaskPublish( + pub fn BeforeTaskBlockPublish( api: &ServerAPI, task_id: String, - instance_id: String, - payload: Arc>>, + instance_ids: Vec, + payloads: Vec>>>, ) -> bool { - before_task_publish_event::BeforeTaskPublishEvent::check(api, task_id, instance_id, payload) + before_task_block_publish_event::BeforeTaskBlockPublishEvent::check( + api, + task_id, + instance_ids, + payloads, + ) } pub fn ServerStart(api: &ServerAPI) { @@ -109,21 +124,36 @@ impl Events { server_before_task_create_event::ServerBeforeTaskCreateEvent::check(api, task_id, payload) } - pub fn ServerTaskCreated( + pub fn ServerTaskBlockCreated( api: &ServerAPI, task_id: String, - instance_id: String, - payload: Arc>>, + instance_ids: Vec, + payloads: Vec>>>, ) { - server_task_created_event::ServerTaskCreatedEvent::fire(api, task_id, instance_id, payload) + server_task_block_created_event::ServerTaskBlockCreatedEvent::fire( + api, + task_id, + instance_ids, + payloads, + ) } pub fn ServerBeforeTaskAcquire(api: &ServerAPI, uid: String, task_id: String) -> bool { server_before_task_acquire_event::ServerBeforeTaskAcquireEvent::check(api, uid, task_id) } - pub fn ServerTaskAcquired(api: &ServerAPI, uid: String, task_id: String, instance_id: String) { - server_task_acquired_event::ServerTaskAcquiredEvent::fire(api, uid, task_id, instance_id) + pub fn ServerTaskBlockAcquired( + api: &ServerAPI, + uid: String, + task_id: String, + instance_ids: Vec, + ) { + server_task_block_acquired_event::ServerTaskBlockAcquiredEvent::fire( + api, + uid, + task_id, + instance_ids, + ) } pub fn ServerBeforeTaskPublish( @@ -142,7 +172,17 @@ impl Events { ) } - pub fn ServerTaskPublished(api: &ServerAPI, uid: String, task_id: String, instance_id: String) { - server_task_published_event::ServerTaskPublishedEvent::fire(api, uid, task_id, instance_id) + pub fn ServerTaskBlockPublished( + api: &ServerAPI, + uid: String, + task_id: String, + instance_ids: Vec, + ) { + server_task_block_published_event::ServerTaskBlockPublishedEvent::fire( + api, + uid, + task_id, + instance_ids, + ) } } diff --git a/enginelib/src/events/server_task_acquired_event.rs b/enginelib/src/events/server_task_block_acquired_event.rs similarity index 51% rename from enginelib/src/events/server_task_acquired_event.rs rename to enginelib/src/events/server_task_block_acquired_event.rs index 116ecf1..4d3f7ce 100644 --- a/enginelib/src/events/server_task_acquired_event.rs +++ b/enginelib/src/events/server_task_block_acquired_event.rs @@ -3,23 +3,23 @@ use macros::Event; use crate::{Identifier, api::ServerAPI}; #[derive(Clone, Debug, Event)] -#[event(namespace = "server", name = "task_acquired")] -pub struct ServerTaskAcquiredEvent { +#[event(namespace = "server", name = "task_block_acquired")] +pub struct ServerTaskBlockAcquiredEvent { pub cancelled: bool, pub id: Identifier, pub uid: String, pub task_id: String, - pub instance_id: String, + pub instance_ids: Vec, } -impl ServerTaskAcquiredEvent { - pub fn fire(api: &ServerAPI, uid: String, task_id: String, instance_id: String) { - let mut event = ServerTaskAcquiredEvent { +impl ServerTaskBlockAcquiredEvent { + pub fn fire(api: &ServerAPI, uid: String, task_id: String, instance_ids: Vec) { + let mut event = ServerTaskBlockAcquiredEvent { cancelled: false, - id: ("server".to_string(), "task_acquired".to_string()), + id: ("server".to_string(), "task_block_acquired".to_string()), uid, task_id, - instance_id, + instance_ids, }; api.event_bus.fire(&mut event); } diff --git a/enginelib/src/events/server_task_block_created_event.rs b/enginelib/src/events/server_task_block_created_event.rs new file mode 100644 index 0000000..9f5a331 --- /dev/null +++ b/enginelib/src/events/server_task_block_created_event.rs @@ -0,0 +1,33 @@ +use std::sync::{Arc, RwLock}; + +use macros::Event; + +use crate::{Identifier, api::ServerAPI}; + +#[derive(Clone, Debug, Event)] +#[event(namespace = "server", name = "task_block_created")] +pub struct ServerTaskBlockCreatedEvent { + pub cancelled: bool, + pub id: Identifier, + pub task_id: String, + pub instance_ids: Vec, + pub payloads: Vec>>>, +} + +impl ServerTaskBlockCreatedEvent { + pub fn fire( + api: &ServerAPI, + task_id: String, + instance_ids: Vec, + payloads: Vec>>>, + ) { + let mut event = ServerTaskBlockCreatedEvent { + cancelled: false, + id: ("server".to_string(), "task_block_created".to_string()), + task_id, + instance_ids, + payloads, + }; + api.event_bus.fire(&mut event); + } +} diff --git a/enginelib/src/events/server_task_published_event.rs b/enginelib/src/events/server_task_block_published_event.rs similarity index 51% rename from enginelib/src/events/server_task_published_event.rs rename to enginelib/src/events/server_task_block_published_event.rs index 5e6ee8a..43b997e 100644 --- a/enginelib/src/events/server_task_published_event.rs +++ b/enginelib/src/events/server_task_block_published_event.rs @@ -3,23 +3,23 @@ use macros::Event; use crate::{Identifier, api::ServerAPI}; #[derive(Clone, Debug, Event)] -#[event(namespace = "server", name = "task_published")] -pub struct ServerTaskPublishedEvent { +#[event(namespace = "server", name = "task_block_published")] +pub struct ServerTaskBlockPublishedEvent { pub cancelled: bool, pub id: Identifier, pub uid: String, pub task_id: String, - pub instance_id: String, + pub instance_ids: Vec, } -impl ServerTaskPublishedEvent { - pub fn fire(api: &ServerAPI, uid: String, task_id: String, instance_id: String) { - let mut event = ServerTaskPublishedEvent { +impl ServerTaskBlockPublishedEvent { + pub fn fire(api: &ServerAPI, uid: String, task_id: String, instance_ids: Vec) { + let mut event = ServerTaskBlockPublishedEvent { cancelled: false, - id: ("server".to_string(), "task_published".to_string()), + id: ("server".to_string(), "task_block_published".to_string()), uid, task_id, - instance_id, + instance_ids, }; api.event_bus.fire(&mut event); } diff --git a/enginelib/src/events/server_task_created_event.rs b/enginelib/src/events/server_task_created_event.rs deleted file mode 100644 index 1326818..0000000 --- a/enginelib/src/events/server_task_created_event.rs +++ /dev/null @@ -1,33 +0,0 @@ -use std::sync::{Arc, RwLock}; - -use macros::Event; - -use crate::{Identifier, api::ServerAPI}; - -#[derive(Clone, Debug, Event)] -#[event(namespace = "server", name = "task_created")] -pub struct ServerTaskCreatedEvent { - pub cancelled: bool, - pub id: Identifier, - pub task_id: String, - pub instance_id: String, - pub payload: Arc>>, -} - -impl ServerTaskCreatedEvent { - pub fn fire( - api: &ServerAPI, - task_id: String, - instance_id: String, - payload: Arc>>, - ) { - let mut event = ServerTaskCreatedEvent { - cancelled: false, - id: ("server".to_string(), "task_created".to_string()), - task_id, - instance_id, - payload, - }; - api.event_bus.fire(&mut event); - } -} diff --git a/enginelib/src/events/task_acquired_event.rs b/enginelib/src/events/task_acquired_event.rs deleted file mode 100644 index 7113cac..0000000 --- a/enginelib/src/events/task_acquired_event.rs +++ /dev/null @@ -1,33 +0,0 @@ -use std::sync::{Arc, RwLock}; - -use macros::Event; - -use crate::{Identifier, api::ServerAPI}; - -#[derive(Clone, Debug, Event)] -#[event(namespace = "client", name = "task_acquired")] -pub struct TaskAcquiredEvent { - pub cancelled: bool, - pub id: Identifier, - pub task_id: String, - pub instance_id: String, - pub payload: Arc>>, -} - -impl TaskAcquiredEvent { - pub fn fire( - api: &ServerAPI, - task_id: String, - instance_id: String, - payload: Arc>>, - ) { - let mut event = TaskAcquiredEvent { - cancelled: false, - id: ("client".to_string(), "task_acquired".to_string()), - task_id, - instance_id, - payload, - }; - api.event_bus.fire(&mut event); - } -} diff --git a/enginelib/src/events/task_block_acquired_event.rs b/enginelib/src/events/task_block_acquired_event.rs new file mode 100644 index 0000000..4b3be50 --- /dev/null +++ b/enginelib/src/events/task_block_acquired_event.rs @@ -0,0 +1,33 @@ +use std::sync::{Arc, RwLock}; + +use macros::Event; + +use crate::{Identifier, api::ServerAPI}; + +#[derive(Clone, Debug, Event)] +#[event(namespace = "client", name = "task_block_acquired")] +pub struct TaskBlockAcquiredEvent { + pub cancelled: bool, + pub id: Identifier, + pub task_id: String, + pub instance_ids: Vec, + pub payloads: Vec>>>, +} + +impl TaskBlockAcquiredEvent { + pub fn fire( + api: &ServerAPI, + task_id: String, + instance_ids: Vec, + payloads: Vec>>>, + ) { + let mut event = TaskBlockAcquiredEvent { + cancelled: false, + id: ("client".to_string(), "task_block_acquired".to_string()), + task_id, + instance_ids, + payloads, + }; + api.event_bus.fire(&mut event); + } +} diff --git a/enginelib/src/lib.rs b/enginelib/src/lib.rs index 7fc70e0..0b375e7 100644 --- a/enginelib/src/lib.rs +++ b/enginelib/src/lib.rs @@ -3,6 +3,7 @@ pub mod api; pub mod config; pub mod event; pub mod events; +pub mod nucleus; extern crate self as enginelib; pub use inventory; pub mod plugin; @@ -17,3 +18,4 @@ pub trait Registry: Default + Clone { fn get(&self, identifier: &Identifier) -> Option>; } pub use chrono; +pub mod error; diff --git a/enginelib/src/nucleus/cancel.rs b/enginelib/src/nucleus/cancel.rs new file mode 100644 index 0000000..e4b7e8b --- /dev/null +++ b/enginelib/src/nucleus/cancel.rs @@ -0,0 +1,48 @@ +use std::sync::Arc; + +use crate::{Identifier, api::ServerAPI, error::Error}; + +/// Cancels the *lease* on a task, not the task itself: removes it from the lease +/// list and resends it into the queue so another worker can pick it up. +/// +/// The persisted record is never touched, and the task_id stays in the dedup set +/// the whole time, so no concurrent `load()` can enqueue a duplicate. The requeue +/// is a non-blocking `try_send` so there is no await window in which the lease is +/// gone but the task isn't yet back in the channel: on success the task moves +/// straight from "leased" to "queued"; if the channel is full or closed we clear +/// the dedup entry instead, dropping the task back to "persisted only" so a later +/// `load()` recovers it. Either way the task is never stranded. +pub fn cancel(api: Arc, task_type: Identifier, task_id: String) -> Result<(), Error> { + // Remove the lease, capturing the stored task so we can requeue it. + let stored_task = { + let mut leases = api + .leased_tasks + .tasks + .get_mut(&task_type) + .ok_or(Error::not_found("TaskNotFound"))?; + let idx = leases + .iter() + .position(|lease| lease.stored_task.task_id == task_id) + .ok_or(Error::not_found("TaskNotFound"))?; + leases.swap_remove(idx).stored_task + }; + + let queue = api + .task_queue + .tasks + .get(&task_type) + .ok_or(Error::not_found("TaskTypeNotFound"))?; + + // Non-blocking requeue: no await means the lease removal and the requeue + // can't be split by an async cancellation. + match queue.0.try_send((*stored_task).clone()) { + // Queued again; task_id stays in the dedup set (still in-flight). + Ok(()) => Ok(()), + // Channel full/closed: clear the dedup entry so the still-persisted + // record is re-enqueued by a later load(). Never left stranded. + Err(_) => { + queue.2.remove(&task_id); + Ok(()) + } + } +} diff --git a/enginelib/src/nucleus/complete.rs b/enginelib/src/nucleus/complete.rs new file mode 100644 index 0000000..191fc57 --- /dev/null +++ b/enginelib/src/nucleus/complete.rs @@ -0,0 +1,62 @@ +use std::sync::Arc; + +use crate::api::{finished_task_key, task_key}; +use crate::{Identifier, Registry, api::ServerAPI, error::Error}; + +pub fn complete( + api: Arc, + task_type: Identifier, + task_id: String, + task_bytes: &[u8], +) -> Result<(), Error> { + let task = api + .task_registry + .get(&task_type) + .ok_or(Error::not_found("TaskTypeNotFound"))?; + if !task.verify(task_bytes) { + return Err(Error::invalid_argument("Failed to verify task bytes")); + } + + // A task can only be completed while it is leased to a worker; drop the + // lease so the reaper can't later re-enqueue it. + let mut leased = false; + if let Some(mut leases) = api.leased_tasks.tasks.get_mut(&task_type) { + let before = leases.len(); + leases.retain(|lease| lease.stored_task.task_id != task_id); + leased = leases.len() != before; + } + if !leased { + return Err(Error::not_found("TaskNotLeased")); + } + + // The lease is already gone, so every remaining exit path must clear the + // dedup entry. On success it prevents load() from re-enqueuing a task whose + // pending record is deleted; on DB failure it lets a later load() re-enqueue + // the *still-persisted* pending record, instead of stranding it (no lease, no + // channel slot, but blocked from re-enqueue by the dedup entry) until restart. + let clear_dedup = || { + if let Some(queue) = api.task_queue.tasks.get(&task_type) { + queue.2.remove(&task_id); + } + }; + + // Persist the finished record before deleting the pending one so a crash + // between the two writes never loses the result. + if let Err(err) = api + .db + .put(finished_task_key(&task_type, &task_id), task_bytes) + { + clear_dedup(); + return Err(Error::io_error(format!( + "Failed to persist finished task: {err}" + ))); + } + if let Err(err) = api.db.delete(task_key(&task_type, &task_id)) { + clear_dedup(); + return Err(Error::io_error(format!("Failed to delete task record: {err}"))); + } + + clear_dedup(); + + Ok(()) +} diff --git a/enginelib/src/nucleus/lease.rs b/enginelib/src/nucleus/lease.rs new file mode 100644 index 0000000..9ba0a93 --- /dev/null +++ b/enginelib/src/nucleus/lease.rs @@ -0,0 +1,40 @@ +use std::sync::Arc; + +use chrono::Utc; + +use crate::api::{LeasedTask, StoredTask}; +use crate::{Identifier, api::ServerAPI, error::Error}; + +pub async fn lease( + api: Arc, + task_type: Identifier, + user_id: String, +) -> Result, Error> { + // Clone the receiver out of the map guard before awaiting; recv() on an + // empty queue would otherwise hold the shard lock indefinitely. + let receiver = api + .task_queue + .tasks + .get(&task_type) + .ok_or(Error::not_found("TaskTypeNotFound"))? + .1 + .clone(); + + let task = receiver + .recv() + .await + .map_err(|err| Error::new(format!("Task queue closed: {err}")))?; + + let stored_task = Arc::new(task); + api.leased_tasks + .tasks + .entry(task_type) + .or_default() + .push(LeasedTask { + stored_task: stored_task.clone(), + user_id, + given_at: Utc::now(), + }); + + Ok(stored_task) +} diff --git a/enginelib/src/nucleus/mod.rs b/enginelib/src/nucleus/mod.rs new file mode 100644 index 0000000..6cd6fef --- /dev/null +++ b/enginelib/src/nucleus/mod.rs @@ -0,0 +1,6 @@ +pub mod cancel; +pub mod complete; +pub mod lease; +pub mod query; +pub mod renew; +pub mod submit; diff --git a/enginelib/src/nucleus/query.rs b/enginelib/src/nucleus/query.rs new file mode 100644 index 0000000..b7915d6 --- /dev/null +++ b/enginelib/src/nucleus/query.rs @@ -0,0 +1,199 @@ +use std::sync::Arc; + +use rust_rocksdb::{Direction, IteratorMode, Range, WriteBatch, properties}; +use serde::{Deserialize, Serialize}; + +use crate::{api::ServerAPI, error::Error}; + +/// Administrative / export operations over the task store. +/// +/// This is the operator-facing surface — inspect, back up, and prune records — +/// distinct from the task lifecycle (submit / lease / complete). Every variant +/// is designed to stay cheap on a store holding 1T+ records: nothing here does +/// a full-keyspace scan or materializes an unbounded result set. +/// +/// Operations are prefix-generic. Build prefixes with the helpers in +/// [`crate::api`] (`task_key_prefix`, `finished_task_key_prefix`, `task_key`, +/// `finished_task_key`) to scope to a task type, its pending records, its +/// finished records, or a single record. +#[derive(Debug, Clone)] +pub enum Query { + /// Fetch a single record by its exact key. O(1) point read (`db.get`, not a + /// scan) — the primitive for addressing one known task, e.g. the id returned + /// by `submit`. `key` must be the full key, built with + /// [`crate::api::task_key`] / [`crate::api::finished_task_key`]. Unlike + /// [`Query::Scan`] this matches the key exactly, never by prefix. + Get { key: String }, + /// Store-wide approximate key count, read from RocksDB SST metadata. + /// O(1) — never scans. Exact counts over a trillion keys are deliberately + /// not offered because they cost a full scan. + EstimateKeys, + /// Approximate on-disk byte size of the records under `prefix`, from SST + /// metadata rather than a scan. Useful for capacity planning and deciding + /// whether a purge is worthwhile. `prefix` must be non-empty. + EstimateSize { prefix: String }, + /// Stream one bounded page of records under `prefix`, resuming strictly + /// after `after`. RocksDB seeks directly to the cursor, so the cost is + /// O(limit) no matter how deep into the keyspace the cursor sits. Drive an + /// export by looping until [`Page::next`] is `None`. + Scan { + prefix: String, + after: Option, + limit: usize, + }, + /// Delete a single record by its exact key. O(1) point delete — the + /// counterpart to [`Query::Get`] and the single-record analogue of + /// [`Query::PurgePrefix`]. Idempotent: absent keys are a no-op. + /// + /// Operates on the raw store only. To remove a *pending* task that may be + /// queued or leased, use `cancel`/`complete` instead, so the in-memory dedup + /// set and lease state stay consistent — this bypasses both. + Delete { key: String }, + /// Drop every record under `prefix` with a single range tombstone. The + /// write cost is independent of how many keys match — the alternative, + /// deleting a billion keys one at a time, is what this exists to avoid. + /// `prefix` must be non-empty. + PurgePrefix { prefix: String }, +} + +/// A single key/value record. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Record { + pub key: String, + pub value: Vec, +} + +/// One bounded page of a [`Query::Scan`]. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Page { + pub records: Vec, + /// Cursor for the next page: pass it back as `Query::Scan { after, .. }`. + /// `None` means the scan is exhausted. + pub next: Option, +} + +/// The outcome of a [`Query`]. +#[derive(Debug, Clone)] +pub enum QueryResult { + /// A single record, or `None` when the key is absent. + Record(Option), + /// Approximate total key count. + EstimateKeys(u64), + /// Approximate on-disk size, in bytes. + EstimateSize(u64), + /// A page of records. + Page(Page), + /// A single-record delete completed. + Deleted, + /// A purge completed. + Purged, +} + +pub fn query(api: Arc, query: Query) -> Result { + match query { + Query::Get { key } => { + let value = api + .db + .get(key.as_bytes()) + .map_err(|err| Error::io_error(format!("Failed to read record: {err}")))?; + Ok(QueryResult::Record(value.map(|value| Record { key, value }))) + } + Query::EstimateKeys => { + let keys = api + .db + .property_int_value(properties::ESTIMATE_NUM_KEYS) + .map_err(|err| Error::io_error(format!("Failed to read key estimate: {err}")))? + .unwrap_or(0); + Ok(QueryResult::EstimateKeys(keys)) + } + Query::EstimateSize { prefix } => { + let end = prefix_successor(prefix.as_bytes()) + .ok_or_else(|| Error::invalid_argument("Cannot size an unbounded prefix"))?; + let size = api + .db + .get_approximate_sizes(&[Range::new(prefix.as_bytes(), &end)]) + .into_iter() + .next() + .unwrap_or(0); + Ok(QueryResult::EstimateSize(size)) + } + Query::Scan { + prefix, + after, + limit, + } => { + // At least one record per page, so the cursor always advances. + let limit = limit.max(1); + // Seek straight to the resume point; no scanning from the store start. + let start = after.as_deref().unwrap_or(prefix.as_str()); + let iter = api + .db + .iterator(IteratorMode::From(start.as_bytes(), Direction::Forward)); + + let mut records: Vec = Vec::with_capacity(limit.min(1024)); + let mut next = None; + for result in iter { + let record = record_from(result)?; + if !record.key.starts_with(&prefix) { + break; + } + // Seeking to the cursor yields it first; resume strictly after it. + if after.as_deref() == Some(record.key.as_str()) { + continue; + } + if records.len() >= limit { + // One more match exists beyond this page: hand back the last + // returned key so the caller resumes right after it. + next = records.last().map(|record| record.key.clone()); + break; + } + records.push(record); + } + Ok(QueryResult::Page(Page { records, next })) + } + Query::Delete { key } => { + api.db + .delete(key.as_bytes()) + .map_err(|err| Error::io_error(format!("Failed to delete record: {err}")))?; + Ok(QueryResult::Deleted) + } + Query::PurgePrefix { prefix } => { + let end = prefix_successor(prefix.as_bytes()) + .ok_or_else(|| Error::invalid_argument("Cannot purge an unbounded prefix"))?; + let mut batch = WriteBatch::default(); + batch.delete_range(prefix.as_bytes(), end.as_slice()); + api.db + .write(&batch) + .map_err(|err| Error::io_error(format!("Failed to purge prefix: {err}")))?; + Ok(QueryResult::Purged) + } + } +} + +/// Smallest key strictly greater than every key with `prefix`, i.e. the +/// exclusive upper bound of the prefix range. `None` when `prefix` is empty or +/// all `0xff` (an unbounded range with no finite successor). +fn prefix_successor(prefix: &[u8]) -> Option> { + let mut end = prefix.to_vec(); + while let Some(last) = end.last_mut() { + if *last < 0xff { + *last += 1; + return Some(end); + } + end.pop(); + } + None +} + +type DbEntry = Result<(Box<[u8]>, Box<[u8]>), rust_rocksdb::Error>; + +fn record_from(result: DbEntry) -> Result { + let (key, value) = + result.map_err(|err| Error::io_error(format!("Failed to read record: {err}")))?; + let key = String::from_utf8(key.into_vec()) + .map_err(|err| Error::io_error(format!("Non-UTF8 key in store: {err}")))?; + Ok(Record { + key, + value: value.into_vec(), + }) +} diff --git a/enginelib/src/nucleus/renew.rs b/enginelib/src/nucleus/renew.rs new file mode 100644 index 0000000..e56ccda --- /dev/null +++ b/enginelib/src/nucleus/renew.rs @@ -0,0 +1,22 @@ +use std::sync::Arc; + +use chrono::Utc; + +use crate::{Identifier, api::ServerAPI, error::Error}; + +pub fn renew(api: Arc, task_type: Identifier, task_id: &str) -> Result<(), Error> { + let mut leases = api + .leased_tasks + .tasks + .get_mut(&task_type) + .ok_or(Error::not_found("TaskTypeNotFound"))?; + + let lease = leases + .iter_mut() + .find(|lease| lease.stored_task.task_id == task_id) + .ok_or(Error::not_found("TaskNotFound"))?; + + lease.given_at = Utc::now(); + + Ok(()) +} diff --git a/enginelib/src/nucleus/submit.rs b/enginelib/src/nucleus/submit.rs new file mode 100644 index 0000000..30ada41 --- /dev/null +++ b/enginelib/src/nucleus/submit.rs @@ -0,0 +1,24 @@ +use std::sync::Arc; + +use crate::api::task_key; +use crate::{Identifier, Registry, api::ServerAPI, error::Error}; +// t:namespace:task_name: -> Serialized Task Record +// f:namespace:task_name: -> Finished Task Record +pub fn submit( + api: Arc, + task_bytes: &[u8], + task_type: Identifier, +) -> Result { + let task = api + .task_registry + .get(&task_type) + .ok_or(Error::not_found("TaskTypeNotFound"))?; + if !task.verify(task_bytes) { + return Err(Error::invalid_argument("Failed to verify task bytes")); + } + let task_id = druid::Druid::default().to_hex(); + api.db + .put(task_key(&task_type, &task_id), task_bytes) + .map_err(|err| Error::io_error(err.to_string()))?; + Ok(task_id) +} diff --git a/enginelib/src/plugin.rs b/enginelib/src/plugin.rs index d9fe447..63e6fee 100644 --- a/enginelib/src/plugin.rs +++ b/enginelib/src/plugin.rs @@ -8,7 +8,8 @@ use std::{collections::HashMap, fs}; use tracing::{debug, error, info}; #[derive(Clone, Debug)] pub struct LibraryInstance { - dynamic_library: Arc>, + // Kept alive for as long as any metadata or registered mod code may be used. + _dynamic_library: Arc>, pub metadata: Arc, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -64,7 +65,8 @@ impl LibraryManager { drop(self); } - pub fn load_modules(&mut self, api: &mut ServerAPI) { + pub fn load_modules(api: &mut ServerAPI) { + let mut library_manager = LibraryManager::default(); let dir_path = "./mods"; let mut files: Vec = Vec::new(); @@ -74,13 +76,10 @@ impl LibraryManager { Ok(entries) => { for entry in entries.filter_map(Result::ok) { let path = entry.path(); - if path.is_file() { - if let Some(extension) = path.extension() { - if extension == "rf" { - debug!("Found valid module file: {}", path.display()); - files.push(path.display().to_string()); - } - } + if path.is_file() && path.extension().is_some_and(|extension| extension == "rf") + { + debug!("Found valid module file: {}", path.display()); + files.push(path.display().to_string()); } } } @@ -92,8 +91,9 @@ impl LibraryManager { info!("Found {} module(s) to load", files.len()); for file in files { - self.load_module(&file, api); + library_manager.load_module(&file, api); } + api.lib_manager = library_manager; } pub fn load_module(&mut self, path: &str, api: &mut ServerAPI) { @@ -170,7 +170,7 @@ impl LibraryManager { self.libraries.insert( metadata.mod_id.clone(), LibraryInstance { - dynamic_library: Arc::new(ManuallyDrop::new(lib)), + _dynamic_library: Arc::new(ManuallyDrop::new(lib)), metadata: Arc::new(metadata.clone()), }, ); diff --git a/enginelib/src/prelude.rs b/enginelib/src/prelude.rs index bee62a5..8e84693 100644 --- a/enginelib/src/prelude.rs +++ b/enginelib/src/prelude.rs @@ -10,11 +10,11 @@ pub use crate::event::{ Event, EventBus, EventCTX, EventHandler, EventRegistrar, register_inventory_handlers, register_inventory_handlers_for_origin, }; -pub use crate::events::admin_auth_event::AdminAuthEvent; +//pub use crate::events::admin_auth_event::AdminAuthEvent; pub use crate::events::auth_event::AuthEvent; -pub use crate::events::before_task_acquire_event::BeforeTaskAcquireEvent; -pub use crate::events::before_task_execute_event::BeforeTaskExecuteEvent; -pub use crate::events::before_task_publish_event::BeforeTaskPublishEvent; +pub use crate::events::before_task_block_acquire_event::BeforeTaskBlockAcquireEvent; +pub use crate::events::before_task_block_execute_event::BeforeTaskBlockExecuteEvent; +pub use crate::events::before_task_block_publish_event::BeforeTaskBlockPublishEvent; pub use crate::events::cgrpc_event::CgrpcEvent; pub use crate::events::client_auth_prepare_event::ClientAuthPrepareEvent; pub use crate::events::client_start_event::ClientStartEvent; @@ -22,11 +22,11 @@ pub use crate::events::server_before_task_acquire_event::ServerBeforeTaskAcquire pub use crate::events::server_before_task_create_event::ServerBeforeTaskCreateEvent; pub use crate::events::server_before_task_publish_event::ServerBeforeTaskPublishEvent; pub use crate::events::server_start_event::ServerStartEvent; -pub use crate::events::server_task_acquired_event::ServerTaskAcquiredEvent; -pub use crate::events::server_task_created_event::ServerTaskCreatedEvent; -pub use crate::events::server_task_published_event::ServerTaskPublishedEvent; +pub use crate::events::server_task_block_acquired_event::ServerTaskBlockAcquiredEvent; +pub use crate::events::server_task_block_created_event::ServerTaskBlockCreatedEvent; +pub use crate::events::server_task_block_published_event::ServerTaskBlockPublishedEvent; pub use crate::events::start_event::StartEvent; -pub use crate::events::task_acquired_event::TaskAcquiredEvent; +pub use crate::events::task_block_acquired_event::TaskBlockAcquiredEvent; pub use crate::events::{Events, ID}; pub use crate::plugin::{LibraryDependency, LibraryMetadata}; pub use crate::task::{Runner, Task, Verifiable}; diff --git a/enginelib/src/task.rs b/enginelib/src/task.rs index 410a90f..4fc08e1 100644 --- a/enginelib/src/task.rs +++ b/enginelib/src/task.rs @@ -1,35 +1,10 @@ use std::fmt::Debug; -use std::{collections::HashMap, sync::Arc}; use crate::Identifier; -use chrono::{DateTime, Utc}; -use crossbeam::queue::ArrayQueue; -use serde::{Deserialize, Serialize}; use tracing::{error, instrument, warn}; -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct StoredTask { - pub bytes: Vec, - pub id: String, -} -#[derive(Debug, Default, Clone, Serialize, Deserialize)] -pub struct LeasedTask { - pub stored_task: Arc, - pub user_id: String, - pub given_at: DateTime, -} -#[derive(Debug, Default)] -pub struct TaskQueue { - pub tasks: HashMap>>, -} - -#[derive(Debug, Default, Clone)] -pub struct LeasedTaskQueue { - pub tasks: HashMap>, -} - pub trait Verifiable { - fn verify(&self, b: Vec) -> bool; + fn verify(&self, b: &[u8]) -> bool; } pub trait Task: Debug + Sync + Send + Verifiable { fn get_id(&self) -> Identifier; @@ -61,6 +36,7 @@ pub trait Task: Debug + Sync + Send + Verifiable { fn to_bytes(&self) -> Vec; #[allow(clippy::wrong_self_convention)] fn from_bytes(&self, bytes: &[u8]) -> Box; + #[allow(clippy::wrong_self_convention)] fn from_toml(&self, d: String) -> Box; fn to_toml(&self) -> String; } diff --git a/enginelib/tests/event_tests.rs b/enginelib/tests/event_tests.rs deleted file mode 100644 index ed5ec9a..0000000 --- a/enginelib/tests/event_tests.rs +++ /dev/null @@ -1,221 +0,0 @@ -use enginelib::{Registry, api::EngineAPI, events::ID, task::Verifiable}; -use macros::Verifiable; -use std::sync::Arc; -use tracing_test::traced_test; - -#[derive(Clone, Debug, macros::Event)] -#[event(namespace = "test", name = "test_event", cancellable)] -struct TestEvent { - pub value: i32, - pub cancelled: bool, -} - -#[macros::event_handler(namespace = "test", name = "test_event")] -fn increment_test_event(event: &mut TestEvent) { - event.value += 1; -} - -#[derive(Clone, Debug, macros::Event)] -#[event(namespace = "test", name = "stateful_event")] -struct StatefulEvent { - pub value: u32, -} - -#[macros::event_handler(namespace = "test", name = "stateful_event", ctx = 7u32)] -fn add_ctx_to_event(event: &mut StatefulEvent, ctx: &u32) { - event.value += *ctx; -} - -#[traced_test] -#[test] -fn id() { - assert!(ID("namespace", "id") == ID("namespace", "id")) -} - -#[traced_test] -#[test] -fn test_event_registration_and_handling() { - let mut api = EngineAPI::test_default(); - - let mut test_event = TestEvent { - value: 0, - cancelled: false, - }; - - enginelib::event::register_inventory_handlers(&mut api); - api.event_bus.fire(&mut test_event); - assert_eq!(test_event.value, 1); - drop(api.db); -} - -#[traced_test] -#[test] -fn test_stateful_event_auto_registration() { - let mut api = EngineAPI::test_default(); - - enginelib::event::register_inventory_handlers(&mut api); - - let mut event = StatefulEvent { value: 0 }; - api.event_bus.fire(&mut event); - assert_eq!(event.value, 7); - drop(api.db); -} - -#[traced_test] -#[test] -fn test_task_registration() { - use enginelib::task::Task; - use serde::{Deserialize, Serialize}; - - #[derive(Debug, Clone, Serialize, Deserialize, Verifiable)] - struct TestTask { - pub value: i32, - pub id: (String, String), - } - - impl Task for TestTask { - fn to_toml(&self) -> String { - "".into() - } - fn get_id(&self) -> (String, String) { - self.id.clone() - } - fn clone_box(&self) -> Box { - Box::new(self.clone()) - } - fn run_cpu(&mut self) { - self.value += 1; - } - fn to_bytes(&self) -> Vec { - postcard::to_allocvec(self).unwrap() - } - fn from_bytes(&self, bytes: &[u8]) -> Box { - Box::new(postcard::from_bytes::(bytes).unwrap()) - } - fn from_toml(&self, d: String) -> Box { - return Box::new(self.clone()); - } - } - let mut api = EngineAPI::test_default(); - let task_id = ID("test", "test_task"); - - // Register the task type - api.task_registry.register( - Arc::new(TestTask { - value: 0, - id: task_id.clone(), - }), - task_id.clone(), - ); - - // Verify it was registered - assert!(api.task_registry.tasks.contains_key(&task_id)); - drop(api.db); -} - -#[traced_test] -#[test] -fn test_task_execution() { - use enginelib::task::{Runner, Task}; - use serde::{Deserialize, Serialize}; - - #[derive(Debug, Clone, Serialize, Deserialize, Verifiable)] - struct TestTask { - pub value: i32, - pub id: (String, String), - } - - impl Task for TestTask { - fn to_toml(&self) -> String { - "".into() - } - fn from_toml(&self, d: String) -> Box { - return Box::new(self.clone()); - } - fn get_id(&self) -> (String, String) { - self.id.clone() - } - fn clone_box(&self) -> Box { - Box::new(self.clone()) - } - fn run_cpu(&mut self) { - self.value += 1; - } - fn to_bytes(&self) -> Vec { - postcard::to_allocvec(self).unwrap() - } - fn from_bytes(&self, bytes: &[u8]) -> Box { - Box::new(postcard::from_bytes::(bytes).unwrap()) - } - } - - // Test task execution directly - let mut task = TestTask { - value: 0, - id: ID("test", "test_task"), - }; - - task.run(Some(Runner::CPU)); - assert_eq!(task.value, 1); -} - -#[traced_test] -#[test] -fn test_task_serialization() { - use enginelib::task::{StoredTask, Task}; - use serde::{Deserialize, Serialize}; - - #[derive(Debug, Clone, Serialize, Deserialize, Verifiable)] - struct TestTask { - pub value: i32, - pub id: (String, String), - } - - impl Task for TestTask { - fn to_toml(&self) -> String { - "".into() - } - fn from_toml(&self, d: String) -> Box { - return Box::new(self.clone()); - } - fn get_id(&self) -> (String, String) { - self.id.clone() - } - fn clone_box(&self) -> Box { - Box::new(self.clone()) - } - fn run_cpu(&mut self) { - self.value += 1; - } - fn to_bytes(&self) -> Vec { - postcard::to_allocvec(self).unwrap() - } - fn from_bytes(&self, bytes: &[u8]) -> Box { - Box::new(postcard::from_bytes::(bytes).unwrap()) - } - } - - let task = TestTask { - value: 42, - id: ID("test", "test_task"), - }; - - // Test serialization and deserialization - let serialized = task.to_bytes(); - let stored_task = StoredTask { - bytes: serialized, - id: "id".into(), - }; - - // Deserialize - let deserialized_task: TestTask = postcard::from_bytes(&stored_task.bytes).unwrap(); - assert_eq!(deserialized_task.value, 42); - - // Test the from_bytes function - let recreated_task = task.from_bytes(&stored_task.bytes); - // We need a way to check the value inside the recreated task - // Since we can't directly access the value, we'll serialize it again and deserialize manually - let bytes = recreated_task.to_bytes(); - let final_task: TestTask = postcard::from_bytes(&bytes).unwrap(); - assert_eq!(final_task.value, 42); -} diff --git a/rfc/rfc1000.md b/rfc/rfc1000.md deleted file mode 100644 index 8ee5e89..0000000 --- a/rfc/rfc1000.md +++ /dev/null @@ -1,735 +0,0 @@ -# RFC 1000: Ergonomic use of the Event System - -## Summary - -Replace the current declarative macro system and manual `Event` trait impls with proc macros and explicit registration that works across both the core engine and dynamically loaded mods (`.so`/`.dll` via `libloading`). - -## Motivation - -The current system requires: -- Hand-written `impl Event for XxxEvent` blocks for every event -- Declarative macros (`RegisterAuthEventHandler!`, `RegisterAdminAuthEventHandler!`, etc.) that generate handler structs with `unsafe` pointer casts via `EventCTX` -- Manual `register_event!` calls to register default event instances into `EngineEventRegistry` -- A separate `Events` struct that serves only as a namespace for init and fire methods - -This is verbose, error-prone, and relies on `unsafe` downcasting. RFC 1000 replaces all of it with two proc macros (`#[derive(Event)]` and `#[event_handler]`). - -### Using `inventory` for per-module aggregation - -While `inventory` cannot be used to collect items **across** dynamic library boundaries (the engine cannot see a mod's inventory items), it **can** be used **within** a single compilation unit. Each mod uses `inventory` to collect its own handler registrations, and the `#[module]` macro automatically iterates over the module's inventory to register all handlers. - -This hybrid approach: -- **Eliminates boilerplate** for stateless handlers - no manual `__register_*` calls needed -- **Works with dynamic loading** - each mod manages its own inventory -- **Still supports stateful handlers** - modules manually register handlers that require context - -The `#[event_handler]` macro generates a registration function and submits it to the module's inventory. The `#[module]` macro injects a call to `register_inventory_handlers()` at the start of the module's `run` function, which iterates over all submitted registrars and registers them with the engine's event bus. - -Stateful handlers (those requiring context) cannot be auto-registered and still require explicit `__register_*` calls with the appropriate context. - -## Design - -### Defining Events - -Events can be defined and fired by both the core engine and individual mods. Simply create a struct and derive `Event`, specifying an appropriate namespace (e.g., `"core"` for engine events, or your mod's ID for mod events). - -```rust -#[derive(Clone, Debug, Event)] -#[event(namespace = "core", name = "auth_event")] -pub struct AuthEvent { - pub cancelled: bool, - pub id: Identifier, - pub uid: String, - pub challenge: String, - pub db: Db, - pub output: Arc>, -} -// Event trait is fully auto-implemented by the derive macro -``` - -### Stateless Handlers - -```rust -#[event_handler(namespace = "core", name = "auth_event")] -fn auth_handler(event: &mut AuthEvent) { - *event.output.write().unwrap() = true; -} -// Generates: -// struct AuthHandler; -// impl EventCTX for AuthHandler { fn expected_event_id() -> ("core", "auth_event"); fn handleCTX(&self, event: &mut AuthEvent) { ... } } -// impl EventHandler for AuthHandler { fn handle(&self, event: &mut dyn Event) { let event = Self::get_event(event); self.handleCTX(event); } } -// fn __register_auth_handler(event_bus: &mut EventBus) { ... } -// -// Note: EventCTX::get_event() uses unsafe casting with runtime event ID validation. -// This is necessary because TypeIds differ between separately-compiled crates. -``` - -### Stateful Handlers (with context) - -```rust -#[event_handler(namespace = "core", name = "admin_auth_event")] -fn admin_auth_handler(event: &mut AdminAuthEvent, token: &Arc) { - if token.as_str() == event.payload { - *event.output.write().unwrap() = true; - } -} -// Generates: -// struct AdminAuthHandler { ctx: Arc } -// impl AdminAuthHandler { fn with_ctx(ctx: Arc) -> Self { ... } } -// impl EventCTX for AdminAuthHandler { fn expected_event_id() -> ("core", "admin_auth_event"); fn handleCTX(&self, event: &mut AdminAuthEvent) { ... } } -// impl EventHandler for AdminAuthHandler { fn handle(&self, event: &mut dyn Event) { let event = Self::get_event(event); self.handleCTX(event); } } -// fn __register_admin_auth_handler(event_bus: &mut EventBus, ctx: Arc) { ... } -// -// Note: EventCTX::get_event() uses unsafe casting with runtime event ID validation. -// This is necessary because TypeIds differ between separately-compiled crates. -``` - -### Registering Handlers - -Each `#[event_handler]` invocation generates a module-level registration function. These functions are called explicitly to register handlers. - -#### Stateless-only (typical mod or core init): - -```rust -#[module] -fn run(api: &mut EngineAPI) { - __register_auth_handler(&mut api.event_bus); -} -``` - -Expands to: - -```rust -#[unsafe(export_name = "run")] -fn run(api: &mut EngineAPI) { - __register_auth_handler(&mut api.event_bus); - // ... one call per #[event_handler] in this crate -} -``` - -#### Stateful handlers require explicit context: - -```rust -#[module] -fn run(api: &mut EngineAPI) { - __register_auth_handler(&mut api.event_bus); - - // Stateful handlers are registered explicitly with context - let token = api.cfg.config_toml.cgrpc_token.clone().unwrap(); - __register_admin_auth_handler(&mut api.event_bus, Arc::new(token)); -} -``` - -### Cancellation - -Events opt into cancellation with the `cancellable` attribute. The `EventBus` skips remaining handlers when an event is cancelled. - -```rust -#[derive(Clone, Debug, Event)] -#[event(namespace = "core", name = "auth_event", cancellable)] -pub struct AuthEvent { ... } -``` - -Handlers can opt in to still receive cancelled events: - -```rust -#[event_handler(namespace = "core", name = "auth_event", receive_cancelled)] -fn audit_handler(event: &mut AuthEvent) { ... } -``` - -Updated dispatch loop: - -```rust -pub fn fire(&self, event: &mut T) { - let id = event.get_id(); - if let Some(handlers) = self.event_handler_registry.event_handlers.get(&id) { - for handler in handlers { - if event.is_cancelled() && !handler.receive_cancelled() { continue; } - handler.handle(event); - } - } -} -``` - -### Firing Events - -The convenience fire methods currently on `impl Events` move to associated functions on the event structs themselves: - -```rust -impl AuthEvent { - pub fn fire(api: &mut EngineAPI, uid: String, challenge: String, db: Db, output: Arc>) { - api.event_bus.fire(&mut AuthEvent { - cancelled: false, - id: ("core".to_string(), "auth_event".to_string()), - uid, - challenge, - db, - output, - }); - } - // Note: Events can be fired by core engine or mods - no pre-registration required -``` -```rust - - pub fn check(api: &mut EngineAPI, uid: String, challenge: String, db: Db) -> bool { - let output = Arc::new(RwLock::new(false)); - Self::fire(api, uid, challenge, db, output.clone()); - *output.read().unwrap() - } -} -``` - -## What gets removed - -### Structs and traits -- `pub struct Events {}` — empty namespace struct in `events/mod.rs` ! -- `pub trait EventCTX` — replaced with safer version that includes `expected_event_id()` and runtime ID validation before unsafe cast -- `pub struct EngineEventRegistry` — default-instance event registry in `event.rs` -- `impl Registry for EngineEventRegistry` — its `Registry` impl -- `impl Clone for Box` — only needed by the registry -- `event_registry` field on `EventBus` — no longer storing default event instances - -### Declarative macros -- `register_event!` in `macros.rs` — replaced by `#[derive(Event)]` proc macro -- `RegisterEventHandler!` in `macros.rs` — replaced by `#[event_handler]` proc macro -- `RegisterAuthEventHandler!` in `auth_event.rs` — replaced by `#[event_handler]` proc macro -- `RegisterAdminAuthEventHandler!` in `admin_auth_event.rs` — replaced by `#[event_handler]` proc macro -- `RegisterCgrpcEventHandler!` in `cgrpc_event.rs` — replaced by `#[event_handler]` proc macro - -### Manual trait impls -- `impl Event for AuthEvent` in `auth_event.rs` — replaced by `#[derive(Event)]` -- `impl Event for AdminAuthEvent` in `admin_auth_event.rs` — replaced by `#[derive(Event)]` -- `impl Event for CgrpcEvent` in `cgrpc_event.rs` — replaced by `#[derive(Event)]` -- `impl Event for StartEvent` in `start_event.rs` — replaced by `#[derive(Event)]` - -### `impl Events` blocks -- `Events::init` in `events/mod.rs` — event/handler registration becomes explicit initialization calls; task queue init moves to `EngineAPI::init` -- `Events::init_auth` in `events/mod.rs` — core auth handler auto-registers via `#[event_handler]`; admin auth handler registers in startup with context -- `Events::CheckAuth` / `Events::AuthEvent` — become `AuthEvent::check` / `AuthEvent::fire` -- `Events::CheckAdminAuth` / `Events::AdminAuthEvent` — become `AdminAuthEvent::check` / `AdminAuthEvent::fire` -- `Events::CgrpcEvent` — becomes `CgrpcEvent::fire` -- `Events::StartEvent` — becomes `StartEvent::fire` - -### What stays -- `pub trait Event` in `event.rs` — the trait definition stays, derive macro generates impls -- `pub trait EventHandler` in `event.rs` — stays as-is -- `pub struct EngineEventHandlerRegistry` + `register_handler` — stays, still the runtime handler store -- `pub struct EventBus` — stays with `event_handler_registry` field only, gains `fire()` with cancellation -- `pub fn ID()` in `events/mod.rs` — utility function, still useful -- Event struct definitions (`AuthEvent`, `AdminAuthEvent`, `CgrpcEvent`, `StartEvent`) — stay, gain `#[derive(Event)]` - -## Impl - -### 1. `#[derive(Event)]` — in `macros/src/lib.rs` - -**Important:** The `cancellable` attribute must be specified if the event struct has a `cancelled` field. The derive macro generates code that accesses `self.cancelled`, so the field must exist. - -```rust -#[proc_macro_derive(Event, attributes(event))] -pub fn derive_event(input: TokenStream) -> TokenStream { - let input = parse_macro_input!(input as DeriveInput); - let name = &input.ident; - - // parse #[event(namespace = "core", name = "auth_event", cancellable)] - let mut namespace = String::new(); - let mut event_name = String::new(); - let mut cancellable = false; - - for attr in &input.attrs { - if attr.path().is_ident("event") { - attr.parse_nested_meta(|meta| { - if meta.path.is_ident("namespace") { - let value = meta.value()?; - let lit: syn::LitStr = value.parse()?; - namespace = lit.value(); - } else if meta.path.is_ident("name") { - let value = meta.value()?; - let lit: syn::LitStr = value.parse()?; - event_name = lit.value(); - } else if meta.path.is_ident("cancellable") { - cancellable = true; - } - Ok(()) - }).unwrap(); - } - } - - let cancel_impl = if cancellable { - quote! { fn cancel(&mut self) { self.cancelled = true; } } - } else { - quote! { fn cancel(&mut self) { /* no-op */ } } - }; - - let is_cancelled_impl = if cancellable { - quote! { fn is_cancelled(&self) -> bool { self.cancelled } } - } else { - quote! { fn is_cancelled(&self) -> bool { false } } - }; - - quote! { - impl enginelib::event::Event for #name { - fn clone_box(&self) -> Box { Box::new(self.clone()) } - #cancel_impl - #is_cancelled_impl - fn get_id(&self) -> enginelib::Identifier { - (#namespace.to_string(), #event_name.to_string()) - } - fn as_any(&self) -> &dyn std::any::Any { self } - fn as_any_mut(&mut self) -> &mut dyn std::any::Any { self } - } - }.into() -} -``` - -### 2. `#[event_handler]` — in `macros/src/lib.rs` - -```rust -#[proc_macro_attribute] -pub fn event_handler(attr: TokenStream, item: TokenStream) -> TokenStream { - let item_fn = parse_macro_input!(item as syn::ItemFn); - let fn_name = &item_fn.sig.ident; - - // parse attrs: event = AuthEvent, namespace = "core", name = "auth_event", receive_cancelled - let mut event_type = None; - let mut ns = String::new(); - let mut ev_name = String::new(); - let mut receive_cancelled = false; - - let meta_parser = syn::meta::parser(|meta| { - if meta.path.is_ident("namespace") { - ns = meta.value()?.parse::()?.value(); - } else if meta.path.is_ident("name") { - ev_name = meta.value()?.parse::()?.value(); - } else if meta.path.is_ident("receive_cancelled") { - receive_cancelled = true; - } - Ok(()) - }); - parse_macro_input!(attr with meta_parser); - - // extract the event param type (e.g. &mut AuthEvent -> AuthEvent) - let event_type = match &item_fn.sig.inputs[0] { - syn::FnArg::Typed(pat_type) => { - match &*pat_type.ty { - syn::Type::Reference(type_ref) => match &*type_ref.elem { - syn::Type::Path(type_path) => type_path.path.segments.last().unwrap().ident.clone(), - _ => panic!("Expected path type for event"), - }, - _ => panic!("Expected reference type for event"), - } - } - _ => panic!("Expected typed argument for event"), - }; - - let handler_name_str = fn_name.to_string().split('_').map(|s| { - let mut c = s.chars(); - match c.next() { - None => String::new(), - Some(f) => f.to_uppercase().collect::() + c.as_str(), - } - }).collect::(); - let handler_name = syn::Ident::new(&format!("{}Handler", handler_name_str), fn_name.span()); - let register_fn_name = format_ident!("__register_{}", fn_name); - - let is_stateful = item_fn.sig.inputs.len() > 1; // more than just &mut Event - - let receive_cancelled_impl = if receive_cancelled { - quote! { fn receive_cancelled(&self) -> bool { true } } - } else { - quote! { fn receive_cancelled(&self) -> bool { false } } - }; - - if is_stateful { - // extract the context param type (e.g. &Arc -> Arc) - let ctx_type = match &item_fn.sig.inputs[1] { - syn::FnArg::Typed(pat_type) => { - match &*pat_type.ty { - syn::Type::Reference(type_ref) => &*type_ref.elem, - _ => &*pat_type.ty, - } - } - _ => panic!("Expected typed argument for context"), - }; - - quote! { - #item_fn - - pub struct #handler_name { - ctx: #ctx_type, - } - - impl #handler_name { - pub fn with_ctx(ctx: #ctx_type) -> Self { - Self { ctx } - } - } - - impl enginelib::event::EventCTX<#event_type> for #handler_name { - fn expected_event_id() -> enginelib::Identifier { - (#ns.to_string(), #ev_name.to_string()) - } - fn handleCTX(&self, event: &mut #event_type) { - #fn_name(event, &self.ctx); - } - } - - impl enginelib::event::EventHandler for #handler_name { - fn handle(&self, event: &mut dyn enginelib::event::Event) { - let event = >::get_event(event); - #fn_name(event, &self.ctx); - } - #receive_cancelled_impl - } - - fn #register_fn_name(event_bus: &mut enginelib::event::EventBus, ctx: #ctx_type) { - event_bus.register_handler( - #handler_name::with_ctx(ctx), - (#ns.to_string(), #ev_name.to_string()), - ); - } - }.into() - } else { - quote! { - #item_fn - - pub struct #handler_name; - - impl enginelib::event::EventCTX<#event_type> for #handler_name { - fn expected_event_id() -> enginelib::Identifier { - (#ns.to_string(), #ev_name.to_string()) - } - fn handleCTX(&self, event: &mut #event_type) { - #fn_name(event); - } - } - - impl enginelib::event::EventHandler for #handler_name { - fn handle(&self, event: &mut dyn enginelib::event::Event) { - let event = >::get_event(event); - #fn_name(event); - } - #receive_cancelled_impl - } - - fn #register_fn_name(event_bus: &mut enginelib::event::EventBus) { - event_bus.register_handler( - #handler_name, - (#ns.to_string(), #ev_name.to_string()), - ); - } - }.into() - } -} -``` - -### 3. Registering Handlers - -There is no `register_handlers!` macro. The `#[event_handler]` macro generates a public `__register_` function. Mod authors and the core engine call them explicitly during initialization. - -```rust -#[module] -fn run(api: &mut EngineAPI) { - __register_auth_handler(&mut api.event_bus); - __register_on_start(&mut api.event_bus); - __register_admin_auth_handler(&mut api.event_bus, Arc::new(token)); -} -``` - -This convention-based approach is simple, robust, and completely compatible with dynamic mod loading. - -### 4. `EventCTX` trait with safe unsafe cast — in `enginelib/src/event.rs` - -Due to TypeId mismatches across separately-compiled crates (engine vs dynamically loaded mods), safe `downcast_mut()` cannot work. Each crate gets its own compilation of types, resulting in different TypeIds even for identical structs. - -The `EventCTX` trait uses an unsafe cast validated by event ID matching: - -```rust -pub trait EventCTX: EventHandler { - fn expected_event_id() -> Identifier; - - fn get_event(event: &mut dyn Event) -> &mut C { - // Runtime validation: ensure event ID matches before unsafe cast - assert_eq!( - event.get_id(), - Self::expected_event_id(), - "Event ID mismatch: handler expects {:?}, got {:?}", - Self::expected_event_id(), - event.get_id() - ); - // Safety: Event ID verified, type is correct - unsafe { &mut *(event as *mut dyn Event as *mut C) } - } - - fn handle(&self, event: &mut dyn Event) { - let event: &mut C = Self::get_event(event); - self.handleCTX(event); - } - - fn handleCTX(&self, event: &mut C); -} -``` - -### 5. `EventHandler` trait — in `enginelib/src/event.rs` - -```rust -pub trait EventHandler: Any + Send + Sync { - fn handle(&self, event: &mut dyn Event); - fn receive_cancelled(&self) -> bool { false } -} -``` - -### 6. Updated `EventBus` — in `enginelib/src/event.rs` - -`EngineEventRegistry` is removed — events no longer need pre-registration. - -```rust -pub struct EventBus { - pub event_handler_registry: EngineEventHandlerRegistry, -} - -impl EventBus { - pub fn register_handler( - &mut self, - handler: H, - identifier: Identifier, - ) { - self.event_handler_registry.register_handler(handler, identifier); - } - - #[instrument] - pub fn fire(&self, event: &mut T) { - let id = event.get_id(); - debug!("EventBus: Firing event {}.{}", id.0, id.1); - - if let Some(handlers) = self.event_handler_registry.event_handlers.get(&id) { - for handler in handlers { - if event.is_cancelled() && !handler.receive_cancelled() { - continue; - } - handler.handle(event); - } - } else { - debug!("EventBus: No handlers for event {}.{}", id.0, id.1); - } - } -} -``` - -## Full example: core engine startup - -**Note:** The `#[event_handler]` macro generates an `EventCTX` implementation with unsafe casting validated by event ID. This is necessary because TypeIds differ across separately-compiled crates (engine vs mods loaded via `libloading`). - -```rust -// enginelib/src/events/auth_event.rs - -#[derive(Clone, Debug, Event)] -#[event(namespace = "core", name = "auth_event", cancellable)] -pub struct AuthEvent { - pub cancelled: bool, - pub id: Identifier, - pub uid: String, - pub challenge: String, - pub db: Db, - pub output: Arc>, -} -// Event trait auto-implemented by derive macro - -impl AuthEvent { - pub fn fire(api: &mut EngineAPI, uid: String, challenge: String, db: Db, output: Arc>) { - api.event_bus.fire(&mut AuthEvent { - cancelled: false, - id: ("core".to_string(), "auth_event".to_string()), - uid, - challenge, - db, - output, - }); - } - - pub fn check(api: &mut EngineAPI, uid: String, challenge: String, db: Db) -> bool { - let output = Arc::new(RwLock::new(false)); - Self::fire(api, uid, challenge, db, output.clone()); - *output.read().unwrap() - } -} - -#[event_handler(namespace = "core", name = "auth_event")] -fn auth_handler(event: &mut AuthEvent) { - *event.output.write().unwrap() = true; -} -// Generates EventCTX impl with unsafe cast validated by event ID -``` - -```rust -// enginelib/src/events/admin_auth_event.rs - -#[derive(Clone, Debug, Event)] -#[event(namespace = "core", name = "admin_auth_event", cancellable)] -pub struct AdminAuthEvent { - pub cancelled: bool, - pub id: Identifier, - pub payload: String, - pub target: Identifier, - pub db: Db, - pub output: Arc>, -} -// Event trait auto-implemented by derive macro - -impl AdminAuthEvent { - pub fn fire(api: &mut EngineAPI, payload: String, target: Identifier, db: Db, output: Arc>) { - api.event_bus.fire(&mut AdminAuthEvent { - cancelled: false, - id: ("core".to_string(), "admin_auth_event".to_string()), - payload, - target, - db, - output, - }); - } - - pub fn check(api: &mut EngineAPI, payload: String, target: Identifier, db: Db) -> bool { - let output = Arc::new(RwLock::new(false)); - Self::fire(api, payload, target, db, output.clone()); - *output.read().unwrap() - } -} - -#[event_handler(namespace = "core", name = "admin_auth_event")] -fn admin_auth_handler(event: &mut AdminAuthEvent, token: &Arc) { - if token.as_str() == event.payload { - *event.output.write().unwrap() = true; - } -} -// Generates EventCTX impl with unsafe cast validated by event ID -``` - -```rust -// Core engine startup (e.g. in EngineAPI::init or server.rs) - -pub fn init(api: &mut EngineAPI) { - // Task queue init (moved from Events::init — not event-related) - for (id, _tsk) in api.task_registry.tasks.iter() { - api.task_queue.tasks.entry(id.clone()).or_default(); - api.executing_tasks.tasks.entry(id.clone()).or_default(); - api.solved_tasks.tasks.entry(id.clone()).or_default(); - } - - // Register stateless core handlers - __register_auth_handler(&mut api.event_bus); - - // Register stateful core handlers - let token = api.cfg.config_toml.cgrpc_token.clone(); - if let Some(token) = token { - __register_admin_auth_handler(&mut api.event_bus, Arc::new(token)); - } else { - // No token configured — accept all admin auth - api.event_bus.register_handler( - AdminAuthAcceptAll, - ("core".to_string(), "admin_auth_event".to_string()), - ); - } -} -``` - -## Full example: a mod - -```rust -// my_mod/src/lib.rs -use enginelib::prelude::*; - -#[derive(Clone, Debug, Event)] -#[event(namespace = "my_mod", name = "custom_event")] -pub struct MyCustomEvent { - pub cancelled: bool, - pub id: Identifier, - pub payload: String, -} - -impl MyCustomEvent { - pub fn fire(mut self, bus: &mut EventBus) -> Result { - bus.fire(&mut self); - self.check() - } - pub fn check(self) -> Result { - if self.cancelled { - Err(self) - } else { - Ok(self) - } - } -} - -#[event_handler(namespace = "core", name = "start_event")] -fn on_start(event: &mut StartEvent) { - info!("My mod loaded! {} modules active.", event.modules.len()); - - // Mods can define and fire their own events - let mut custom = MyCustomEvent { - cancelled: false, - id: ("my_mod".to_string(), "custom_event".to_string()), - payload: "Hello from my mod".to_string(), - }; - if let Ok(_) = custom.fire(&mut event.api.event_bus) { - info!("Custom event fired successfully"); - } -} - -#[event_handler(namespace = "core", name = "cgrpc_event")] -fn on_cgrpc(event: &mut CgrpcEvent) { - if event.handler_id == ("my_mod".to_string(), "echo".to_string()) { - *event.output.write().unwrap() = event.payload.clone(); - } -} - -#[metadata] -fn metadata() -> LibraryMetadata { - LibraryMetadata { - mod_id: "my_mod".to_string(), - mod_name: "My Mod".to_string(), - ..Default::default() - } -} - -#[module] -fn run(api: &mut EngineAPI) { - // Registers on_start and on_cgrpc handlers - __register_on_start(&mut api.event_bus); - __register_on_cgrpc(&mut api.event_bus); -} -``` - -This works identically whether the mod is compiled into the engine binary or loaded dynamically as a `.so`/`.dll` via `libloading`, because registration is explicit function calls through `&mut EngineAPI` — no linker magic required. - -**Important:** The `#[event_handler]` macro generates an `EventCTX` implementation that uses unsafe casting with runtime event ID validation. This is necessary because TypeIds differ between the engine and dynamically loaded mods — each compiles its own copy of types even when depending on the same `enginelib` source. - -## Why Unsafe Casting is Necessary - -Rust's `TypeId` is computed per **compilation unit**, not per source code. When the engine and a mod each compile their own copy of `enginelib` (even from identical source), they get different TypeIds: - -``` -engine binary → compiles enginelib → TypeId(0x33b8d3855f222018...) -mod.so → compiles enginelib → TypeId(0x724cac8ae3425191...) ← DIFFERENT! -``` - -This means safe `downcast_mut::()` **always fails** across dynamic library boundaries because the TypeId check fails. - -The `EventCTX` trait works around this by: -1. Storing the expected event ID as a string tuple `(namespace, name)` -2. Validating the runtime event's ID matches before casting -3. Using an unsafe pointer cast (sound because we verified the ID) - -This is **safe enough**: if the event ID doesn't match, we panic with a clear error message instead of silently corrupting memory. - -## Testing This Yourself - -```bash -# TypeIds MATCH when using workspace-shared crate -cd /tmp/dlopen_test && cargo run --release -# Output: Match: true - -# TypeIds DON'T MATCH when crates are compiled separately -cd /tmp/separate_test && cargo run --release -# Output: Match: false -``` - -The second scenario is what happens with dynamically loaded mods — each mod is a separate Cargo project compiling its own copy of `enginelib`, resulting in different TypeIds. diff --git a/rfc/rfc1001.md b/rfc/rfc1001.md deleted file mode 100644 index dd45f9c..0000000 --- a/rfc/rfc1001.md +++ /dev/null @@ -1,92 +0,0 @@ -# RFC 1001: Expanded `enginelib::prelude` Re-exports - -## Summary - -Expand `enginelib::prelude` to re-export the commonly used traits/macros/types needed when writing engine code and mods, so most consumers can start with: - -```rust -use enginelib::prelude::*; -``` - -…and avoid repeated “bring trait into scope” imports. - -## Motivation - -Today, users frequently need to import traits (and a few macros/types) explicitly to make code compile, especially: - -- Trait method calls that require the trait in scope (`Task`, `Registry`, `EventHandler`, etc.) -- `#[derive(Verifiable)]` generating `impl Verifiable for T` (currently requires the `Verifiable` trait to be in scope) -- Common logging macros (`debug!`, `info!`, `warn!`, `error!`, `instrument`) -- Core engine/mod entry macros (`#[module]`, `#[metadata]`, `#[event_handler]`, `#[derive(Event)]`) - -This is noisy boilerplate in every mod and in many internal modules. - -## Design - -### Prelude contents (curated) - -`enginelib::prelude` should re-export a curated set of items that are: - -- “Always used” or “used in most files” for mods -- Low-risk to re-export (stable names, low likelihood of conflicts) -- Primarily traits and macros (the current pain point) - -Proposed re-exports (names shown as they would be available via `use enginelib::prelude::*;`): - -- **Proc macros (from `enginelib/macros`)** - - `Event` (derive) - - `Verifiable` (derive) - - `event_handler` (attribute) - - `module` (attribute) - - `metadata` (attribute) - - Keep existing `macros` re-export for compatibility (`prelude::macros::...`) -- **Core traits** - - `Registry` - - `task::{Task, Verifiable}` - - `event::{Event, EventHandler, EventCTX}` -- **Core types used by mods** - - `Identifier` - - `api::EngineAPI` - - `event::EventBus` - - `plugin::LibraryMetadata` - - Core event structs: `StartEvent`, `CgrpcEvent` (and optionally `AuthEvent`, `AdminAuthEvent`) -- **Logging macros** - - `tracing::{debug, info, warn, error, instrument}` - -The intent is that a typical mod can compile with only: - -```rust -use enginelib::prelude::*; -``` - -### Non-goals - -- Re-exporting “everything” from enginelib. -- Replacing explicit imports in performance-critical or clarity-critical code where local imports are preferred. -- Making `prelude` a hard dependency (it remains optional sugar). - -## Backwards compatibility - -This RFC is **additive**: - -- Existing `enginelib::prelude` exports remain available. -- New exports only reduce required imports; they do not change behavior. - -Potential downside: wildcard-prelude imports can increase the likelihood of name conflicts in user code. This is common for preludes and can be mitigated by opting out (import explicitly) or using qualified paths. - -## Alternatives - -1. **Fix the proc-macro expansions to use fully-qualified paths** (e.g. generate `impl ::enginelib::task::Verifiable for T`). - - Pros: removes the “trait must be in scope” requirement at the source. - - Cons: doesn’t address other recurring imports (logging macros, event/task traits, etc.). - -2. **Document a recommended import block** instead of a prelude. - - Pros: zero API surface changes. - - Cons: still boilerplate; users copy/paste blocks everywhere. - -This RFC prefers an expanded prelude (and may still adopt fully-qualified macro expansions later for extra ergonomics). - -## Implementation plan - -1. Update `enginelib/src/prelude.rs` to `pub use` the curated set above (explicitly, not globbed). -2. Update examples/docs (`rfc/rfc1000.md`) to rely on `use enginelib::prelude::*;` consistently (no extra `use macros::...`). diff --git a/rfc/rfc1002.md b/rfc/rfc1002.md deleted file mode 100644 index 3c0b7d3..0000000 --- a/rfc/rfc1002.md +++ /dev/null @@ -1,96 +0,0 @@ -# RFC 1002: Rustforge Mod Package Format (.rf) - -## Summary - -This RFC defines the **mod package file format** used by GE Engine. Mods are distributed as a single file with the extension **`.rf`**, placed in the engine’s `./mods/` directory. - -`.rf` replaces the legacy `*.rustforge.tar` naming convention. The engine loads **only** `.rf` packages. - -## Discovery - -- The engine MUST scan the `./mods/` directory at startup. -- The engine MUST treat any file ending in `.rf` as a mod package candidate. -- The engine MUST ignore any non-`.rf` files (including `*.rustforge.tar`). - -## Container Format - -- A `.rf` file MUST be an **uncompressed tar archive**. - -### Archive safety constraints - -To avoid path traversal and unsafe extraction behavior: - -- All archive entries MUST be **relative** paths. -- Archive entries MUST NOT contain `..` path segments. -- Archive entries SHOULD NOT be absolute paths. - -## Required Layout - -A `.rf` archive MUST contain the platform-specific dynamic library at the archive root: - -- **Linux**: `mod.so` -- **Windows**: `mod.dll` -- **macOS**: `mod.dylib` - -Additional files MAY be present and are currently ignored by the engine runtime. - -## Module ABI (Exports) - -The dynamic library inside the `.rf` archive MUST export the following symbols: - -### `metadata` - -- Symbol name: `metadata` -- Signature: `unsafe extern "Rust" fn() -> enginelib::plugin::LibraryMetadata` - -### `run` - -- Symbol name: `run` -- Signature: `unsafe extern "Rust" fn(api: &mut enginelib::api::EngineAPI)` - -The recommended authoring path is to use the existing proc macros: - -- `#[metadata]` to export `metadata` -- `#[module]` to export `run` - -## Compatibility Checks (Strict Mode Only) - -`LibraryMetadata` includes the fields: - -- `api_version` -- `rustc_version` - -When strict mode is enabled, the engine MUST reject a module when either field does not match the engine build: - -- `metadata.api_version != enginelib::GIT_VERSION` -- `metadata.rustc_version != enginelib::RUSTC_VERSION` - -Strict mode is enabled when the environment variable `GE_STRICT_MODS` is set (to any value). If `GE_STRICT_MODS` is not set, these fields are informational and MUST NOT cause the module to be rejected. - -## Errors - -The engine MUST fail to load a mod package when: - -- The required `mod.*` file for the current platform is missing. -- The required exported symbols (`metadata`, `run`) are missing. - -## Examples - -Example `.rf` tar contents (Linux): - -``` -mod.so -assets/icon.png -README.txt -``` - -Example `.rf` tar contents (macOS): - -``` -mod.dylib -``` - -## Migration Note - -Mods previously distributed as `*.rustforge.tar` MUST be republished as `*.rf` for the engine to load them. - diff --git a/rfc/rfc1003.md b/rfc/rfc1003.md deleted file mode 100644 index 97b04d2..0000000 --- a/rfc/rfc1003.md +++ /dev/null @@ -1,152 +0,0 @@ -# RFC 1003: Minimal Client Event Hooks for Modding - -## Summary - -Introduce a small client-side event surface for modding, limited to **6 hooks** in v1: - -1. `client:start` -2. `client:auth_prepare` -3. `client:before_task_acquire` *(cancellable)* -4. `client:task_acquired` -5. `client:before_task_execute` *(cancellable)* -6. `client:before_task_publish` *(cancellable)* - -This RFC follows the same event model already used by the engine (`#[derive(Event)]`, `#[event_handler]`, `EventBus`) and avoids adding broader lifecycle/noise events until real usage requires them. - -## Motivation - -Client mods need a few strategic control points, not a full telemetry matrix. - -The current client loop has five meaningful boundaries: - -- startup -- auth metadata construction -- acquire decision -- execute decision -- publish decision - -Adding more hooks now (e.g. metadata/shutdown/retry/error families) increases API surface and maintenance cost without proven need. - -## Design - -### Event namespace - -All new hooks are in namespace `"client"`. - -### Event list and semantics - -#### `client:start` -Fired once after client initialization and before entering the long-running loop. - -- cancellable: **no** -- intent: mod bootstrap, cache warm-up, one-time setup - -#### `client:auth_prepare` -Fired before outbound RPCs so handlers can attach request auth/identity metadata. - -- cancellable: **no** -- intent: token/header injection - -#### `client:before_task_acquire` -Fired before `AquireTask` request for a `task_id`. - -- cancellable: **yes** -- if cancelled: skip acquire attempt for this cycle - -#### `client:task_acquired` -Fired after successful `AquireTask` and before execution. - -- cancellable: **no** -- intent: payload inspection/validation/annotation - -#### `client:before_task_execute` -Fired before `run_hip()` / `run_cpu()`. - -- cancellable: **yes** -- if cancelled: do not execute task in this cycle - -#### `client:before_task_publish` -Fired before `PublishTask`. - -- cancellable: **yes** -- if cancelled: do not publish in this cycle (policy may defer/retry/drop) - -### Why these six - -They map directly to control boundaries in the existing client runtime and are sufficient for most policy/auth/routing mods. - -Anything beyond these should be added only with concrete use-cases. - -## Integration points - -Primary file: `engine/src/bin/client.rs` - -Insert event firing at: - -1. After `EngineAPI::init_client(&mut api)` -> `client:start` -2. Request metadata/interceptor path -> `client:auth_prepare` -3. Before each `aquire_task(...)` call -> `client:before_task_acquire` -4. Immediately after successful `aquire_task(...)` -> `client:task_acquired` -5. Immediately before `task.run_hip()` -> `client:before_task_execute` -6. Immediately before `publish_task(...)` -> `client:before_task_publish` - -## Event definitions (shape) - -Proposed event structs follow existing conventions: - -```rust -#[derive(Clone, Debug, Event)] -#[event(namespace = "client", name = "before_task_execute", cancellable)] -pub struct BeforeTaskExecuteEvent { - pub cancelled: bool, - pub id: Identifier, - pub task_id: String, - pub instance_id: String, - pub payload: Vec, -} -``` - -Non-cancellable events omit behavior (or keep `cancelled` field with non-cancellable derive usage, consistent with current style). - -## Cancellation policy - -- `before_task_acquire.cancelled == true` -> skip acquire attempt -- `before_task_execute.cancelled == true` -> skip execution -- `before_task_publish.cancelled == true` -> skip publish attempt - -No cancellation should panic; cancellation is treated as expected control flow. - -## Out of scope (deferred) - -The following are explicitly deferred until needed: - -- `client:shutdown` -- `client:metadata_received` -- `client:after_task_execute` -- retry/error-classified events -- heartbeat/status events - -## Compatibility - -Additive only: - -- No breaking changes to core/server event APIs. -- No behavior change when no handlers are registered. - -## Implementation plan - -1. Add new client event modules under `enginelib/src/events/`. -2. Export through `enginelib/src/events/mod.rs` and `enginelib/src/prelude.rs`. -3. Fire hooks in `engine/src/bin/client.rs` at integration points listed above. -4. Add tests validating cancellable behavior preserves loop stability. - -## Acceptance criteria - -- Exactly the 6 events above are implemented in v1. -- 3 cancellable events honor cancellation semantics. -- Client still behaves as today with zero handlers. -- `cargo test --workspace` passes. - ---- - -Last updated: 2026-04-17 diff --git a/rfc/rfc1004.md b/rfc/rfc1004.md deleted file mode 100644 index 0632dfd..0000000 --- a/rfc/rfc1004.md +++ /dev/null @@ -1,149 +0,0 @@ -# RFC 1004: Server Lifecycle Events for Modding - -## Summary - -Introduce a first-class **server event surface** for modding, following the same event model already adopted by core/events in RFC 1000. - -This RFC defines a **tiered rollout**, with only **Tier 1** intended for immediate implementation: - -- `server:start` -- `server:before_task_create` *(cancellable)* -- `server:task_created` -- `server:before_task_acquire` *(cancellable)* -- `server:task_acquired` -- `server:before_task_publish` *(cancellable)* -- `server:task_published` - -## Motivation - -Today, server mod hooks are concentrated around auth/cgrpc/start. Task lifecycle transitions (`create -> acquire -> publish`) are implemented directly in `server.rs` with no dedicated server-namespace hooks. - -That makes policy extensions harder than they need to be: - -- no clean interception point before creation/acquisition/publish, -- no canonical post-success signal for metrics/audit, -- mod authors must patch/replace behavior instead of composing handlers. - -RFC 1004 adds explicit lifecycle hooks while keeping the initial scope small. - -## Design - -### Namespace and conventions - -All events in this RFC use namespace `"server"` and follow the existing conventions: - -- event structs derive `Event` via proc macro, -- handlers use `#[event_handler(...)]`, -- firing goes through `EventBus` via helper methods in `events/mod.rs`. - -### Tier 1 events (Phase A) - -#### `server:start` -Fired once during startup after API init and before serving requests. - -- cancellable: no -- purpose: startup bootstrap/observability - -#### `server:before_task_create` *(cancellable)* -Fired before a create request is accepted and queued. - -- cancellable: yes -- expected context: `task_id`, mutable payload bytes -- on cancel: reject request - -#### `server:task_created` -Fired after queue insertion succeeds. - -- cancellable: no -- expected context: `task_id`, instance id, payload snapshot - -#### `server:before_task_acquire` *(cancellable)* -Fired before dequeue/assignment in `aquire_task`. - -- cancellable: yes -- expected context: requester identity (`uid`), `task_id` -- on cancel: abort acquire attempt - -#### `server:task_acquired` -Fired after assignment succeeds. - -- cancellable: no -- expected context: `uid`, `task_id`, instance id - -#### `server:before_task_publish` *(cancellable)* -Fired before publish is accepted and transition to solved is committed. - -- cancellable: yes -- expected context: `uid`, `task_id`, instance id, mutable payload bytes -- on cancel: reject publish request - -#### `server:task_published` -Fired after solved transition succeeds. - -- cancellable: no -- expected context: `uid`, `task_id`, instance id - -### Cancellation semantics - -Default status on cancellation in Tier 1 is: - -```rust -tonic::Status::aborted("...cancelled by server event handler") -``` - -Cancellation is considered valid control flow (not an internal error). - -## Integration points (current code) - -Primary file: `engine/src/bin/server.rs`. - -Hooks map to: - -1. `main()` startup flow -> `server:start` -2. `create_task(...)` pre-validation/pre-queue -> `server:before_task_create` -3. `create_task(...)` post-queue success -> `server:task_created` -4. `aquire_task(...)` pre-dequeue -> `server:before_task_acquire` -5. `aquire_task(...)` post-assignment -> `server:task_acquired` -6. `publish_task(...)` pre-verify/pre-transition -> `server:before_task_publish` -7. `publish_task(...)` post-solved success -> `server:task_published` - -Note: method name `aquire_task` uses current in-tree spelling. - -## What gets added - -- New server event structs under `enginelib/src/events/` for Tier 1. -- New helper fire/check methods in `enginelib/src/events/mod.rs`. -- Re-exports in `enginelib/src/prelude.rs`. -- Event firing calls in server lifecycle boundaries. - -## What stays unchanged - -- gRPC schema (`engine.proto`) remains unchanged. -- Existing auth event contracts remain unchanged. -- Behavior without registered handlers should remain equivalent. - -## Implementation plan (Phase A) - -1. Add Tier 1 event modules in `enginelib/src/events/`. -2. Add `Events::Server*` helpers in `events/mod.rs`. -3. Export new types in `prelude.rs`. -4. Wire call sites in `server.rs`. -5. Add tests for cancellation behavior and no-handler parity. -6. Run `cargo test --workspace`. - -## Acceptance criteria - -- Tier 1 events exist and are fired at the integration points above. -- `before_*` hooks are cancellable and return `aborted` by default. -- No-handler behavior remains effectively unchanged. -- Workspace tests pass. - -## Open questions - -1. Should cancellation reasons be standardized (code + message convention)? -2. Should `before_task_acquire` remain metadata-only, or allow payload mutation in future? -3. Should Tier 2 include auth success events, or only failures? - ---- - -Last updated: 2026-04-17 diff --git a/rfc/rfc1005.md b/rfc/rfc1005.md deleted file mode 100644 index 19e510b..0000000 --- a/rfc/rfc1005.md +++ /dev/null @@ -1,14 +0,0 @@ -# RFC 1005 -Rewrite Enginelib and Engine to be Fast - -## New Data Model -TaskQueue Stores x full tasks Ready to give in an arc - - - - - -## Todo Settings -- [ ] RAM only Leasing -- [ ] TaskQueue only loads ids and not full data -- [ ] No taskqueue, read from disk