From 52000c62ade16c64797d47f306903399750f7f34 Mon Sep 17 00:00:00 2001 From: "Yoshiaki Ueda (bootjp)" Date: Fri, 22 May 2026 16:06:07 +0900 Subject: [PATCH 1/3] docs/design: propose admin web UI for DynamoDB item and S3 object CRUD Extends the admin Web Console with item-level CRUD for DynamoDB and object-level CRUD for S3, mirroring the SQS Messages tab pattern PR #798 established. Scope per user discussion: - DynamoDB: basic CRUD only (Scan/Get/Put/Delete); no Query, GSI, conditional updates, or partial updates in this round. - S3: prefix + delimiter listings (pseudo-directory tree), Get (download), Put (upload, 100 MiB cap, raw octet-stream), Delete. - Read-side calls gated by AllowsRead; write-side by AllowsWrite, with the action-verb parameter introduced in the SQS Phase 4 fix carrying through. - Audit + Prometheus counters scoped to writes only (read paths poll-heavy). Six rollout phases; Phase 2 (backend RPCs) can split per adapter if review surface gets large. --- .../2026_05_22_proposed_admin_data_browser.md | 297 ++++++++++++++++++ 1 file changed, 297 insertions(+) create mode 100644 docs/design/2026_05_22_proposed_admin_data_browser.md diff --git a/docs/design/2026_05_22_proposed_admin_data_browser.md b/docs/design/2026_05_22_proposed_admin_data_browser.md new file mode 100644 index 00000000..c74bb7d0 --- /dev/null +++ b/docs/design/2026_05_22_proposed_admin_data_browser.md @@ -0,0 +1,297 @@ +# Admin Web UI for DynamoDB Item and S3 Object CRUD + +**Status:** Proposed +**Author:** bootjp +**Date:** 2026-05-22 + +--- + +## 1. Background and Motivation + +The admin Web Console (`web/admin/`) now ships per-adapter detail pages for SQS (`SqsDetail.tsx`), DynamoDB (`DynamoDetail.tsx`), and S3 (`S3Detail.tsx`). All three pages handle resource-level operations (create / describe / delete the queue, table, or bucket) and SQS additionally ships message-level operations (peek + purge) after the 2026-05-16 admin-purge-queue work landed. + +DynamoDB and S3 detail pages still stop at the resource level — they show table / bucket metadata but cannot inspect or modify what is **inside** the table / bucket. Two operator workflows are blocked: + +1. **Inspect what is actually stored.** An operator triaging a misbehaving service needs to look at the items / objects the service is reading and writing. Today they have to drop to the AWS CLI or `cmd/elastickv-admin` and re-authenticate against whatever credentials store backs that path. + +2. **Apply ad-hoc fixes.** A bad config blob, a stuck queue's parking-lot record, a stale routing entry, a corrupted DLQ message — operators need an in-Console way to delete or replace a small number of items / objects without writing a one-off script. + +This proposal extends the admin Web Console to support **List / Get / Put / Delete** on DynamoDB items and S3 objects, following the same pattern the SQS Messages tab established. + +--- + +## 2. Goals and Non-Goals + +### 2.1 Goals + +1. **DynamoDB item CRUD** — `AdminScanTable`, `AdminGetItem`, `AdminPutItem`, `AdminDeleteItem` admin RPCs on `*adapter.DynamoDBServer`. List uses `Scan` semantics (no `Query` / GSI / conditional ops in this phase). Cursor pagination via `LastEvaluatedKey`. +2. **S3 object CRUD** — `AdminListObjects`, `AdminGetObject`, `AdminPutObject`, `AdminDeleteObject` admin RPCs on `*adapter.S3Server`. List supports `prefix` + `delimiter` so the SPA can render the bucket as a pseudo-directory tree. +3. **HTTP endpoints** — + - `GET /admin/api/v1/dynamo/tables/{name}/items?cursor=C&limit=N` — Scan + - `GET /admin/api/v1/dynamo/tables/{name}/items/{key}` — Get item by primary key + - `PUT /admin/api/v1/dynamo/tables/{name}/items/{key}` — Put item + - `DELETE /admin/api/v1/dynamo/tables/{name}/items/{key}` — Delete item + - `GET /admin/api/v1/s3/buckets/{name}/objects?prefix=P&delimiter=D&continuation_token=T` — List + - `GET /admin/api/v1/s3/buckets/{name}/objects/{key}` — Get (download) + - `PUT /admin/api/v1/s3/buckets/{name}/objects/{key}` — Put (upload) + - `DELETE /admin/api/v1/s3/buckets/{name}/objects/{key}` — Delete +4. **SPA UI**: + - **DynamoDetail Items tab**: paginated table of items, click for full JSON modal, Edit / Delete buttons per row, "Add item" button at the top. + - **S3Detail Objects tab**: tree view (folders from `delimiter=/` collapses), Download button per row, Delete per row, "Upload" button at the top. +5. **100 MiB upload cap** for S3 PutObject; raw octet-stream body (not JSON-encoded). The DynamoDB PutItem stays JSON because items are small (DynamoDB's own item-size limit is 400 KiB). +6. **Read role can browse, write role can mutate** — peek-equivalent gate (`AllowsRead`) on every Get/List, write gate (`AllowsWrite`) on every Put/Delete. Mirrors the SQS Phase 4 division. +7. **Audit lines** — `admin.dynamo.put_item`, `admin.dynamo.delete_item`, `admin.s3.put_object`, `admin.s3.delete_object` with subject + role + table/bucket + key. + +### 2.2 Non-Goals + +1. **DynamoDB Query / GSI / conditional updates / partial updates (`UPDATE` action).** Scope decision: basic CRUD only. A future RFC can add Query when operators ask. +2. **DynamoDB UpdateItem (PATCH semantics).** Put-full-item replaces the whole record; partial mutations are out of scope. +3. **S3 multipart upload.** A single PUT handles up to 100 MiB; anything larger needs the AWS SDK against the public endpoint. +4. **S3 versioning operations.** Even though the underlying engine supports MVCC, the admin UI surfaces only the current version. Listing prior versions belongs to a future Versioning UI. +5. **Batch operations** (BatchGetItem, BatchWriteItem, DeleteObjects). One key per request keeps the audit shape simple and the failure handling unambiguous; bulk operations belong to a follow-up. +6. **Cross-table / cross-bucket views.** The SPA already lists tables / buckets at the index page; per-resource detail pages cover the rest. +7. **Server-side filtering on Scan** (FilterExpression). Adds wire complexity for a workflow the SPA can do client-side on the visible page. Future RFC if a 10K-row table needs it. + +--- + +## 3. Design + +### 3.1 Backend RPCs + +#### 3.1.1 DynamoDB item RPCs + +```go +type AdminItemKey struct { + PartitionKey AdminAttributeValue + SortKey *AdminAttributeValue // nil for hash-only tables +} + +type AdminAttributeValue struct { + Type string // "S", "N", "B", "BOOL", "NULL", "L", "M", "SS", "NS", "BS" + Value any // shape depends on Type; mirrors the DynamoDB wire shape +} + +type AdminItem struct { + Attributes map[string]AdminAttributeValue +} + +type AdminScanResult struct { + Items []AdminItem + LastEvaluatedKey *AdminItemKey // nil when scan is fully drained +} + +type AdminScanOptions struct { + Limit int // [1, 100], default 25 + StartKey *AdminItemKey + ProjectAll bool // false to project only key attributes; true to project full items +} + +// Read-only. +AdminScanTable(ctx, principal, tableName, opts) (AdminScanResult, error) +AdminGetItem(ctx, principal, tableName, key) (*AdminItem, bool, error) + +// Write. +AdminPutItem(ctx, principal, tableName, item) error +AdminDeleteItem(ctx, principal, tableName, key) error +``` + +**Scan implementation.** Reuse the existing internal `Scan` machinery; the new admin call is a thin wrapper that: +- Validates `tableName` against the catalog +- Calls `scan` with the cursor (LastEvaluatedKey) translated from `AdminItemKey` +- Clamps Limit to `[1, adminItemScanMaxLimit=100]` +- Returns the AdminAttributeValue-shaped items and the next cursor + +**Key encoding.** `AdminItemKey` on the wire is `{partition: AttrValue, sort: AttrValue|null}`. The HTTP `{key}` path segment carries a base64-url-encoded JSON of the same shape (single round-trip per key, no need to plumb composite-key URI encoding through the routing layer). + +#### 3.1.2 S3 object RPCs + +```go +type AdminObject struct { + Key string + Size int64 + ETag string + LastModified time.Time + StorageClass string +} + +type AdminObjectListing struct { + Objects []AdminObject + CommonPrefixes []string // when delimiter is set, these are pseudo-directories + NextContinuationToken string // empty when fully drained +} + +type AdminListObjectsOptions struct { + Prefix string + Delimiter string // typically "/" for directory-style listings + ContinuationToken string + MaxKeys int // [1, 1000], default 100 +} + +// Read-only. +AdminListObjects(ctx, principal, bucket, opts) (AdminObjectListing, error) +AdminGetObject(ctx, principal, bucket, key) (body io.ReadCloser, meta AdminObject, err error) + +// Write. +AdminPutObject(ctx, principal, bucket, key string, body io.Reader, contentType string) error +AdminDeleteObject(ctx, principal, bucket, key string) error +``` + +**Streaming.** `AdminGetObject` / `AdminPutObject` use `io.Reader` so the 100 MiB body cap (see §3.3) does not force the leader to buffer the entire payload before responding. The HTTP handler relays the stream byte-for-byte. + +### 3.2 Authorization gates + +Same model as the SQS Peek + Purge division: + +| Operation | Gate | Notes | +|-----------|------|-------| +| AdminScanTable / AdminListObjects | `principalForReadSensitive` | Payload contents are sensitive; nil principal denied. | +| AdminGetItem / AdminGetObject | `principalForReadSensitive` | Same. | +| AdminPutItem / AdminPutObject | `principalForWrite` | Live-role re-check, action string passed through (e.g. "put item", "upload object"). | +| AdminDeleteItem / AdminDeleteObject | `principalForWrite` | Same. | + +The `principalForWrite` action-verb parameter (added in Phase 4 / r1 of the SQS work) carries forward — each new write handler passes its own verb so 403 bodies do not say "delete queues" for a put-object operation. + +### 3.3 HTTP endpoints + +#### 3.3.1 Path routing + +DynamoDB and S3 handlers extend the same 6-step routing procedure the SQS handler adopted (escape-aware, dot-segment-rejecting, dispatch on segment count). Three resource families: + +``` +/admin/api/v1/dynamo/tables → list tables (existing) +/admin/api/v1/dynamo/tables/{name} → describe / delete (existing) +/admin/api/v1/dynamo/tables/{name}/items → scan / put (NEW) +/admin/api/v1/dynamo/tables/{name}/items/{key} → get / put / delete (NEW) + +/admin/api/v1/s3/buckets → list buckets (existing) +/admin/api/v1/s3/buckets/{name} → describe / delete (existing) +/admin/api/v1/s3/buckets/{name}/objects → list / put (NEW) +/admin/api/v1/s3/buckets/{name}/objects/{key} → get / put / delete (NEW) +``` + +`{key}` is base64-url-encoded so arbitrary bytes (Dynamo item keys, S3 object keys with `/` or non-ASCII characters) traverse the path validator's `%`-ban without ambiguity. + +#### 3.3.2 Body shapes + +| Endpoint | Method | Request body | Response body | +|----------|--------|--------------|---------------| +| `/dynamo/tables/{n}/items` | GET | — | `{items: AdminItem[], next_cursor?: string}` | +| `/dynamo/tables/{n}/items` | POST | `{key, attributes}` (single item) | 204 | +| `/dynamo/tables/{n}/items/{key}` | GET | — | `AdminItem` | +| `/dynamo/tables/{n}/items/{key}` | PUT | `{attributes}` | 204 | +| `/dynamo/tables/{n}/items/{key}` | DELETE | — | 204 | +| `/s3/buckets/{n}/objects` | GET | — | `{objects: AdminObject[], common_prefixes: string[], next_continuation_token?: string}` | +| `/s3/buckets/{n}/objects/{key}` | GET | — | raw octet-stream + `Content-Type` + `Last-Modified` + `ETag` headers | +| `/s3/buckets/{n}/objects/{key}` | PUT | raw octet-stream | 204 | +| `/s3/buckets/{n}/objects/{key}` | DELETE | — | 204 | + +S3 GET responses stream the body directly (no JSON envelope, no base64 wrapper) so the browser can offer a Save-As dialog via `Content-Disposition: attachment`. + +#### 3.3.3 Upload cap + +S3 PutObject body is capped at `adminS3UploadMaxBytes = 100 MiB` (100 × 1024 × 1024 = 104857600 bytes). Enforced via `http.MaxBytesReader`. Exceeding the cap returns `413 Payload Too Large` with a structured `{"error": "payload_too_large", "message": "object exceeds 100 MiB admin upload cap"}` body. DynamoDB PutItem stays under DynamoDB's own item-size limit (400 KiB) so the JSON-body cap inherits the existing `BodyLimit` middleware (1 MiB). + +### 3.4 SPA pages + +#### 3.4.1 DynamoDetail Items tab + +Below the table's metadata / counters sections, a new "Items" section: + +- Top bar: "Add item" button (full role only). +- Paginated table (default 25 rows, server cap 100): columns Partition Key, Sort Key (if defined), Attributes (truncated preview), Created/Modified time if exposed. +- Row click → full-item JSON modal with Edit (replaces the item) and Delete buttons. +- Next / Refresh buttons driven by `next_cursor`. + +#### 3.4.2 S3Detail Objects tab + +Below the bucket's metadata / ACL section, a new "Objects" section: + +- Top bar: "Upload" button (file picker → `PUT /objects/{key}`); current prefix path (breadcrumb) so the operator can navigate up. +- Paginated table (default 100 rows): columns Key (truncated to filename when in a prefix), Size, Last Modified, ETag. +- Row click on a folder (CommonPrefix) navigates into it (`prefix=foo/bar/`); row click on an object opens a detail modal with Download and Delete. +- Delimiter is fixed to `/` so the bucket renders directory-style. A future "flat view" toggle can expose `delimiter=""` for operators who want the raw list. + +#### 3.4.3 SPA types + +`web/admin/src/api/client.ts` gains the new wire types and four × two = eight new methods (`scanTable` / `getItem` / `putItem` / `deleteItem`, `listObjects` / `getObject` / `putObject` / `deleteObject`). Upload progress is reported via the `fetch` `Request` `body` stream's existing `XMLHttpRequest`-fallback if a future enhancement adds progress indication; the MVP does not show a progress bar (operators upload 1–10 MiB config files typically, well under any threshold worth visualising). + +### 3.5 Audit + observability + +Four new audit lines (parallel to `admin.sqs.purge_queue` shape): + +``` +admin.dynamo.put_item subject role table key +admin.dynamo.delete_item subject role table key +admin.s3.put_object subject role bucket key bytes +admin.s3.delete_object subject role bucket key +``` + +Read-side calls (scan / get / list) DO NOT generate audit lines (the SPA polls; per-poll audit would drown the log). The admin request-log line covers them at the `route` / `subject` / `status_code` level. + +Two new Prometheus counters per adapter: + +``` +elastickv_admin_dynamo_item_writes_total{outcome ∈ ok,forbidden,not_found,validation,internal_error} +elastickv_admin_s3_object_writes_total{outcome ∈ ok,forbidden,not_found,validation,payload_too_large,internal_error} +``` + +Reads are bounded by Limit / MaxKeys, so a separate counter has low marginal value; the request-log line covers their cost analysis. + +--- + +## 4. Failure Modes + +1. **Concurrent writes.** Both backends already serialise writes through Raft; the admin RPCs use the same commit path. No new TOCTOU class. +2. **Large object download mid-stream.** If a follower loses leadership while streaming a 100 MiB Get, the connection is closed by the leader-step-down path. The operator's browser shows a partial download error; safest re-try is from scratch (the partial bytes have an unknown integrity boundary). +3. **Truncated upload.** `http.MaxBytesReader` cuts the stream at the cap; the leader receives a partial body and returns 413 without committing. No state change. +4. **DynamoDB item with binary attributes.** `B` / `BS` types arrive as base64 in the JSON wire shape (mirroring AWS); the admin RPC decodes lazily where needed. +5. **S3 versioning leakage.** This admin UI uses the latest-version path only; pre-version operations from another client are visible only via the underlying SDK. +6. **Invalid base64-url key.** Path validator rejects with 400 `invalid_path` before dispatch. + +--- + +## 5. Testing Plan + +1. **Unit (adapter)**: each new admin RPC gets a focused unit test for happy path + forbidden + missing-table-or-bucket + validation. Mirrors the SQS Phase 2 / Phase 3 pattern. +2. **Integration (`internal/admin`)**: HTTP handler tests cover routing (the 6-step path validation, table-level keys with `/` and binary attributes), role gates, 429-shaped errors where applicable, upload cap enforcement. +3. **SPA**: `npm run lint` (`tsc -b --noEmit`) plus an explicit "render after queue change clears state" test mirroring the SQS r3 fix. +4. **Jepsen**: out of scope. CRUD on existing primitives runs through the same OCC commit path Jepsen workloads already exercise. + +--- + +## 6. Rollout Plan + +| Phase | Content | +|-------|---------| +| 1 | This doc lands. | +| 2 | Backend `Admin{Scan/Get/Put/Delete}{Item,Object}` RPCs + sentinel errors + tests. Two adapters; can split into 2a (Dynamo) + 2b (S3) if review surface gets large. | +| 3 | HTTP handlers + bridges + integration tests (also potentially split per adapter). | +| 4 | SPA: DynamoDetail Items tab. | +| 5 | SPA: S3Detail Objects tab + Upload. | +| 6 | Doc rename `_proposed_` → `_implemented_`. | + +Each phase ships as a separate PR. Phase 2 (backend) lands first because the HTTP and SPA layers depend on the RPCs being callable. Phases 4 and 5 are SPA-only and can land in either order — 5 is slightly larger (upload + streaming) so it lands second. + +--- + +## 7. Open Questions + +1. **Should we expose `Query` on DynamoDB in this round?** The user opted for basic CRUD only; a follow-up RFC can revisit when operators ask for GSI inspection. +2. **Should S3 Get bypass the JSON envelope?** Yes — the design above commits to raw octet-stream so the browser's Save-As works without a base64 wrapper. The trade-off is that errors on the get path can't render as JSON; they use `text/plain` bodies on non-2xx like the existing S3 SigV4 path does. +3. **What's the threshold for a multipart upload?** Out of scope for this MVP. If operators routinely need > 100 MiB through the admin UI, a future RFC adds chunked PUT via the same endpoint signature with a `Content-Range` header. + +--- + +## 8. Alternatives Considered + +### 8.1 Build only DynamoDB now, defer S3 to a separate RFC + +Rejected: the SPA modal / pagination patterns the two pages share are nearly identical to the SQS Messages tab; building both at once amortises that work. Code-sharing happens at the helper level (`MessagesTable` style components) and the bridge / handler layer. + +### 8.2 Use the existing SigV4 paths through an admin proxy + +Rejected: forces the SPA's TypeScript client to construct SigV4-flavored bodies for what is otherwise plain JSON. The cost of the dedicated admin RPCs is one short wrapper per operation, and the wire shape stays consistent with the rest of the admin surface. + +### 8.3 Item-level update via PATCH instead of full PUT replace + +Rejected for the MVP: PATCH semantics in DynamoDB (UpdateItem) carry conditional-expression baggage that this round opted out of. A future RFC can add UpdateItem with a constrained expression vocabulary. From 01ee05701b5833f8c68b21821ef89dc38335d65f Mon Sep 17 00:00:00 2001 From: "Yoshiaki Ueda (bootjp)" Date: Sat, 23 May 2026 01:39:01 +0900 Subject: [PATCH 2/3] =?UTF-8?q?docs/design:=20r1=20review=20fixes=20?= =?UTF-8?q?=E2=80=94=20reconcile=20contracts,=20clarify=20Raft,=20add=20se?= =?UTF-8?q?curity=20headers?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Eight findings on PR #801 r1: - Codex P2 (POST /items vs PUT /items/{key}): dropped POST, keep only PUT-by-key; route summary updated to 'scan' / 'list' only on collection roots. Body-shape table now matches the RPC's AdminItem signature directly. - Codex P2 (collection-level PUT on S3 objects): removed; path routing now says 'list (NEW)' only for /objects collection root. - Gemini HIGH (100 MiB upload + Raft heartbeat instability): clarified — the admin path reuses the SigV4 chunked dispatch (s3ChunkSize=1 MiB × s3ChunkBatchOps=3 ~= 3 MiB Raft entries), well under MaxSizePerMsg=4 MiB. The 100 MiB cap is the admin UI's exposed limit, not Raft's capacity (SigV4 path supports 5 GiB on the same chunked write). - Gemini security-medium (XSS via raw object GET): added explicit defence-in-depth headers — CSP sandbox + nosniff + Disposition attachment + no-store. New §3.3.2 paragraph. - Gemini medium (AdminObject ContentType): added the field so the GET handler can set the response header. - Gemini medium (Scan 1 MiB response cap): explicit — admin does NOT loop internally on partial pages; SPA's Next-page button is the operator-driven refill. - Gemini medium (AdminAttributeValue recursive): annotated Value comment as recursive for L/M types. - Gemini medium (HTTP body shape vs AdminItem RPC): aligned — body is the flat AdminItem shape; key segment + body key are cross-checked before PutItem. --- .../2026_05_22_proposed_admin_data_browser.md | 35 ++++++++++++++----- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/docs/design/2026_05_22_proposed_admin_data_browser.md b/docs/design/2026_05_22_proposed_admin_data_browser.md index c74bb7d0..51d7f7fe 100644 --- a/docs/design/2026_05_22_proposed_admin_data_browser.md +++ b/docs/design/2026_05_22_proposed_admin_data_browser.md @@ -68,7 +68,7 @@ type AdminItemKey struct { type AdminAttributeValue struct { Type string // "S", "N", "B", "BOOL", "NULL", "L", "M", "SS", "NS", "BS" - Value any // shape depends on Type; mirrors the DynamoDB wire shape + Value any // recursive for "L" ([]AdminAttributeValue) and "M" (map[string]AdminAttributeValue); raw value for scalars } type AdminItem struct { @@ -101,6 +101,8 @@ AdminDeleteItem(ctx, principal, tableName, key) error - Clamps Limit to `[1, adminItemScanMaxLimit=100]` - Returns the AdminAttributeValue-shaped items and the next cursor +**Scan-response budget.** A single `Scan` call may return a `LastEvaluatedKey` before reaching `Limit` because DynamoDB's 1 MiB response cap fires (the engine inherits the same bound). The admin RPC does NOT loop internally to refill the page — partial pages are documented behaviour, and the SPA treats any non-nil `LastEvaluatedKey` as "more available" regardless of how full the current page is. Looping internally would multiply scan latency on tables with large items and pin the leader; the operator's "Next page" click is the cheap fix when a page comes back short. + **Key encoding.** `AdminItemKey` on the wire is `{partition: AttrValue, sort: AttrValue|null}`. The HTTP `{key}` path segment carries a base64-url-encoded JSON of the same shape (single round-trip per key, no need to plumb composite-key URI encoding through the routing layer). #### 3.1.2 S3 object RPCs @@ -109,6 +111,7 @@ AdminDeleteItem(ctx, principal, tableName, key) error type AdminObject struct { Key string Size int64 + ContentType string // backend-stored Content-Type so the GET handler can set the header ETag string LastModified time.Time StorageClass string @@ -160,15 +163,17 @@ DynamoDB and S3 handlers extend the same 6-step routing procedure the SQS handle ``` /admin/api/v1/dynamo/tables → list tables (existing) /admin/api/v1/dynamo/tables/{name} → describe / delete (existing) -/admin/api/v1/dynamo/tables/{name}/items → scan / put (NEW) +/admin/api/v1/dynamo/tables/{name}/items → scan (NEW) /admin/api/v1/dynamo/tables/{name}/items/{key} → get / put / delete (NEW) /admin/api/v1/s3/buckets → list buckets (existing) /admin/api/v1/s3/buckets/{name} → describe / delete (existing) -/admin/api/v1/s3/buckets/{name}/objects → list / put (NEW) +/admin/api/v1/s3/buckets/{name}/objects → list (NEW) /admin/api/v1/s3/buckets/{name}/objects/{key} → get / put / delete (NEW) ``` +The `/items` and `/objects` collection roots accept only GET. Creating an item or uploading an object always names the key explicitly via `PUT /items/{key}` / `PUT /objects/{key}` — put-create-or-replace semantics match DynamoDB's PutItem and S3's PutObject directly, and there is no implicit-key allocator the admin path needs to invent. A collection-level POST was considered (Codex r1 flagged the earlier draft's inconsistency between the route summary and the body-shape table) but the put-by-key form keeps the wire surface smaller and avoids two ways to do the same thing. + `{key}` is base64-url-encoded so arbitrary bytes (Dynamo item keys, S3 object keys with `/` or non-ASCII characters) traverse the path validator's `%`-ban without ambiguity. #### 3.3.2 Body shapes @@ -176,21 +181,35 @@ DynamoDB and S3 handlers extend the same 6-step routing procedure the SQS handle | Endpoint | Method | Request body | Response body | |----------|--------|--------------|---------------| | `/dynamo/tables/{n}/items` | GET | — | `{items: AdminItem[], next_cursor?: string}` | -| `/dynamo/tables/{n}/items` | POST | `{key, attributes}` (single item) | 204 | | `/dynamo/tables/{n}/items/{key}` | GET | — | `AdminItem` | -| `/dynamo/tables/{n}/items/{key}` | PUT | `{attributes}` | 204 | +| `/dynamo/tables/{n}/items/{key}` | PUT | `AdminItem` (flat `{attributes: {...}}` payload) | 204 | | `/dynamo/tables/{n}/items/{key}` | DELETE | — | 204 | | `/s3/buckets/{n}/objects` | GET | — | `{objects: AdminObject[], common_prefixes: string[], next_continuation_token?: string}` | | `/s3/buckets/{n}/objects/{key}` | GET | — | raw octet-stream + `Content-Type` + `Last-Modified` + `ETag` headers | | `/s3/buckets/{n}/objects/{key}` | PUT | raw octet-stream | 204 | | `/s3/buckets/{n}/objects/{key}` | DELETE | — | 204 | -S3 GET responses stream the body directly (no JSON envelope, no base64 wrapper) so the browser can offer a Save-As dialog via `Content-Disposition: attachment`. +DynamoDB PUT body matches the `AdminItem` shape directly (a flat `attributes` map). The `{key}` path segment supplies the primary key independently; the handler validates that the key segment's decoded attributes match the corresponding key attributes in the body so a client mis-typed key + body combination is rejected with `400 invalid_request` before the underlying PutItem runs. + +S3 GET responses stream the body directly (no JSON envelope, no base64 wrapper) so the browser can offer a Save-As dialog via `Content-Disposition: attachment; filename="..."`. + +**Security headers on S3 GET.** S3 objects can carry arbitrary bytes including HTML / SVG / JavaScript. Without active sandboxing, an operator clicking Download on a malicious HTML object would let the browser render attacker-supplied content in the admin origin — XSS. The GET handler defends in depth: -#### 3.3.3 Upload cap +- `X-Content-Type-Options: nosniff` — blocks MIME-sniffing escapes if the stored `Content-Type` is `text/plain` but the body is HTML. +- `Content-Security-Policy: default-src 'none'; sandbox` — even if the browser decides to render the body, every active behaviour (scripts, forms, navigation, plugins) is blocked. +- `Content-Disposition: attachment; filename="..."` — instructs the browser to download rather than render, with the original key as the filename. +- `Cache-Control: no-store` — operator inspections must not leak into shared caches. + +Codex r1 / Gemini security-medium on PR #801 flagged the bare-stream version as XSS-prone; the defence-in-depth above closes the class. + +#### 3.3.3 Upload cap and Raft sizing S3 PutObject body is capped at `adminS3UploadMaxBytes = 100 MiB` (100 × 1024 × 1024 = 104857600 bytes). Enforced via `http.MaxBytesReader`. Exceeding the cap returns `413 Payload Too Large` with a structured `{"error": "payload_too_large", "message": "object exceeds 100 MiB admin upload cap"}` body. DynamoDB PutItem stays under DynamoDB's own item-size limit (400 KiB) so the JSON-body cap inherits the existing `BodyLimit` middleware (1 MiB). +**Raft entry size is bounded by the existing chunked-write path.** Gemini r1 high-priority on PR #801 raised the concern that a 100 MiB body committed as a single Raft entry would block heartbeats / destabilise leader election / pressure memory. The admin path does NOT do that: `AdminPutObject` reuses the same chunked dispatch helper the SigV4 `PutObject` path uses (`adapter/s3.go` `s3ChunkSize = 1 MiB`, `s3ChunkBatchOps = 3` — see the long comment on lines 38–58 explaining the sizing). A 100 MiB upload commits ~33 Raft entries of ~3 MiB each, every entry strictly under the post-PR-#593 `MaxSizePerMsg = 4 MiB`. The SigV4 path already supports up to 5 GiB on the same chunked write; the admin path's 100 MiB cap is a deliberately conservative slice of that envelope for browser-driven uploads (most operator-driven uploads are config blobs / log fragments / DLQ exports under 10 MiB). Future raise to match the SigV4 5 GiB ceiling is one constant change away. + +The 100 MiB cap therefore expresses "what the admin UI deliberately exposes" rather than "what Raft can handle" — the latter is the SigV4 path's 5 GiB envelope. + ### 3.4 SPA pages #### 3.4.1 DynamoDetail Items tab @@ -241,7 +260,7 @@ Reads are bounded by Limit / MaxKeys, so a separate counter has low marginal val ## 4. Failure Modes -1. **Concurrent writes.** Both backends already serialise writes through Raft; the admin RPCs use the same commit path. No new TOCTOU class. +1. **Concurrent writes.** Both backends already serialise writes through Raft; the admin RPCs use the same commit path (chunked at `s3ChunkSize = 1 MiB` × `s3ChunkBatchOps = 3` for S3 large objects — see §3.3.3). No new TOCTOU class. 2. **Large object download mid-stream.** If a follower loses leadership while streaming a 100 MiB Get, the connection is closed by the leader-step-down path. The operator's browser shows a partial download error; safest re-try is from scratch (the partial bytes have an unknown integrity boundary). 3. **Truncated upload.** `http.MaxBytesReader` cuts the stream at the cap; the leader receives a partial body and returns 413 without committing. No state change. 4. **DynamoDB item with binary attributes.** `B` / `BS` types arrive as base64 in the JSON wire shape (mirroring AWS); the admin RPC decodes lazily where needed. From 14af408217c8d564806c691d563da7d4eccf6ea6 Mon Sep 17 00:00:00 2001 From: "Yoshiaki Ueda (bootjp)" Date: Sat, 23 May 2026 01:48:52 +0900 Subject: [PATCH 3/3] docs/design: correct truncated-upload state guarantee (Codex r2 P2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex r2 P2 on PR #801: the original §4 Failure Modes #3 stated that 413 returns 'without committing' and 'No state change'. Codex correctly pointed out that the SigV4 chunked write path commits chunk batches BEFORE the body-read error is surfaced, then performs best-effort cleanup via cleanupManifestBlobs. A coordinator dispatch failure during cleanup can leave orphan chunk blobs in storage. Reword the failure mode to spell out the actual contract: client-visible state (the object manifest) is unchanged but partial chunk blobs may need reaping by the per-generation bucket reaper. --- docs/design/2026_05_22_proposed_admin_data_browser.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/2026_05_22_proposed_admin_data_browser.md b/docs/design/2026_05_22_proposed_admin_data_browser.md index 51d7f7fe..e4b101f9 100644 --- a/docs/design/2026_05_22_proposed_admin_data_browser.md +++ b/docs/design/2026_05_22_proposed_admin_data_browser.md @@ -262,7 +262,7 @@ Reads are bounded by Limit / MaxKeys, so a separate counter has low marginal val 1. **Concurrent writes.** Both backends already serialise writes through Raft; the admin RPCs use the same commit path (chunked at `s3ChunkSize = 1 MiB` × `s3ChunkBatchOps = 3` for S3 large objects — see §3.3.3). No new TOCTOU class. 2. **Large object download mid-stream.** If a follower loses leadership while streaming a 100 MiB Get, the connection is closed by the leader-step-down path. The operator's browser shows a partial download error; safest re-try is from scratch (the partial bytes have an unknown integrity boundary). -3. **Truncated upload.** `http.MaxBytesReader` cuts the stream at the cap; the leader receives a partial body and returns 413 without committing. No state change. +3. **Truncated upload.** `http.MaxBytesReader` cuts the stream at the cap; the leader returns 413 to the client. The chunked write path (§3.3.3) may have already committed earlier chunk batches before the body-read failure surfaced, so this is **not** a guaranteed no-write outcome. The existing SigV4 `PutObject` path handles partial uploads via `cleanupManifestBlobs` (best-effort delete of the partial manifest's chunks — see `adapter/s3.go:914–967`), and the admin path reuses the same cleanup. Cleanup is best-effort: a coordinator dispatch failure during cleanup leaves orphan chunk blobs in storage until the per-generation bucket reaper sweeps them. Codex r2 on PR #801 caught the earlier draft's "No state change" claim as too strong; the corrected statement is "client-visible state is unchanged (no manifest commit) but partial chunk blobs may need reaping." 4. **DynamoDB item with binary attributes.** `B` / `BS` types arrive as base64 in the JSON wire shape (mirroring AWS); the admin RPC decodes lazily where needed. 5. **S3 versioning leakage.** This admin UI uses the latest-version path only; pre-version operations from another client are visible only via the underlying SDK. 6. **Invalid base64-url key.** Path validator rejects with 400 `invalid_path` before dispatch.