Skip to content

QA run · priority:P0 · e4e5c6e3 · 2026-08-17 · 7 PASS / 7 PARTIAL / 2 FAIL / 2 BLOCKED #9334

Description

@os-zhuang

Round R1 of the platform-checklist wave tracked at #9296 — the release-gating smoke set, run locally in the R0-provisioned tree to answer "is anything release-blocking broken?" ahead of the three cloud sessions.

Two P0 failures found. Both were surfaced on #9296 the moment they survived second derivation, rather than held to round end:

Environment fingerprint

framework sha (subject) e4e5c6e3c608b1b807c83a0d5b734f213eb1a1ddnot moved
.objectui-sha (pin) 665661ab093263f39f2e660a295ea615dbcee35a
console dist stamp 665661ab093263f39f2e660a295ea615dbcee35astamp == pin, /_console/ served 200 text/html
spec source PR #9309 head fbd1c0491215406c379405cef262befad1de458d (verified with git rev-parse FETCH_HEAD; in the merge queue at run time, not yet on main). docs/qa/platform-checklist/ is byte-identical to the earlier d23a746c7 — the newer commit only merges main — so the ledger ran is exactly the reviewed one
validator check-platform-checklist: OK — 15 areas, 190 items (190 active); coverage: 30 kinds mapped, 0 waived
selector node scripts/checklist-select.mjs priority:P0 --json18 runnable items (16 → 18 after the R4 sweep, as predicted)
ports / DBs 3110 file:/tmp/qa-wave/r1-core.db · 3111 r1-portshift.db · 3112 r1-del.db (band 3110–3129, ≤3 concurrent)
seed --seed-adminadmin@objectos.ai / admin123; members rls-a@verify.test, rls-b@verify.test
repo hygiene nothing committed; git status shows only docs/qa/**. No JSON, no screenshots, no PR

Boot: health 200, /api/v1/ready{"status":"ready","state":"running"}, Seeds: com.example.showcase 130 rows, 0 ERROR lines.

Environment correction worth carrying forward: Playwright 1.62.1 expects chromium build 1234; this box has only chromium-1194. The first e2e attempt reported 31 failed — pure environment, zero product signal. Passing launchOptions.executablePath=/opt/pw-browsers/chromium-1194/chrome-linux/chrome turns it into 31 passed. Any later round that runs Playwright here must set this or it will file a spectacular false P0.

Scope — per-item revision and derived verdict

# item rev surface verdict
1 access-security.rls-both-sides 3 api partial
2 access-security.write-path-guards 2 api pass
3 access-security.crud-permission-matrix 1 api partial
4 access-security.owd-sharing-matrix 2 api pass
5 access-security.anonymous-deny-surfaces 1 api pass
6 access-security.no-active-org-session-semantics 1 api blocked(fixture)
7 ai.mcp-stdio-fail-closed 3 cli partial
8 api-backend.query-contract-matrix 1 api partial
9 approvals.account-app-entry 1 browser partial
10 cli.dev-boot-contract 2 cli partial
11 integration-system.datasource-credential-refusal-matrix 1 mixed FAIL
12 platform-core.boot-health 2 mixed pass
13 platform-core.seed-integrity 1 api pass
14 platform-core.console-login 2 browser pass
15 platform-core.nav-surfaces-render 3 browser pass
16 platform-core.builtin-apps-nav-render 1 browser partial
17 records-forms.crud-roundtrip 3 browser FAIL
18 studio-authoring.first-run-loop 1 mixed blocked(fixture)

Per-clause verdicts (text oracle evidence)

Automated refs run and cited (RUNNER rule 6) — 8 items

ref result
8 access-security dogfood files (anonymous-deny, private-owd, owner-anchor-and-bulk-writes, static-readonly, readonly-when-parent, public-read-owd, controlled-by-parent, invoice-cbp) 8 files / 58 tests passed, 0 failed (130s)
packages/mcp/src/__tests__/plugin.test.ts 17 passed, incl. all 4 ADR-0101 stdio fail-closed cases
packages/objectql/src/engine.test.ts 135 passed (the ERROR log lines are deliberate negative-path assertions)
examples/app-showcase/e2e/showcase-smoke.spec.ts 31 passed — all 31 nav surfaces render, no pageerror, charts laid out
objectstack verify 15 verified, 0 gaps, 0 FAILED, 1 needs-fixture, 7 skipped — reproduces R0 exactly
objectstack verify --rls 20 PROVEN, 0 HOLES; 9/9 position personas probed, 0 HOLES across all
packages/plugins/plugin-security/src/no-active-organization-write-refusal.test.ts 19 passed (corroboration for item 6, see below)

⚠️ objectstack verify --rls is a separate invocation. R0's reading (and my first pass) ran bare objectstack verify, which emits no RLS section at allrunRlsProofs only runs behind the --rls flag. Any round citing plain verify as the oracle for an RLS clause has not actually consulted it.

1 · access-security.rls-both-sides (rev 3) → partial

clause verdict oracle evidence
0 A's list is exactly A's pass A list total=2 ["A note 1","A note 2"]; B list total=1 ["B note 1"] — presence and absence, both directions
1 foreign by-id read denied pass A → B's note: 404 RECORD_NOT_FOUND (not-found-shaped denial, permitted)
2 foreign by-id write denied, row unchanged pass A PATCH → 403 PERMISSION_DENIED; admin re-read body="b1", updated_at unchanged from creation
3 admin reads full set pass admin list total=3 — all three run-created ids
4 runRlsProofs no holes pass 20 PROVEN (20 consistent, 0 HOLES); all 9 position personas 0 HOLES
5 skips explainable + D11 probe blocked(dependency) see below

Clause 5: the 3 skips are each benign — showcase_project_membership "no plain-text probe field", showcase_ext_customer/showcase_ext_order external read-only. Hand spot-check of the skipped object: member A → 403 PERMISSION_DENIED, admin sees rows, i.e. genuinely protected. But the clause requires judging against a write-up held privately with the maintainer and deliberately absent from this repo — this runner does not hold it, and the clause says never to tick it green over that. Recorded blocked(dependency), not pass. Nothing about it is restated here.

5 · access-security.anonymous-deny-surfaces (rev 1) → pass

The dogfood pin covers /data, /meta, /actions, /automation, /packages (25 tests). It does not cover /batch or /security/explain, both named in the item's variants — driven by hand so the item is not left partial:

variant (anonymous) status envelope family
GET /api/v1/data/showcase_private_note 401 UNAUTHENTICATED rest-flat
GET /api/v1/meta 401 UNAUTHENTICATED rest-flat
POST /api/v1/actions/showcase_task/showcase_mark_done/anon-probe-id 401, not 404 dispatcher-wrapper
GET /api/v1/automation 401 UNAUTHENTICATED dispatcher-wrapper
POST /api/v1/batch 401 UNAUTHENTICATED rest-flat
GET /api/v1/security/explain 401 UNAUTHENTICATED rest-flat

Exactly the two declared families, mapped as the item specifies — no third dialect. Member contrast: data + meta both 200. Declared-public route: anonymous GET /api/v1/forms/<slug>404 FORM_NOT_FOUND, not 401 — the gate keys on declared posture, not blanket. Destructive negative: anonymous DELETE /api/v1/automation/showcase_reassign_wizard → 401, and an authed listing still shows the flow among 29.

12 · platform-core.boot-health (rev 2) → pass

health 200 · /ready 200 {"status":"ready","state":"running"} · Flows banner 29 flow(s) 22 bound … · 7 draft with no ⚠ and 0 ERROR lines · /_console/ 200 text/html 6467 bytes · nav diff empty — served grp_data children equal the 12 authored in src/ui/apps/index.ts, nothing added or dropped.

3 WARN lines exist, none of them a SeedLoader rejection: one is the #8686 autonumber tenancy note (which fires at 13:37:39, 3 s before the default organization row is created at 13:37:42 — the "0 rows in sys_organization" it reports is a boot-ordering artifact, not the steady state), and two are SHARING_NOT_ENABLED rule-backfill failures for share_open_tasks_with_manager / share_red_projects_with_execs. Those two mean the showcase ships two authored sharing rules that can never be consulted — noted as a fixture observation, not scored here.

13 · platform-core.seed-integrity (rev 1) → pass

Counts re-derived from src/data/seed/index.ts rather than trusted from the item's baseline. All match, and are identical after a restart against the same DB (idempotent upsert, 130 rows both boots, no rejection lines): account 14 · contact 32 · project 5 · task 10 · invoice 12 · invoice_line 5 · expense_report 4 · expense_line 13 · field_zoo 2 · announcement 2 · product 4 · team 2 · category 4 · preference 1.

Values verbatim: f_multiselect ["red","green"], f_json {"nested":{"k":"v"},"list":[1,2,3]} structurally intact, f_lookup resolves to the real Northwind account id. Expected-absent fields (f_user, f_users, f_secret) null as documented.

14 · platform-core.console-login (rev 2) → pass

clause verdict evidence
0 lands in shell pass screenshot showed the ObjectOS shell — greeting "Good afternoon, Dev Admin", header/search/avatar, "Your apps 2", "3 pending approvals"; 0 password fields; url /_console/home
1 session survives reload pass post-reload 200 GET /api/v1/runtime/config, 200 /auth/get-session, 200 /i18n/translations/en; no bounce to login
2 cleared session → 401 on the wire pass after clearCookies(), in-page fetch /api/v1/meta/app401 UNAUTHENTICATED
3 console returns to login pass navigation redirected to /_console/login?redirect=%2Fapps%2Fshowcase_app%2Fshowcase_project, login surface rendered — not a dead shell
4 re-auth restores pass back at /_console/home, authed fetch → 200
NEG wrong password pass 401 POST /api/v1/auth/sign-in/email, no cookies set, visible error "Invalid email or password"

0 page errors across the whole drive.

17 · records-forms.crud-roundtrip (rev 3) → FAIL

clause verdict evidence
0 create + verbatim re-read pass 201, name/status/industry returned verbatim
1 update touches only the edited field pass before/after full-row diff = ["updated_at","annual_revenue"]; null → 123456
2 delete removes the row FAIL 400 INVALID_FILTER; row survives
3 list reflects mutations after reload not-run browser walk not driven this round
4 History tab not-run browser walk not driven this round
5 unicode round-trip pass os-qa-r1-华宁 read back byte-identical
6 clone re-derives engine-owned columns pass 201, new id ≠ source, sourceId echoed, values copied, created_at re-derived
7 clone RLS-gated blocked(fixture) member A can read showcase_account (public_read_write), so no non-reading persona is bound on this boot — the item explicitly allows recording this as blocked. A's clone returned 403 PERMISSION_DENIED (no create grant), which does not exercise the 404 path the clause is about
NEG missing required status pass, checklist wrong see checklist findings
NEG bad tax_id pass 400 VALIDATION_FAILED, fields:[{field:"tax_id",code:"invalid_format"}]

Reproduction rule for clause 2 — full write-up and causal chain in the #9296 comment:

DELETE /api/v1/data/showcase_account/<any id>
  • expected: 200, row gone.
  • actual: 400 {"code":"INVALID_FILTER","object":"showcase_account","error":"The bare equality spelling { \"f_lookups\": value } WAS NOT APPLIED: … Use \"$contains\" …"}, and a follow-up GET on the same id returns 200 — the row survives.

Cause: packages/objectql/src/engine.ts:10112 probes dependents with find(childName, { where: { [fieldName]: id } }) — bare equality — for every referencing field. showcase_field_zoo.f_lookups is Field.lookup('showcase_account', { multiple: true }), a JSON-valued column the engine's own #5869 filter guard refuses. The catch below rethrows anything that is not a missing-table error, so the probe failure aborts the delete.

Marked regression: before the #8895 hardening that catch swallowed probe failures, so this delete previously succeeded (while silently skipping the integrity check). The #8895 change is correct in itself — it converted a silent fail-open into a loud refusal, which is what made the pre-existing bad filter shape visible. The remedy is the one the error text itself gives: $contains for multi-value fields.

Scope measured, not assumed: showcase_team / showcase_category / showcase_announcement / showcase_private_note / showcase_project delete 200; showcase_product409 DELETE_RESTRICTED (correct guard); sys_user405 OBJECT_API_METHOD_NOT_ALLOWED, which is the only reason the multi-value Field.user columns (f_users, team_members) don't reach the same probe. On stock showcase the blast radius is exactly showcase_account; the framework rule is broader.

11 · integration-system.datasource-credential-refusal-matrix (rev 1) → FAIL

clause verdict evidence
0 every inline-credential door refuses at publish not-run the 6-door publish matrix was not driven this round
1 credentialsRef alternative works not-run not driven
2 both read doors redact pass no cleartext in either door; /datasources/showcase_externalhasSecret:false, redactedConfigKeys:[]; metadata door shows config.filename only. The two doors agree
3 redaction presence-preserving partial only an unset-credential datasource exists on stock showcase, so the set-vs-unset distinction could not be exercised
4 legacy alias spellings redacted blocked(fixture) requires planting a stored row directly; not done — the item mandates recording this rather than passing it
5 unknown-driver canonical keys redacted not-run not driven
6 echoed-mask write guard not-run not driven
7 non-admin/anonymous get no more than admin FAIL anonymous responses are byte-identical to the platform admin's on both read doors; anonymous create returns 201 and anonymous delete 204

Reproduction withheld from this issue per the wave's rule for real access-control holes — it is on #9296 for the maintainer. Summary only: the datasource-admin family is mounted directly on IHttpServer by a CLI-built plugin, bypassing the @objectstack/rest enforceAuth seam, with no guard in the registrar (packages/services/service-datasource/src/admin-routes.ts:113). The family's own ledger already describes this posture as "mounted, working, and guarded by nothing". #7744 closed having added ledger entries without adding a guard; I find no open issue tracking the authentication hole itself. Long-standing, therefore not labelled a regression.

6 · access-security.no-active-org-session-semantics (rev 1) → blocked(fixture)

Brand-new ADR-0123 item, never run before. Its own fixture precondition is not met on stock showcase, so every clause is vacuous — the item instructs confirming the posture before scoring, and that is what this verdict records.

Posture evidence, three independent sources agreeing:

  1. Boot banner: Tenancy: single.
  2. Runtime log, authoritative: [security] tenancy posture 'single' — Layer 0 is inert; the platform's own tenant-scoped RLS policies are stripped (ADR-0105 D3).
  3. Source: tenant-layer.ts:119if (!postureEnforcesWall(input.tenancyPosture)) return nullsingle contributes no wall.

The state itself is also not producible here: a fresh signup lands with activeOrganizationId: "org_msxa3kt9sxctxowj" (the seeded Default Organization), so the racy producer the item names is unavailable and the removed-member producer needs a fixture nobody ships.

Corroboration rather than a verdict: plugin-security/src/no-active-organization-write-refusal.test.ts runs 19/19 green at this sha and pins every clause of this item at unit level — including a control that is precisely "the single posture has no wall, so an org-less write is ordinary", i.e. the test itself agrees the showcase configuration makes this vacuous.

18 · studio-authoring.first-run-loop (rev 1) → blocked(fixture)

Studio is not installed in the showcase. The boot plugin list carries @objectstack/setup and @objectstack/account but no studio plugin; GET /api/v1/meta/app as platform admin returns ["showcase_app","setup","account"], and the post-login screenshot independently shows "Your apps 2" (Showcase, Setup). STUDIO_APP exists (packages/platform-objects/src/apps/studio.app.ts, name: 'studio', requiredPermissions: ['studio.access']) and packages/apps/studio ships, but the showcase config does not install it. Every step of this item begins in Studio, so nothing is scoreable. Clause 5 (server-side read-only-package gate) is independently reachable via API and was not driven this round.

Remaining partials — what was and was not proved

  • 2 write-path-guards → pass via the 3-file automation pin (17 tests: owner forge/disown/transfer, bulk update+delete scoping, static readonly stripping, parent-scoped readonlyWhen, X-ObjectStack-Dropped-Fields). Clause mapping inherited from the pin, not independently re-driven.
  • 4 owd-sharing-matrix → pass via its 4-file pin (16 tests: private OWD, public-read OWD, controlled-by-parent both fixtures). Clause 4 deliberately not double-counted against the private D11 defect — the item's own negative says it is recorded once, on rls-both-sides.
  • 3 crud-permission-matrix → the verify half is green (CRUD 15/0/0, RLS 0 holes, 9 position personas). Clause 6 requires a verdict per cell across ~31 access-matrix rows × 4 verbs with a member provisioned per permission set; not driven. Honest partial, not pass.
  • 7 ai.mcp-stdio-fail-closed → the 4 ADR-0101 unit cases are green (keyless refusal, unresolvable key, objectql unavailable, no key demanded when stdio is off). The api clauses (member-keyed MCP reads vs REST row/field sets, aggregate parity, FLS aggregate-input gate, revocation on next read) were not driven — they need a real stdio transport with a provisioned API key.
  • 8 api-backend.query-contract-matrix → driven live: $top/$skip window 200; $select narrows to exactly name,status; $orderby on a bad field → 400 INVALID_SORT; scalar comparand on a collection operator → 400 INVALID_FILTER; unknown $-param → 400 UNSUPPORTED_QUERY_PARAM; unknown field → 400 INVALID_FIELD; empty-list boundary {status:{$in:[]}} → 200/0 rows and {status:{$nin:[]}} → 200/18 rows (full visible set), both spellings. Clause 4 (expand RLS/FLS) rests on the engine pin. Clauses 0 and 8 (full operator matrix set-equality, verdict per variant) not driven → partial.
  • 9 approvals.account-app-entry → clause 0 pass on the build oracle: grep -o 'approvals:inbox' packages/console/dist/assets/index-*.js hits in index-B1QKp1HQ.js, so the registration survives bundling — exactly what the item says the unit pin cannot see. Clauses 1–6 need the non-admin persona plus a drawer walk; not driven.
  • 10 cli.dev-boot-contract → clause 0 pass (health/ready//_console); clause 4 pass — instance B on a busy port printed ↪ server bound to port 3111 (requested 3110) and the banner advertised 3111, no silent divergence; clause 6 pass — no config, no artifact exits 1 with the exact remedy Run in a directory with objectstack.config.ts, pass --artifact <path|url>, or run from the monorepo root. Clauses 1, 2, 3, 5 (seed idempotence under a changed --admin-password, the DB-resolution matrix, --fresh tempdir, stale-artifact warning) not driven.
  • 16 builtin-apps-nav-render → clause 3 pass, server-side: member GET /api/v1/meta/app?id=setup{"items":[]} while admin gets items:1; the member's app list is ["showcase_app","account"] vs the admin's ["showcase_app","setup","account"] — the server prunes, the launcher does not merely hide. Clause 4 pass (all Setup entries absent for the member; nav_jwks absent for admin too, per Setup › Advanced › Signing Keys (JWKS) is a dead nav entry — sys_jwks is API-disabled and a requiredPermissions gate cannot prune it #7544, so it proves nothing either way). Clause 1 partial — Setup's 9 group anchors and Account's Profile/Inbox/Security/Developer all present; Studio absent per item 18. Clauses 0/2/6 (per-destination render walk) not driven.

Checklist accuracy findings — for the ledger owner, not product defects

Each of these made a correct product behaviour look wrong on first read. Filed here rather than edited into the tree (this round commits nothing).

  1. platform-core.seed-integrity step 1 states showcase_contact=33 (9 named + 24 bulk prospects). The seed authors 8 named + 24 = 32, and the server reports 32. The baseline is stale; the item's own instruction to re-derive from the seed module is what saved the clause from a false FAIL.
  2. platform-core.boot-health step 6 / clause 3 and builtin-apps-nav-render steps 0–1 use GET /api/v1/meta/app?id=com.example.showcase (and ?id=com.objectstack.setup / .studio / .account). That parameter keys on the app name, not the package id — the showcase app is showcase_app, the built-ins are setup / studio / account. The documented spellings return {"type":"app","items":[]}, which reads exactly like "the app metadata is gone". The dogfood skill §1 carries the same shape and would benefit from the same correction.
  3. records-forms.crud-roundtrip negative expects a direct POST missing required status to answer 400 VALIDATION_FAILED with code:'required'. showcase_account.status is required: true but its prospect option carries default: true, so the default satisfies it and the row is created with status:"prospect". Correct behaviour; the negative as written can never hold for this object.
  4. access-security.no-active-org-session-semantics carries no automated.ref, yet plugin-security/src/no-active-organization-write-refusal.test.ts pins every one of its clauses at unit level. Per RUNNER rule 6's converse, that test is the natural automated value for a future revision — and it would tell any runner up front that the showcase posture makes the item vacuous.
  5. access-security.rls-both-sides clause 4 and crud-permission-matrix both name runRlsProofs as the oracle without noting it requires objectstack verify --rls. Bare verify prints no RLS section, so the flag is load-bearing.

Fixture gaps

  • tenancy: single on stock showcase makes the whole ADR-0123 tenant-semantics surface unscoreable (item 6). A walled fixture — or a documented isolated/group boot recipe — is what would unblock it.
  • Studio is not installed in the showcase, so item 18 and the Studio third of item 16 have no surface.
  • No non-reading persona is bound on a stock boot, so the clone-RLS clause (17/7) cannot be exercised; showcase_account is public_read_write.
  • integration-system.… clauses 3 and 4 need a datasource with a bound credential and a directly-planted legacy-alias row; neither exists on stock showcase.
  • Two authored sharing rules (share_open_tasks_with_manager, share_red_projects_with_execs) fail backfill at every boot with SHARING_NOT_ENABLED — they are inert fixtures.

Anti-false-positive work this round

Three findings were disproven before they reached this issue, which is the protocol earning its cost:

  • "31 e2e surfaces failed" — Playwright/chromium build mismatch, not the product. 31/31 pass with the right executablePath.
  • "showcase_reassign_wizard was deleted / is missing" — my parser read f.name from data.flows, which is an array of strings. The flow is present among 29.
  • "$nin: [] is rejected 400" — I had spelled the operator "not in", which is not in the recognised set; $nin / nin both answer 200 with the full visible set.

Conversely, both real failures survived: reproduced on two independent boots each with fresh DBs and distinct ports, driven with plain curl (no browser, so no automation-input/hydration-race exposure), wire-level proof of no credentials on the auth-hole probes, every listed trap ruled out, and a second independent re-derivation from the captured evidence and source alone before either was written down (RUNNER rule 7, which applies to all 18 items here since every one is P0).

Parent: #9296 · baseline R0: #9298 · spec sweep: #9299 / PR #9309.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions