Skip to content

Prepare stable signed regression runners for durable guest authorization - #108

Merged
jdylanmc merged 10 commits into
pocketfrom
fix/stable-regression-runner
Oct 4, 2026
Merged

jdylanmc merged 10 commits into
pocketfrom
fix/stable-regression-runner

Conversation

@jdylanmc

@jdylanmc jdylanmc commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Stabilize the regression runner before further capture-based tests

Verified for the targeted unattended-runner model: changed-source execution and two final-source headless boots passed without additional human interaction, grants or resets. Merged into pocket at b8b70228d82bd0e8b208c2e34b4c94f6670fd9b2; no release was performed. Related to #18 and the foundation delivered by #101. This changes only the regression harness, its contracts, and documentation. The delivered change does not alter application behavior or replace the installed Notch app; no release is included.

Criterion Evidence
Explain authorization loss after rebuilding Preserved observations distinguish the previously approved ad-hoc runner from a rebuilt, differently identified runner. Normal designated requirements are now verified independently of code hashes.
Use a stable, explicit approved signing identity Normal host-side Xcode signing uses the same approved Developer ID certificate, signer, team and role identifiers. No credential transfer, custom weakened requirement, post-build re-signing or automated privacy grant.
Verify genuinely different native sources Protected before-source 7d357dd30eb7fccd3d033dd862cead8e091af254 and final source abfc8a2c7e986e9670ac7e23b33abd33a236df1d compiled, packaged and verified. Both role executable/CDHashes differ; signer configuration, entitlements, identifiers, flags and normal designated requirements match. P1 passed admission plus the full suite; final P2 passed admission plus the full suite on two fresh boots, with actual recording access true throughout.
Preserve prepared artifacts Schema v2 binds read-only Products container modes, including the root, and checks native ACL absence. Complete inventory stayed unchanged through execution, normal shutdown and cold boot. Same-UID write attempts were denied without creating files. Mutable state stays outside Products; earlier failures are preserved.
Preserve operation and gates Nine registered cases, expected controls, native deadlines and ad-hoc/unsigned defaults are unchanged. Current source passed 16 noninteractive local gates and all 12 hosted checks, including app tests on all three isolated matrix runners. Final-source suites achieved 18/18 expected outcomes over two boots.

Concrete startup correction

Native diagnostics exposed a real signing-input defect: Xcode's UI-test target fell back to the SDK entitlement file and claimed com.apple.application-identifier without a provisioning profile. The guest's taskgated-helper rejected it before any test completed.

Stable mode now supplies an explicit, empty, source-pinned StableRunner.entitlements using ordinary CODE_SIGN_ENTITLEMENTS. This replaces the generic SDK input, not XCTest's product-type or base/debug entitlements. Native comparison proved that only the unsupported application-ID claim disappeared; every other entitlement and the existing signing identity remained unchanged. Clearing DEVELOPMENT_TEAM was tried, did not fix the claim, and was not retained.

Verification now rejects that unprovisioned claim and requires actual Boolean sandbox and get-task-allow entitlements. Base-entitlement injection remains enabled. The signing correction does not edit privacy databases, container access-control records or existing signed binaries.

Native receipts additionally observe CGPreflightScreenCaptureAccess() before interaction and, in B2, after restoration. These are non-prompting observations, not substitutes for output assertions or permission grants.

Unattended update model

build_runner.py compare now verifies two complete portable packages using independently supplied pins and each package's own source snapshot. It rejects signer, role, architecture, designated-requirement, entitlement or flag drift. It reports source/executable/code-hash changes without claiming consent or native acceptance. Unchanged packages are compatible, not cross-source proof.

The setup, test-authoring and independent-suite workflows now distinguish:

  • One-time setup of the guest account and stable identity, before any ad-hoc grant on a fresh guest.
  • Explicitly owner-directed one-time migration of legacy approvals; no app-data deletion, direct privacy-database edits, or reset/grant in the normal build/run path.
  • Normal test additions and builds using the same configured identity and fixed runtime path, with verification before promotion and invocation.
  • Actual changed-source and cold-restart proof without another prompt before calling the baseline unattended.

New OS capabilities or intentional identity changes remain separate setup work. Routine source changes must not reactivate setup or make consent a per-run step.

Protected artifact lifecycle

Metadata files reappeared across reboot after setup browsing, while every approved file remained unchanged. Repeatedly restoring a writable package did not establish an unattended baseline.

Stable builds now protect Products and its verified runner-ancestor directories, deepest first, after signing verification. Signed app/test-bundle contents and modes are not changed. Only those declared container-mode transitions are allowed; newly appearing files fail rather than being incorporated into expectations.

Schema v2 binds the root mode and rechecks read-only, owned containers and native file-security state before/after signature verification. ACL lookup uses checked fstatx_np/filesec, requiring populated owner/group/mode before accepting ACL absence. This closes the review finding where listing inference or libc's masked allocation failure could appear ACL-free.

Legacy schema v1 remains inspectable for migration but cannot satisfy verify --require-protected-products; comparison rejects a v2-to-v1 protection downgrade. Ad-hoc/unsigned defaults remain v1. Protection addresses incidental directory-entry writes in the named containers, not privileged writers or an owner changing permissions.

Actual same-UID write attempts were denied with EACCES on the host and guest, including after archive extraction and both final-source boots. The fixed runtime path remained unchanged. No metadata appeared in the protected containers during the completed proof.

Validation and independent review

At current source abfc8a2c7e986e9670ac7e23b33abd33a236df1d:

  • All 16 noninteractive local gates passed without a validation retry: helper 69, Python 368, workflow contracts 176, PR policy 22, and pixel oracles 220, plus declared builds/lints and default standalone-runner compilation.
  • App-hosted tests were not rerun on the user's normal desktop. All three current-head isolated GitHub Test steps succeeded. The local result explicitly records that delegated step; it is not presented as 17/17 local completion.
  • Existing non-serious lint warnings remain; no checks, assertions or deadlines were weakened.
  • Independent Roast completed the recovery/lifecycle reviews. The protected-container ACL finding was corrected and re-reviewed at this exact head; no open findings remain. Initial flawed ACL-inspection candidates were not promoted.
  • Native administration restored original foreground/pointer both without movement and with a necessary pointer warp returning raw CoreGraphics code 0. All nine latest A2 tests restored their initially closed Settings state.
  • All 12 current-head hosted checks passed, including Swift analysis, the three app build/test matrix legs and CodeQL. A watcher network timeout was retained as an observation failure, then recovered; no CI job was requeued or prior-head result substituted.

Runtime evidence and completed acceptance

The corrected A2 runner exposed a second, distinct prerequisite: its stable identity was not yet authorized for the existing sandbox data container created by the old ad-hoc runner. An independent full-suite attempt timed out on the first About case; eight cases were unrun. A native process sample and container-service logs identified the startup wait. No completed test, capture-permission result or suite pass was claimed.

The human subsequently consented through the actual runner prompt; macOS recorded that consent for the expected Developer ID identity and existing test container. No additional programmatic approval occurred. A consent-only probe that timed out before the human response remains preserved as BLOCKED.

Fresh A2 headless verification passed 9/9 expected outcomes: four functional PASS results and five controls producing the registered raw two FAIL / three BLOCKED outcomes. Nine actual native tests, zero skips, eight inspected public Settings captures, complete UI/app/process/job/lock cleanup, and unchanged candidate/signatures/preferences/data. Both VMs stopped normally.

At that historical point, screenCapturePreflightAccess was false in all nine receipts. Native logs identified the existing Screen Capture entry's old ad-hoc cdhash requirement. Container consent fixed startup; a successful XCTest screenshot did not establish Screen Recording permission.

The UI remove/re-add did not replace the effective legacy requirement. After the owner's explicit request, one targeted supported tccutil reset ScreenCapture operation for this runner bundle was executed; native Reset/Delete events confirmed removal. It did not grant access and was not added to routine automation. A fresh boot confirmed the obsolete requirement was gone and the decision was Unknown/None.

The human then enabled the exact stable runner again. Initial verification stopped at recurring .DS_Store inventory drift before it could measure permission; that was not another denied grant. Protected containers resolved this separate storage problem without further permission changes.

The entire unprotected package is preserved. Protected P1 then passed one admission plus all nine expected suite outcomes: ten actual true capture preflights, zero skips and complete cleanup. The parent verified P1's complete protected inventory again after another cold boot before promoting P2.

Final P2 update/restart proof passed:

  • Two fresh headless boots; one admission plus the full nine-case suite on each.
  • Twenty actual native tests total. The full suites achieved 18/18 expected outcomes across nine distinct registered cases run twice.
  • All twenty BEFORE and twenty AFTER recording preflights were true. No screenshot-based permission inference.
  • Full-suite raw totals were 8 PASS / 4 FAIL / 6 BLOCKED; all negative controls produced their expected reasons.
  • Zero additional human interactions, grants, resets, artifact changes or native retries.
  • Protected v2 inventory, 0555 container modes and native ACL policy remained valid across reboot. Four write-denial probes returned EACCES without creating files.
  • App/UI/pointer restored both rounds; source, candidate, signatures and scoped data preserved; jobs/processes/locks cleared; both normal shutdowns returned Tart 0.

An extra read-only cross-reboot diagnostic initially compared kernel st_dev numbers. Its failure is preserved: all 141 entries differed only in the mount-assigned device number, while persistent metadata and canonical inventory/signature checks remained unchanged. No test expectation, artifact or native run was repaired or retried.

Local acceptance receipt: evidence/pr108-unattended-runner-acceptance.json in the ignored VM environment, linked to the independently produced P1 and combined P2 reports. Raw images, credentials and VM data are not published.

This proves the targeted model for the configured guest and approved capability set: normal rebuilt-runner updates and these suite runs needed no intervention. It does not claim all application features, new OS capabilities, future OS/signing changes, or indefinite/overnight reliability. Broader foundation work remains separate.

Earlier provisioning failures, incomplete result bundles, whole-package metadata-drift recovery, and failed administrative restoration remain historical evidence. Stale locks were archived only after exact ownership/liveness checks. No old failure has been relabeled as a pass.

@jdylanmc
jdylanmc marked this pull request as ready for review October 4, 2026 03:12
@jdylanmc
jdylanmc merged commit b8b7022 into pocket Oct 4, 2026
18 checks passed
@jdylanmc jdylanmc mentioned this pull request Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant