test(source-maps): e2e snapshot harness for more programs - #1154
Merged
gewenyu99 merged 26 commits intoAug 31, 2026
Merged
Conversation
Stacks the self-driving e2e testing onto the feature. The detect screen is interactive-only, so it's listed no-action; the harness covers the rest of the flow plus the offline flow-trace snapshot. Re-enables the e2e-harness test suite the base PR excludes from its run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
in-program-phases.ts was removed; the host now advances each composed step the same way run-wizard does (auth → run steps → gating screens). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run-step composition replaced runProgram/completePhase with run/ completeRunStep; the trace test completes the integrate-run step the same way. Regenerate the self-driving golden (detect -> integrate -> handoff -> run). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tui-host injects the detect pick (detect the framework at the install dir, single-app fixtures) so the live snapshot run advances past the interactive picker the store driver can't actuate. - e2e.json path now reflects the real flow (detect, integration run, handoff). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The auto-pick detected the framework at the repo root, which a Turborepo root reports as generic node, so it would integrate the workspace root. Scan apps/ then packages/ for the first sub-app with a registered framework, falling back to the root for a single-app fixture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ving's asks E2E snapshotting is driven, not CI — the host answers wizard_ask via its driver, so it sets WIZARD_ASK_AUTODRIVE and the runner keeps the ask bridge wired despite ci (which here is only for headless auth). Self-driving's profile declines every ask (ask: cancel) so the agent sets nothing up and walks to the outro. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cancelling declined the required GitHub step and aborted. Answering with the
first option gives each ask its affirmative 'continue' ("GitHub connected →
done"), so with a scoped key the run completes to the success outro. Reverts
the unused cancel strategy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make error-tracking-upload-source-maps e2e-drivable, stacked on the self-driving harness work: - e2e.json profile + path for the program; registered in profiles.ts - pick_source_maps_project driver action commits the picker's three selected* context keys (the candidate list lives in the detect screen's agentic report, so the caller supplies the pick); the host computes it with the static prerequisite detector - dismiss_outro wired for source-maps-outro (dismissed by keypress interactively, so the driver needs a real action) - host ask overrides: STEP 1's "api-key" answered with SOURCE_MAPS_CLI_KEY (raw — the wizard_ask tool vaults it into a secretRef), STEP 8's "test-affordance" declined - offline trace snapshot + driver tests for the pick Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aps sol-medium binding Generated-By: PostHog Desktop Task-Id: 7e06dd69-14e8-4105-b15e-655c10cc47d5
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
gewenyu99
changed the title
Opt-in via SOURCE_MAPS_RUN_BUILD=1: the host answers the test-affordance ask 'yes' and, at the test-done prompt, runs the app's real build (npm run build → posthog-cli sourcemap process) before answering honestly. Stands in for the human the skill defers the build to, so a snapshot run can exercise the actual source-map upload. Harness-only. Generated-By: PostHog Desktop Task-Id: 7e06dd69-14e8-4105-b15e-655c10cc47d5
The Next.js posthog plugin reads credentials from process.env at build time, unlike posthog-cli which self-loads. Generated-By: PostHog Desktop Task-Id: 7e06dd69-14e8-4105-b15e-655c10cc47d5
A POSTHOG_API_KEY inherited from the host shell shadowed the fixture's upload key. Generated-By: PostHog Desktop Task-Id: 7e06dd69-14e8-4105-b15e-655c10cc47d5
…urce-maps-sol-medium-e2e
gewenyu99
commented
Aug 27, 2026
Collaborator
Author
There was a problem hiding this comment.
This is the only line of prod/non test code.
Co-authored-by: Edwin Lim <edwin@posthog.com>
Co-authored-by: Edwin Lim <edwin@posthog.com>
…estion (#1161) Co-authored-by: Edwin Lim <edwin@posthog.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Vincent (Wen Yu) Ge <29069505+gewenyu99@users.noreply.github.com>
Co-authored-by: releaser-wizard[bot] <251022448+releaser-wizard[bot]@users.noreply.github.com>
…ns (#1165) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: releaser-wizard[bot] <251022448+releaser-wizard[bot]@users.noreply.github.com>
The host picks the project with `detectSourceMapsPrerequisites`, which
deliberately does not classify native platforms — so every native fixture
died at the detect screen with `{"kind":"unsupported-platform"}` after three
frames, before the run agent started. `go`, `rust`, `rust-workspace`,
`ios-pod`, `ios-spm`, `android`, `flutter`, `react-native` and
`react-native-expo` were all undrivable.
Mirror the verdict the live screen's agentic picker would reach, two ways:
the detector already names ios/android/flutter/react-native in its rejection
payload, and Go and Rust fall through as "unknown", where the identifying
manifest names them. Log the fallback when it fires so a frame never silently
misattributes a variant.
Verified by a full sweep of all 28 source-maps fixtures: 24 pass, and the
four failures are unrelated and pre-existing (two monorepo fixtures whose
result probe reads only the app root, plus next-no-posthog and python-flask,
which abort correctly in 5s).
Keeping the static detector rather than awaiting the screen's agentic
detection: the latter is more faithful but adds an LLM call to every run, and
the e2e computes the pick statically for determinism. The cost is a manifest
list here that can drift from the product's detector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
@gewenyu99 I pushed a change which unlocked some native technologies tests. Tested all of this. LGTM! |
ablaszkiewicz
approved these changes
Aug 31, 2026
…ring (#1099) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Edwin Lim <edwin@posthog.com>
Co-authored-by: releaser-wizard[bot] <251022448+releaser-wizard[bot]@users.noreply.github.com>
# Conflicts: # e2e-harness/__tests__/wizard-ci-driver.test.ts # e2e-harness/e2e-profile.ts # e2e-harness/profiles.ts # scripts/tui-host.no-jest.ts
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| /* no env file of this name */ | ||
| } | ||
| } | ||
| const r = spawnSync('npm', ['run', 'build'], { |
There was a problem hiding this comment.
Medium: Untrusted build scripts inherit CI secrets
npm run build executes the target repository's package-defined script while env contains the entire host process.env, including the PostHog and source-map credentials used by this harness. An attacker can commit a build script that reads and exfiltrates those credentials when the build-enabled source-maps route runs. Execute builds in an isolated environment with a minimal allowlist of required variables rather than inheriting the host environment.
| // profile strategy. | ||
| if (state.currentScreen === Overlay.WizardAsk) { | ||
| const q = state.pendingQuestion?.questions[0]; | ||
| const override = q ? askOverrides[programId]?.[q.id] : undefined; |
There was a problem hiding this comment.
Medium: Agent-controlled question ID releases the upload key
The agent controls both q.id and whether the question is marked sensitive. It can issue a non-sensitive question with ID api-key, causing the harness to submit SOURCE_MAPS_CLI_KEY and the ask tool to return the raw value to the model instead of vaulting it. Only provide this override for a sensitive text question, and reject repeated or mismatched requests.
PR overviewThis PR expands the source-map end-to-end snapshot harness to cover more programs, including projects that require a build step and interactive agent questions. Two credential-exposure paths remain open. Repository-defined build scripts can inherit CI credentials, while agent-controlled question metadata can cause the source-map upload key to be returned directly; either path could disclose usable secrets to untrusted code or a model. No reported issues have yet been addressed. Open issues (2)
Fixed/addressed: 0 · PR risk: 7/10 |
gewenyu99
merged commit Aug 31, 2026
add6601
into
posthog/bind-source-maps-to-sol-medium
17 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the source-maps → pi/sol-medium binding. Ports the e2e snapshot harness (e2e.json profiles, tui-host driver actions, offline trace snapshots) from the stale #767/#794 onto current main so the binding change is e2e-drivable via
/wizard-ci-snapshots.Verification runs, one PR per run (
error-tracking-upload-source-maps, project 228144):nextreact-vitenode-rawCI enablement for those runs: workbench#3617.
Generated-By: PostHog Desktop
Task-Id: 7e06dd69-14e8-4105-b15e-655c10cc47d5
Created with PostHog Desktop