Skip to content

[build-tools] upload maestro screenshots#3872

Open
hSATAC wants to merge 1 commit into
mainfrom
ash/eng-19036-upload-maestro-screenshots
Open

[build-tools] upload maestro screenshots#3872
hSATAC wants to merge 1 commit into
mainfrom
ash/eng-19036-upload-maestro-screenshots

Conversation

@hSATAC

@hSATAC hSATAC commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Depends on https://github.com/expo/universe/pull/28185 — www must persist the artifact metadata this PR sends.

Why

Surface Maestro per-flow failure screenshots in the workflow Maestro test results UI, shown next to the failing test-case row. Part of ENG-19036.

How

  • build-tools: After each Maestro attempt (junit output only), harvest failure screenshots from the debug dirs, then reduce to what the UI actually surfaces: a flow that failed on every attempt uploads only its final-attempt screenshot, while a flaky/mixed flow keeps every failed-attempt screenshot (classified from the per-attempt JUnit pass/fail, not screenshot counts). The kept screenshots upload as workflow artifacts carrying metadata (kind, flowName, attemptIndex, capturedAtMs) that the website matches to the corresponding test-case result. Best-effort and verdict-neutral: errors are logged rather than thrown, and uploads are capped at 30 per job.
  • worker: Thread an optional metadata field through the artifact upload path (uploadArtifactuploadWorkflowArtifactAsynccreateUploadSessionAsync) so it reaches www.

Test Plan

Unit tests added/updated (maestroScreenshots.test.ts, maestroTests.test.ts), all passing in CI — covering filename parsing, the mtime/capturedAt harvest gating, the pure-vs-flaky reduction (final-only for all-failed flows, every attempt for flaky), the 30-upload cap, upload-before-failure-throw, and per-item error swallowing.

@hSATAC hSATAC added the no changelog PR that doesn't require a changelog entry label Jun 17, 2026
@linear-code

linear-code Bot commented Jun 17, 2026

Copy link
Copy Markdown

ENG-19036

@codecov

codecov Bot commented Jun 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.91837% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.55%. Comparing base (84015eb) to head (173cd06).

Files with missing lines Patch % Lines
...ld-tools/src/steps/functions/maestroScreenshots.ts 93.75% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3872      +/-   ##
==========================================
+ Coverage   58.46%   58.55%   +0.09%     
==========================================
  Files         922      923       +1     
  Lines       40237    40332      +95     
  Branches     8475     8490      +15     
==========================================
+ Hits        23520    23611      +91     
- Misses      16621    16625       +4     
  Partials       96       96              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hSATAC hSATAC force-pushed the ash/eng-19036-upload-maestro-screenshots branch from b888f27 to 863bd75 Compare June 17, 2026 14:07

hSATAC commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@hSATAC hSATAC marked this pull request as ready for review June 17, 2026 14:21
@hSATAC hSATAC requested a review from sjchmiela June 17, 2026 14:21
@github-actions

Copy link
Copy Markdown

Subscribed to pull request

File Patterns Mentions
**/* @douglowder

Generated by CodeMention

@hSATAC hSATAC force-pushed the ash/eng-19036-upload-maestro-screenshots branch from 863bd75 to 5fe7e8d Compare June 18, 2026 06:36
Comment thread packages/build-tools/src/context.ts Outdated
Comment thread packages/build-tools/src/steps/functions/maestroTests.ts Outdated
Comment thread packages/build-tools/src/steps/functions/maestroTests.ts Outdated
Comment thread packages/build-tools/src/steps/functions/maestroTests.ts Outdated
Comment thread packages/build-tools/src/steps/functions/maestroTests.ts Outdated
Comment thread packages/build-tools/src/steps/functions/maestroTests.ts Outdated
Comment thread packages/build-tools/src/steps/functions/maestroTests.ts Outdated
Comment thread packages/build-tools/src/steps/functions/maestroScreenshots.ts
Comment thread packages/build-tools/src/steps/functions/maestroScreenshots.ts Outdated
Comment on lines +44 to +45
// Scans every debug dir under testsDirectory whose mtime is at/after dirSinceMtimeMs (mtime-based
// to cover the maestro <2.5.0 bug that splits one invocation across two dirs), then keeps only

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's remember to drop any additional complexity as soon as we decide to drop the extra support for <2.5?

or maybe we want to do it already?

@hSATAC hSATAC Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a Linear ticket for that.

Around 75% of users are on 2.6+ today. If we drop the old path now, ~25% of users would lose these features immediately.

I was thinking something closer to 90%+ adoption before removing it, but happy to discuss.

@hSATAC hSATAC force-pushed the ash/eng-19036-upload-maestro-screenshots branch from 5fe7e8d to 4c1cb19 Compare June 18, 2026 14:19
Signed-off-by: Ash Wu <hsatac@gmail.com>
@hSATAC hSATAC force-pushed the ash/eng-19036-upload-maestro-screenshots branch from 4c1cb19 to 173cd06 Compare June 18, 2026 14:19
@github-actions

Copy link
Copy Markdown

⏩ The changelog entry check has been skipped since the "no changelog" label is present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no changelog PR that doesn't require a changelog entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants