Skip to content

Replace the E2E suite cross product with Maestro tags in CI - #557

Merged
kieran-osgood-shopify merged 1 commit into
mainfrom
kieran-osgood/e2e-matrix-tags
Aug 12, 2026
Merged

Replace the E2E suite cross product with Maestro tags in CI#557
kieran-osgood-shopify merged 1 commit into
mainfrom
kieran-osgood/e2e-matrix-tags

Conversation

@kieran-osgood-shopify

@kieran-osgood-shopify kieran-osgood-shopify commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What changes are you making?

Replaced Suites (running by name) with tags
This allows us to include/exclude tests by platform
e.g. smoke test for light touch, full suite before a release, filter tags to iOS only or Android, filter by "Guest" flows etc.

Utilise the appID as the scheme for deep links, giving a consistent env var to provide to every app and make sure its unique to avoid pollution of opening the wrong app

Before you merge

Important

  • I've added tests to support my implementation
  • I have read and agree with the Contribution Guidelines
  • I have read and agree with the Code of Conduct
  • I've updated the relevant platform README (platforms/swift/README.md and/or platforms/android/README.md)
Releasing a new Swift version?
  • I have bumped the version in ShopifyCheckoutKit.podspec
  • I have bumped the version in platforms/swift/Sources/ShopifyCheckoutKit/ShopifyCheckoutKit.swift
  • I have updated the SwiftPM/CocoaPods version snippets in platforms/swift/README.md (major version only)
Releasing a new Embedded Checkout Protocol version?
  • I have bumped embeddedCheckoutProtocolAndroid in platforms/android/gradle/libs.versions.toml
  • I have updated protocol/languages/kotlin/embedded-checkout-protocol/api/embedded-checkout-protocol.api if the public API changed
Releasing a new Android version?
  • I have bumped checkoutKitAndroid in platforms/android/gradle/libs.versions.toml
  • I have updated the Gradle/Maven version snippets in platforms/android/README.md

Tip

See the Contributing documentation for the full release process per platform.

@github-actions github-actions Bot added the #gsd:50662 Rebase Checkout Kit on UCP label Jul 31, 2026
This was referenced Jul 31, 2026
Merged
Merged
@kieran-osgood-shopify Graphite App

kieran-osgood-shopify commented Jul 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

This was referenced Jul 31, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Closed
@tiagocandido
tiagocandido force-pushed the kieran-osgood/e2e-maestro-tags branch from 9fb7cc3 to f6b1c17 Compare August 3, 2026 12:12
@tiagocandido
tiagocandido force-pushed the kieran-osgood/e2e-matrix-tags branch 2 times, most recently from 05b95c0 to b63d34b Compare August 3, 2026 14:42
@tiagocandido
tiagocandido force-pushed the kieran-osgood/e2e-maestro-tags branch from f6b1c17 to 7fe2c6b Compare August 3, 2026 14:42
This was referenced Aug 3, 2026
Merged
Merged
Merged
Closed
Closed
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-maestro-tags branch from 7fe2c6b to c92b49f Compare August 4, 2026 17:04
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from b63d34b to 275339e Compare August 4, 2026 17:04
This was referenced Aug 4, 2026
Merged
Merged
Merged
Closed
Open
Closed
Open
Closed
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-maestro-tags branch 2 times, most recently from c8d37f7 to 8c9d852 Compare August 7, 2026 11:21
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch 2 times, most recently from 183a50a to 54f9911 Compare August 7, 2026 13:51
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-maestro-tags branch from 8c9d852 to 3baed39 Compare August 7, 2026 13:51
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from 54f9911 to 73013c1 Compare August 10, 2026 09:08
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-maestro-tags branch from 3baed39 to 262631b Compare August 10, 2026 09:08
@kieran-osgood-shopify
kieran-osgood-shopify marked this pull request as ready for review August 10, 2026 13:45
@kieran-osgood-shopify
kieran-osgood-shopify requested a review from a team as a code owner August 10, 2026 13:45
@github-actions

github-actions Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown

React Native — Coverage Report

Lines Statements Branches Functions
Coverage: 92%
 91.64% (307/335) 88.88% (176/198) 100% (81/81)

@github-actions

github-actions Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Package Size

Platform Artifact Base Head Delta
React Native npm tarball 101.6 KiB 101.6 KiB 0 B
Android release AAR 281.8 KiB 281.8 KiB 0 B
React Native file breakdown
File Base Head Delta
node_modules/@shopify/checkout-kit-protocol/src/generated/Models.ts 85.4 KiB 85.4 KiB 0 B
node_modules/@shopify/checkout-kit-protocol/src/generated/Models.d.ts 53.0 KiB 53.0 KiB 0 B
ios/AcceleratedCheckoutButtons.swift 14.1 KiB 14.1 KiB 0 B
ios/ShopifyCheckoutKit.swift 13.5 KiB 13.5 KiB 0 B
src/components/AcceleratedCheckoutButtons.tsx 13.0 KiB 13.0 KiB 0 B
android/src/main/java/com/shopify/reactnative/checkoutkit/ShopifyCheckoutKitModule.java 12.8 KiB 12.8 KiB 0 B
lib/commonjs/index.js 12.3 KiB 12.3 KiB 0 B
src/index.ts 12.2 KiB 12.2 KiB 0 B
lib/commonjs/components/AcceleratedCheckoutButtons.js 11.4 KiB 11.4 KiB 0 B
lib/commonjs/components/AcceleratedCheckoutButtons.js.map 10.4 KiB 10.4 KiB 0 B
lib/module/components/AcceleratedCheckoutButtons.js 10.2 KiB 10.2 KiB 0 B
lib/module/index.js 10.2 KiB 10.2 KiB 0 B
node_modules/@shopify/checkout-kit-protocol/src/generated/ProtocolNotifications.ts 9.5 KiB 9.5 KiB 0 B
src/index.d.ts 9.1 KiB 9.1 KiB 0 B
lib/module/components/AcceleratedCheckoutButtons.js.map 9.1 KiB 9.1 KiB 0 B
src/present-dispatcher.ts 8.0 KiB 8.0 KiB 0 B
lib/module/index.js.map 7.9 KiB 7.9 KiB 0 B
lib/commonjs/index.js.map 7.7 KiB 7.7 KiB 0 B
node_modules/@shopify/checkout-kit-protocol/src/generated/ProtocolNotifications.d.ts 7.6 KiB 7.6 KiB 0 B
node_modules/@shopify/checkout-kit-protocol/src/generated/ProtocolRenameMap.ts 7.2 KiB 7.2 KiB 0 B
…and 110 smaller files
Android file breakdown
File Base Head Delta
classes.jar 297.7 KiB 297.7 KiB 0 B
res/layout/checkout_view_content.xml 2.6 KiB 2.6 KiB 0 B
res/layout/checkout_sheet_content.xml 2.0 KiB 2.0 KiB 0 B
res/values/values.xml 1.3 KiB 1.3 KiB 0 B
R.txt 1.2 KiB 1.2 KiB 0 B
AndroidManifest.xml 922 B 922 B 0 B
proguard.txt 798 B 798 B 0 B
res/drawable/close.xml 431 B 431 B 0 B
res/menu/checkout_menu.xml 354 B 354 B 0 B
META-INF/com/android/build/gradle/aar-metadata.properties 157 B 157 B 0 B

Measured from the PR base SHA and PR head SHA. The file breakdown shows uncompressed sizes within each package artifact, so individual files do not sum to the compressed artifact total. This comment reports package artifact sizes only; it is not a final app binary-size report.

@bitrise

bitrise Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown

Install this build

Open Tophat, select your target device, then click Install. Links open on the Mac running Tophat.

SDK Install
React Native Install with Tophat
Swift Install with Tophat
Kotlin Install with Tophat

Checkout Kit E2E results

Status Suite Target Platform OS version tag Device
✅ react-native-ios react-native ios latest iPhone 15
iOS 27 Beta
✅ react-native-android react-native android latest Google Pixel 9
Android 17.0
✅ kotlin-android kotlin android latest Google Pixel 9
Android 17.0
✅ swift-ios swift ios latest iPhone 15
iOS 27 Beta

@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-maestro-tags branch from 262631b to 4b74857 Compare August 12, 2026 10:24
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from 73013c1 to 5cbe2a3 Compare August 12, 2026 10:24
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-maestro-tags branch from 4b74857 to 9ad8299 Compare August 12, 2026 11:12
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from 5cbe2a3 to ba3548d Compare August 12, 2026 11:12
@github-actions

github-actions Bot commented Aug 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Web — Coverage Report

Lines Statements Branches Functions
Coverage: 96%
 94.62% (299/316) 81.72% (152/186) 98.71% (77/78)

@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-maestro-tags branch 2 times, most recently from f79b83b to 522faec Compare August 12, 2026 12:32
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from ba3548d to a87df2e Compare August 12, 2026 12:32
@kieran-osgood-shopify
kieran-osgood-shopify changed the base branch from kieran-osgood/e2e-maestro-tags to graphite-base/557 August 12, 2026 13:49
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from a87df2e to 5203657 Compare August 12, 2026 13:49
@graphite-app
graphite-app Bot changed the base branch from graphite-base/557 to main August 12, 2026 13:50
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from 5203657 to fb92dd6 Compare August 12, 2026 13:50
@kieran-osgood-shopify @claude
Replace the E2E suite cross product with Maestro tags in CI
f88ff08 
Every BrowserStack run now executes the whole e2e/tests folder and tags decide
what runs inside it. Adding a test file adds no matrix rows: it only needs a tag
the matrix already includes. That removes the suites x applications cross
product, which grew four rows per new test.

matrix.yml gains tests_path and a top-level tags block holding the default
include and exclude lists. An application may override either list to adopt a
test before the other three carry it.

Deliberate deviation from the plan: the plan said add E2E_CONTROL_LINK per
application in matrix.yml. The library derives it as "<app_id>://e2e" instead,
because the deep link scheme equals the app id on all four targets. A second
copy would drift from app_id. A test pins the derivation for every row.

Validation rejects an include tag no test carries. Without it a typo produced a
green run that executed nothing. declared_tags reads the flow headers, so both
the top-level and per-application lists are checked.

Also fixes three real defects found while wiring this up:

- RN Android could not receive the control link at all. Its manifest restricted
  the custom scheme to host "cart" while the contract uses host "e2e". Host
  "cart" was referenced nowhere else.
- The parser tests on all three platforms pinned
  com.shopify.checkout_kit_android_demo, which no app uses. The Kotlin sample's
  applicationId is com.shopify.checkoutkit.androiddemo. They now pin the four
  real schemes plus one unknown scheme as an explicit robustness case.
- CI and local runs disagreed on variable names. Both now use E2E_APP_ID,
  E2E_READY_MARKER and E2E_CONTROL_LINK, and the bootstrap flow composes
  /cart?... from E2E_CONTROL_LINK itself. The APP_ID and
  CART_BOOTSTRAP_BASE_LINK bridge variables are gone.

The run result payload carries include_tags and exclude_tags so a run that
selected nothing is visible in the artifact instead of silently green.

Verified: 163 Ruby tests, 29 Swift, 28 Kotlin, 203 jest, dev check all 12 green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Assisted-By: devx/62deda35-c064-4e16-99eb-02b2792eaf51
@kieran-osgood-shopify
kieran-osgood-shopify force-pushed the kieran-osgood/e2e-matrix-tags branch from fb92dd6 to f88ff08 Compare August 12, 2026 13:57
@kieran-osgood-shopify
kieran-osgood-shopify merged commit 4a7e58f into main Aug 12, 2026
37 checks passed
@kieran-osgood-shopify
kieran-osgood-shopify deleted the kieran-osgood/e2e-matrix-tags branch August 12, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kyle-schellen kyle-schellen kyle-schellen approved these changes

Assignees

No one assigned

Labels

#gsd:50662 Rebase Checkout Kit on UCP

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kieran-osgood-shopify @kyle-schellen