feat(e2e): add the customer account checkout test on all four targets - #599
Closed
kieran-osgood-shopify wants to merge 15 commits into
Closed
Conversation
Both waits were outliers against their own neighbours and failed on consecutive CI runs on slow macOS runners. testPreloadKeepAliveFailureInvalidatesCache polled to a 2s deadline and waited 2s, leaving no slack. The neighbouring expiry test already pairs a 2s deadline with a 3s wait. testWindowOpenRequestReturnsInvalidParamsForMalformedBody was the only sendResponse wait at 1.0s; every other one in the file uses 5.0s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
The E2E suite drives four apps: Swift iOS, Kotlin Android, React Native iOS and React Native Android. Only React Native could read a deep link, and only for a cart. Widen the contract to <scheme>://e2e/<command> with the commands reset, cart and signIn, and port the parser to Swift and Kotlin so one Maestro flow drives every app. Every platform rewrites the scheme to https before parsing. Neither Foundation URLComponents nor java.net.URI accepts the underscores in the Android application id, and the React Native URL polyfill parses a host only for http(s). A test per platform pins that behaviour with both app schemes. The parser stays pure and keeps three outcomes: nil when the link belongs to another feature, a parsed command, or an error that names the bad parameter. A parameter that is present must carry a valid value. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every E2E command now enters one controller per sample. The controller composes existing primitives only, so E2E logic stays out of the cart, storefront and account code that merchants read as integration guidance. An injectable E2ECommandTarget makes the decision order testable without a network or a UI. Eight tests per platform assert an ordered list of calls, which pins the one rule that matters: selecting a buyer identity mode resets the cart, so it must run before seeding. Production code touches E2E in one named line per call site: SceneDelegate.scene(_:openURLContexts:), MainActivity, and Routes. Android needs two touch points, not one. Navigation state lives in Compose, so E2ENavigation plus one E2ENavigationEffect line lets the app self-navigate to the cart. The Maestro flow stops the app before opening the link, so a cold start delivers the intent to onCreate; both entry points route through E2EControlLinkHandler. The manifest gains a VIEW filter and singleTask. Swift gains a second URL scheme, without which openURLContexts never fires. Cart markers now match the React Native precedent on all four targets: the button carries checkout-button, the label carries cart-checkout-ready. signIn reports that it is not implemented yet. A later PR adds it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two scenario flags only picked a file, and each test already carries its
scenario in its own env block. Tags scale to the tests Phase 6 adds without a
new flag per test.
Every test now declares tags from one taxonomy: one journey (launch, cart,
checkout, account), one cost tier (smoke, full), plus optional quarantine
(flaky, wip) and platform capability (ios-only, android-only) tags.
maestro_test_tags_test.rb enforces all of it, and a platform tag must name the
capability that earns it, so nobody tags an unported test.
config.yaml excludes flaky and wip, so a quarantined test stays in the tree and
out of every run.
`dev rn e2e {ios|android}` now runs the whole tests folder and takes
--tags/--exclude-tags. Both match any listed tag, because that is how Maestro
filters. The runner also supplies E2E_APP_ID and E2E_READY_MARKER so the shared
launch smoke runs locally; a later commit unifies those names.
ci.yml gates flow edits on the ruby job, so editing a test reaches the
validator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every BrowserStack run now executes the whole e2e/tests folder and tags decide what runs inside it. Adding a test file adds no matrix rows: it only needs a tag the matrix already includes. That removes the suites x applications cross product, which grew four rows per new test. matrix.yml gains tests_path and a top-level tags block holding the default include and exclude lists. An application may override either list to adopt a test before the other three carry it. Deliberate deviation from the plan: the plan said add E2E_CONTROL_LINK per application in matrix.yml. The library derives it as "<app_id>://e2e" instead, because the deep link scheme equals the app id on all four targets. A second copy would drift from app_id. A test pins the derivation for every row. Validation rejects an include tag no test carries. Without it a typo produced a green run that executed nothing. declared_tags reads the flow headers, so both the top-level and per-application lists are checked. Also fixes three real defects found while wiring this up: - RN Android could not receive the control link at all. Its manifest restricted the custom scheme to host "cart" while the contract uses host "e2e". Host "cart" was referenced nowhere else. - The parser tests on all three platforms pinned com.shopify.checkout_kit_android_demo, which no app uses. The Kotlin sample's applicationId is com.shopify.checkoutkit.androiddemo. They now pin the four real schemes plus one unknown scheme as an explicit robustness case. - CI and local runs disagreed on variable names. Both now use E2E_APP_ID, E2E_READY_MARKER and E2E_CONTROL_LINK, and the bootstrap flow composes /cart?... from E2E_CONTROL_LINK itself. The APP_ID and CART_BOOTSTRAP_BASE_LINK bridge variables are gone. The run result payload carries include_tags and exclude_tags so a run that selected nothing is visible in the artifact instead of silently green. Verified: 163 Ruby tests, 29 Swift, 28 Kotlin, 203 jest, dev check all 12 green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first shared test that runs on Swift iOS, Kotlin Android, React Native iOS and React Native Android. It drives the `cart` control link and asserts the `cart-checkout-ready` marker, which appears only after the app parsed the link, resolved a variant, created a cart, added the line, and navigated. Adds `e2e/scripts/run_maestro`, the single Maestro invocation every local runner calls, plus `dev swift e2e` and `dev android e2e` so all four targets run locally. Verified locally: React Native iOS 23s, Swift iOS 23s, Kotlin Android 15s, React Native Android 19s. Three deliberate deviations from the plan: - `E2E_CONTROL_LINK` is derived as `<app_id>://e2e` rather than configured per application. The scheme equals the app id on all four targets, so a second copy would drift from `app_id`. - Maestro runs the workspace root, not `tests`. Maestro resolves the `flows:` glob in `config.yaml` relative to the path on the command line, so `tests` would look for `tests/tests/**` and match nothing. `tests_path` remains for tag discovery. - The flow no longer stops the app before the link. `SceneDelegate` discards a cold-start URL and Android would need launch-intent handling, so one flow line replaces native work on two platforms. The account journey needs a running app anyway. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second test of the rollout. It seeds a cart through the control link, presents checkout, closes it, and asserts the sheet is gone. Swift iOS, Kotlin Android, React Native iOS and React Native Android all run it. Assert on the close control, not on checkout copy: the React Native catalog sells a product called "Email", which the checkout field pattern matches, so copy-based assertions pass on the wrong screen. Extract flows/checkout/close.yaml so the close identifier, which differs by SDK version rather than by platform, has one home. The React Native order flow now calls it instead of carrying its own platform split. The four targets return to different screens after the sheet closes, so this test asserts dismissal only. The tests that place an order assert the cart. Add the checkout tag to the CI matrix and exclude full, because the remaining full tests place a real order and live under tests/react-native/. Pass the installed device to Maestro from both native runners. Maestro otherwise picks a device itself, and it can pick one the app is missing from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rgets Row 3 of the test rollout. The cart carries the contact details and the delivery address, so checkout asks only for payment. That makes this the shortest real order, and the first test to look at when checkout breaks. The test moves from tests/react-native/ to tests/shared/, so Swift iOS, Kotlin Android, React Native iOS and React Native Android all run it. The matrix drops the `full` exclude entry that kept the ordering tests off the native rows. Android supplied only email, phone and country in its cart, while Swift and React Native both supplied a delivery address. Checkout therefore showed an empty Delivery form on Android only. CartInput.delivery now carries a pre-selected one-time address there too, which matches Swift. The address fields come from the same .env values the other samples read. Two Maestro details drive the flow changes: - A `visibilityPercentage` below 100 normalizes to zero, so `scrollUntilVisible` stops without scrolling. `centerElement: true` replaces it. A tap that lands on the keyboard does nothing, and Maestro still reports it as completed. - Android `hideKeyboard` issues a Back press when no keyboard is up, which closes the checkout sheet. dismiss-active-field.yaml therefore only ever follows an inputText. Verified green on all four targets, plus dev check, the Ruby suite, the Swift package and sample targets, the Android library and sample tests, the React Native JS and iOS tests, web and protocol. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moves the guest checkout test out of tests/react-native and into tests/shared, so Swift, Kotlin, React Native iOS and React Native Android all run it. The old file carried a wip tag to keep it out of the matrix; both go away together. Three flow fixes came out of running it: - fill-shipping-address.yaml asked for visibilityPercentage 10 in four places. A value under 100 rounds down to zero, so those scrolls stopped before they moved. centerElement replaces all four. - The shop requires a phone number on a delivery address, and no flow filled it. The field only appears when a shop asks for it, so the new step is optional. - Checkout copies the delivery name into "Name on card", and inputText appends to it. fill-payment-card.yaml now erases the field first. Adds e2e/FLAKES.md, which records every unstable test and every trap that gives a false verdict, so we clear them before the suite ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
The test signs a test customer in through the hosted login page, seeds a cart with the customer buyer identity, and orders. Checkout reads the contact and the saved delivery address from the account, so the test asserts the prefill instead of typing an address. Three sample repairs fall out of running it: - The Swift E2E build signed nothing, so the app carried no application-identifier and every keychain write returned -34018. The cart then dropped the customer and checkout opened as a guest. The build now signs ad hoc. - generate_android_env wrote only EMAIL and PHONE, so the Android sample used its Canadian fallbacks and the billing labels never matched. It now writes the same eleven keys as the Swift and React Native generators, and test_setup_storefront_env asserts all three. - The React Native sign-in link opened Login with no AccountHome beneath it, so goBack() left the tab and the sample landed on the catalog. The link now passes initial: false. The account credentials and the login web view user agent stay out of the repository. A run without them skips the account tag instead of failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
BrowserStack answers HTTP 422 BROWSERSTACK_DUPLICATE_TAGS when one tag appears in both includeTags and excludeTags. The account skip added `account` to the exclude list and left it in the include list, so every CI row failed before it started a build. `resolve_tags` now removes the tag from the include list at the same moment it adds it to the exclude list. `run_maestro` applies the same rule locally, and it exits early when the requested tags need account credentials, because an empty include list makes Maestro run the whole suite. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
Two columns carried no information. `execute` is the tests folder on every row now that tags select the flows, so the Suite column read `.` four times. The column now names the include tags, and reads `all` when a row carries none. The failure heading and the missing-run label used the same value, so both change too. A row that fails before it starts a build reports no test cases. `result.json` already holds `error` and `error_class`, but the report dropped both, so a reader saw an empty row and had to open Bitrise. The report now names the class and the message above the table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
Six tests pass locally on all four targets and fail on BrowserStack. PR #561 and PR #562 show the same two patterns, so no sample change causes them. Both patterns need the Maestro logs, which need BrowserStack access. Separate them from the flake register, because a flake gives a different result from the same code and these give the same result every run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
Bitrise holds no address secrets, so a CI clone had no root .env, no React Native .env and no xcconfig. Every address key fell through to the Canadian default, and the generated cart carried a Toronto address into a US shop. That shop offers no delivery method for Canada, so checkout-hardcoded-buyer-identity died deep inside checkout on both Android targets. The address is the same public fixture the guest test already types, so it needs no secret. The Android sample's Gradle build carried a second copy of the same Canadian fallbacks; it now matches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
Eight BrowserStack iOS builds carrying clearState failed on five different units. Three builds without it passed, two of them on units that had just failed. The fault is deterministic, so a retry cannot recover it and the removal stands on measurement rather than on one run. Also confirms flake B4. The Android sample gives a guest cart a Canadian buyer identity, while Swift and React Native take the shop market, so the Kotlin checkout renders a Canadian form against a United States fixture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Assisted-By: devx/5b4b8d5b-690a-4389-841e-039723ade1da
This was referenced Aug 6, 2026
This was referenced Aug 6, 2026
Contributor
Author
kieran-osgood-shopify
deleted the
kieran-osgood/e2e-guest-cart-shop-market
branch
August 6, 2026 16:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are you making?
How to test
Before you merge
Important
platforms/swift/README.mdand/orplatforms/android/README.md)Releasing a new Swift version?
ShopifyCheckoutKit.podspecplatforms/swift/Sources/ShopifyCheckoutKit/ShopifyCheckoutKit.swiftplatforms/swift/README.md(major version only)Releasing a new Embedded Checkout Protocol version?
embeddedCheckoutProtocolAndroidinplatforms/android/gradle/libs.versions.tomlprotocol/languages/kotlin/embedded-checkout-protocol/api/embedded-checkout-protocol.apiif the public API changedReleasing a new Android version?
checkoutKitAndroidinplatforms/android/gradle/libs.versions.tomlplatforms/android/README.mdTip
See the Contributing documentation for the full release process per platform.