test-matrix.yml

# The cross-platform test matrix.
#
# One row per test that the app test suites implement. This file is the
# parity mechanism for testing, the way spec/*.yml is for the file format:
# Android and iOS must cover the same rows (minus recorded platform
# differences), and tools/check_test_matrix.py - run by the docs build when
# the app checkouts sit next to this repository - verifies it mechanically
# instead of by discipline.
#
# Each app test carries its row id in a tag comment in its source:
#
#     // phyphox-test: corpus-valid-load        (Kotlin/Java)
#     // phyphox-test: corpus-valid-load        (Swift)
#
# The checker greps both repos for these tags. A tag that names no row here
# fails the build (typo guard). A row's meaning by status:
#
#   planned  - agreed and listed, not yet implemented everywhere. Missing
#              tags are reported as information, never as a failure.
#   active   - implemented on every platform the row names. From then on a
#              missing tag FAILS the docs build: a deleted or renamed test
#              is a regression, not a cleanup. Rows are flipped to active by
#              the docs session once both platforms carry the tag.
#
# Fields: id (kebab-case, the tag), area (a short topic label), tier (T0-T3),
# platforms (subset of [android, ios]; name only one for recorded platform
# differences and say why in the description), status, description, and the
# optional host_driven: true for a row implemented entirely in this
# repository's tools/ - for those, and only those, a tag found there counts
# as the implementation for every platform the row names.
#
# Open question (2026-08-31, from the editor audit): the Blockly editor now
# also runs the corpus - its AUDIT=true vitest suite round-trips every
# corpus file against this repo's spec. Whether the editor becomes a third
# platform here (with the checker grepping phyphox-blockly-editor too, and
# per-row applicability, since a generator-and-parser covers different rows
# than a parser) is the maintainer's call; recorded so the idea is not lost.

tests:

  - id: corpus-valid-load
    area: file-format
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      Every file in corpus/valid and corpus/generated whose declared format
      version is at most the platform's supported version loads without
      error. Files declaring a newer version are skipped, not failed - they
      exist for future format versions. A file with an entry in the
      directory's expected.yml deviates per platform: the platform mapped
      to rejects asserts the refusal instead (deliberate platform
      differences, e.g. bluetooth address= on iOS). See corpus/README.md,
      "The app test suites".

  - id: corpus-invalid-reject
    area: file-format
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      Every corpus/invalid file whose expected.yml entry says
      `parser: rejects` fails to load. Any error is acceptable; error
      message texts are platform wording and are not asserted.

  - id: corpus-tolerated-attributes-load
    area: file-format
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      Every corpus/invalid file whose expected.yml entry says
      `parser: accepts` LOADS successfully: its defects are tolerated for
      compatibility per the unknown-attribute-ignored and
      duplicate-metadata-last-wins rules (spec/rules.yml). This pins the
      compatibility guarantee - a parser that starts rejecting unknown
      attributes or legacy duplicates breaks files in the wild.

  - id: analysis-golden-vectors
    area: analysis
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      Every case under corpus/analysis/vectors/ (a mini-experiment plus an
      .expected.json) loads through the real parser, its analysis block is
      executed the stated number of cycles, and the listed output buffers
      match the expected values - NaN-aware, within the stated tolerances.
      Cases whose file declares a newer format version than the platform
      supports are skipped, like corpus-valid-load. See
      corpus/analysis/README.md for the runner contract.

  - id: corpus-version-gate
    area: file-format
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      A file declaring a format version newer than the app supports (the
      supported version + 0.1) is refused with the app's file-too-new
      handling, and one declaring the exact supported version loads. Guards
      the version gate that all feature rollout relies on.

  - id: remote-contract-t1
    area: remote-interface
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      tools/contract_test.py in single-target mode against the platform's
      emulator/simulator build: every response validates against the
      OpenAPI schemas and the annotated contract semantics (exact statuses
      and result values). Host-driven - the phyphox-test tag lives in the
      repo's T1 CI workflow, not in app source. Phone-vs-phone two-target
      runs stay the release gate.

  - id: network-http
    area: network
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      The http fixture experiments (phyphox-docs fixtures/network/, served
      by tools/network_fixture.py): receive sequence, GET and POST send
      round trips, malformed-response and server-down error paths - no
      crash, no hang, buffer contents asserted through the remote API.
      See fixtures/network/README.md for the runner contract including
      the FIXTURE-HOST substitution.

  - id: network-mqtt
    area: network
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      The mqtt fixture experiment against a local mosquitto broker
      (fixtures/network/mosquitto.conf): publish/subscribe loopback
      through a real broker; the TLS variants join once the mqtts
      fixtures exist.

  - id: experiments-e2e-t1
    area: experiments
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      tools/t1_experiments.py over the shipped collection on an emulator/
      simulator: open via phyphox://asset=, start, run, stop, download and
      validate every export format (tools/validate_export.py), app stays
      responsive. Android with emulator-injected sensors. iOS covers the
      loadable subset: the driver skips audio-input experiments (they
      abort the simulator app) and classifies sensor-unavailable ones as
      not-loadable rather than failed - recorded platform difference.
      Host-driven - the tag lives in the T1 CI workflow. Uses the
      remote-enable and auto-confirm switches.

  - id: switch-bypassed-ui
    area: ui
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      Everything the host-controlled switches bypass must still work
      WITHOUT them (the switches must never mask a broken user path):
      the per-experiment menu toggle enables the remote server with
      debug.phyphox.remote / -phyphoxRemote unset; without auto-confirm
      the network privacy notice appears and is dismissed by its single
      OK - deliberately informational, not a consent gate (no decline
      exists: opening an experiment records nothing sensitive, and a
      connection before start has nothing meaningful to transmit), so a
      decline control turning up is a finding, and the privacy-policy
      link works when the experiment sets one; the
      photosensitivity warning appears, and the
      save-downloaded-experiment offer appears and works both ways;
      opening a phyphox:// URL the regular way (not via -phyphoxUrl)
      still reaches the app, confirmation dialog included on iOS.
      UI-automated - lands with the phase-4 UI suites.

  - id: view-snapshots
    area: ui
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      Golden-image tests of the non-graph view elements over the
      phyphox-docs fixtures/views/ set (values, edits, buttons-toggles,
      sliders-dropdowns, info-separator-image), rendered in the
      configuration matrix of fixtures/views/README.md (light/dark, two
      font scales, phone/tablet width, RTL layout smoke). Android:
      Robolectric; iOS: swift-snapshot-testing. Goldens live in the app
      repos.

  - id: graph-snapshots
    area: ui
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      Golden-image tests of the OpenGL graph rendering over
      fixtures/views/graphs-*.phyphox (styles, axes incl. log and fixed
      scales, NaN gaps, history, follow-x, color maps, pick labels).
      Android: emulator + PixelCopy; iOS: GLKView snapshot in the
      simulator. Goldens live in the app repos.

  - id: view-behavior
    area: ui
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      Interaction tests over the same fixtures: typing into edits
      (bounds, signed/decimal, factor), pressing buttons (value writes,
      appends, empty-clear, trigger), toggles, both slider types,
      dropdown selection - each asserted by its buffer effect through
      the remote API. Also the starting state, which no interaction
      touches: a container's init beats a control's default, while a
      default still fills an empty container, and both survive the
      views being rebuilt. Espresso / XCUITest.

  - id: app-chrome
    area: ui
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      The screens around the experiment: collection (categories,
      ordering, platform hiding), experiment menu and all its actions,
      dialogs (timed run, export, remote access incl. privacy note,
      save state, delete/rename), settings, about, hints, rotation per
      screen, tablet layouts, permission flows (grant AND deny paths).

  - id: lifecycle
    area: robustness
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      Rotation mid-run, background/foreground with a running experiment,
      screen-off with keep-on, kill and relaunch, permission denied
      yields a sensible error UI, timed-run edges, rapid start/stop,
      opening a second experiment, low-memory smoke, remote access
      toggled mid-run.

  - id: accessibility-smoke
    area: ui
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      Espresso AccessibilityChecks over the chrome suite on Android;
      performAccessibilityAudit() on the main iOS screens. Report-only
      at first; escalates to failing after the findings are triaged.

  - id: translations-build
    area: translations
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      Language handling follows the build: the enabled set (LOCALE_ARRAY
      / Bundle.main.localizations) loads, deviations from the canonical
      languages.yml in phyphox-docs and between the platforms are
      WARNINGS in the report, never failures. The hard enforcement
      against built artifacts is the T2 release check.

  - id: translations-ui
    area: translations
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      UI regression per build-enabled language: the collection screen, a
      representative experiment screen and the critical menus/dialogs
      render without truncation/overlap regressions in every enabled
      language (snapshot or layout-assertion based, the platform's
      choice).

      Splittable across CI jobs by a convention both platforms share, so
      the shards stay comparable: sort the build's language set, then
      honour PHYPHOX_TEST_LANGUAGE_SHARD=i/n (1-based, round-robin over
      that sorted list) and PHYPHOX_TEST_LANGUAGES=de,fr (an explicit
      subset, for reproducing one language). A code named in
      PHYPHOX_TEST_LANGUAGES that the build does not enable FAILS rather
      than being skipped - a typo must not quietly remove coverage. With
      neither variable set, every enabled language runs, which is what a
      local run and a release run do.

  - id: device-sensors
    host_driven: true        # implemented once in phyphox-docs tools/lab/
    area: hardware
    tier: T2
    platforms: [android, ios]
    status: active
    description: >-
      Per-device sensor truth on the seven lab phones, manifest-driven
      (tools/lab/devices/): liveness and achieved rate FAIL; the value
      plausibility windows (|accel| at rest, gyro near zero, earth-field
      magnitude, pressure range, light positive) WARN only - bench
      magnetics and stale calibration violate them routinely, and no
      plausible app bug changes values while keeping the rate right.
      Graceful GPS no-fix indoors. Implemented once host-side in
      tools/lab/ - the tag there counts for both platforms.

  - id: device-audio
    host_driven: true        # implemented once in phyphox-docs tools/lab/
    area: hardware
    tier: T2
    platforms: [android, ios]
    status: active
    description: >-
      The self-contained audio loopback (fixtures/audio/loopback.phyphox)
      per device: speaker plays 1 kHz while the microphone records, the
      FFT peak matches (corrected by the achieved rate), level above the
      floor. Covers audio input, output and analysis in one test.

  - id: device-experiments
    host_driven: true        # implemented once in phyphox-docs tools/lab/
    area: hardware
    tier: T2
    platforms: [android, ios]
    status: active
    description: >-
      The full shipped experiment matrix on every lab device through
      tools/t1_experiments.py: open via phyphox://asset=, run, stop, all
      six export formats validated. Bluetooth experiments load-phase
      only (the phase-6 board lab owns their data plane).

  - id: device-languages
    host_driven: true        # implemented once in phyphox-docs tools/lab/
    area: translations
    tier: T2
    platforms: [android, ios]
    status: active
    description: >-
      The release language gate: the BUILT artifact's locale set (aapt
      dump badging / the .lproj set) must equal languages.yml exactly -
      a missing language or a testing locale left in FAILS. Runs when
      the lab host names the release artifacts.

  - id: containers-load
    area: file-format
    tier: T0
    platforms: [android, ios]
    status: active
    description: >-
      The container forms through the real intake route
      (fixtures/containers/): the multi-experiment zip unpacks to
      exactly its two experiments, the resource zip delivers its image
      to the experiment, the traversal zip's ../ entry is REJECTED with
      nothing written outside the extraction directory (security pin),
      and the headerless partial zip (the QR/BLE form) is detected by
      its trailing signature, rebuilt and loaded. See
      fixtures/containers/README.md.

  - id: save-to-collection
    area: ui
    tier: T1
    platforms: [android, ios]
    status: active
    description: >-
      Saving external experiments to the collection - the flow the
      auto-confirm switch deliberately declines - driven by UI
      automation accepting the offer: the with-resource zip lands in
      the collection with its image extracted into the CRC32-named
      resource folder and reopens intact; the multi-experiment zip
      saves both via the picker. Hermetic: the saved entries are
      deleted at the end.

  - id: ble-compat-arduino
    area: hardware
    tier: T2
    platforms: [android, ios]
    status: active
    description: >-
      A release must not break Arduino projects that worked with the
      previous one. Seven library examples, flashed UNMODIFIED as
      stimulus (the library is not under test and is due its own
      session), are driven end to end: discovery with another device
      advertising nearby, the experiment transferred over cddf0002,
      data decoded, and the phone-to-board direction observed on the
      board's own serial output. Four scenarios run on every phone
      because Android's BLE behaviour varies by version and vendor; the
      rest on the newest phone of each platform. Definitions in
      fixtures/ble/scenarios.yml, baselines recorded from the store
      release. Release-gated: this is the slowest suite in the lab.

  - id: ble-compat-micropython
    area: hardware
    tier: T2
    platforms: [android, ios]
    status: active
    description: >-
      The same guarantee for MicroPython projects, on the ESP32 (the
      only board that library supports): all three of its examples, with
      randomNumbers and getDataFromSmartphone on every phone and
      createExperiment on the newest of each platform.

  # ---------------------------------------------------------------- T3 --
  # The human checklist, run once per platform at release. These rows carry
  # `manual: true`: there is no code to tag, so the checker never asks for
  # one, and tools/lab/run.py --merge appends them to the release report as
  # a checklist. They are here rather than in a document of their own for
  # the same reason everything else is: one list, checked mechanically, so
  # a step cannot quietly disappear.
  #
  # The budget is 30 minutes per platform. Anything that grows past that
  # belongs in T2 as automation, not here as more reading.
  #
  # Two rows were taken out again on 2026-08-31, and both for the same
  # reason - a checklist step nobody can actually perform is worse than no
  # step at all, because the list then reads as if someone were watching:
  #
  # * audio-call-interruption (a phone call during an audio experiment).
  #   Almost none of the lab phones have a mobile subscription, so it would
  #   be skipped on nearly every run. Audio interruption handling is left to
  #   user reports and to colleagues testing deliberately on their own
  #   phones.
  #
  # store-screenshots was taken out with them and came back on 2026-09-02,
  # together with store-texts, once the fastlane lanes were replaced by
  # scripts a person runs at the release (tools/store_screenshots.py and the
  # two uploaders). Both carry `optional: true`: they are release
  # preparation rather than tests, most releases do not need them, and an
  # unticked box is a normal outcome. The checklist prints them in their own
  # group, outside the 30-minute budget, and RELEASE.md has the procedure.

  - id: gps-real-fix
    area: sensors
    tier: T3
    platforms: [android, ios]
    status: active
    manual: true
    description: >-
      Outdoors, with a real satellite fix: the GPS experiment shows a
      position, an accuracy and a satellite count that settle within a
      minute or two, and a short walk moves the coordinates, the speed and
      the distance travelled. T2 covers only the indoor no-fix path, which
      is a different code path and proves nothing about a working fix.

  - id: qr-scan-printed
    area: transfer
    tier: T3
    platforms: [android, ios]
    status: active
    manual: true
    description: >-
      Scan the codes on the transfer page
      (https://phyphox.org/docs/transferring-experiments/, section "Online
      QR-Codes" and "Offline QR-Codes"), once from a printout and once from
      a second screen, using the app's own scanner. Five codes are there and
      each is a different path: an https link, the same link under the
      phyphox:// scheme, one code holding a whole experiment, and a two-code
      set that has to be assembled. Each opens an experiment whose title
      says which route delivered it, so a wrong result is visible without
      reading anything. Covers the camera path, the partial-zip decoding and
      the multi-code assembly, none of which a URL opened from the shell
      touches. The codes are generated from the example experiments by
      tools/generate_qr.py and verified on every docs build; the link codes
      need the site to be published.

  - id: ble-third-party-device
    area: hardware
    tier: T3
    platforms: [android, ios]
    status: active
    manual: true
    description: >-
      Connect to a Bluetooth device nobody wrote for this test - a heart
      rate strap, a thermometer, whatever is in the room - and confirm the
      scan lists it, an experiment for it loads, and disconnecting is
      graceful. The lab's boards all speak the phyphox protocol correctly
      by construction; this is the only step where the app meets hardware
      that does not.

  - id: accessibility-spot-check
    area: ui
    tier: T3
    platforms: [android, ios]
    status: active
    manual: true
    description: >-
      With TalkBack or VoiceOver on, reach an experiment from the
      collection, start and stop it, and read one value. Not an audit - a
      spot check that the main path is operable, which is how the invisible
      add-experiment menu was found in 2026-08.

  - id: store-screenshots
    area: release
    tier: T3
    platforms: [android, ios]
    status: active
    manual: true
    optional: true
    description: >-
      Only if the app's appearance changed, or a scene was reworked:
      regenerate the store screenshots for every form factor from the
      revision being shipped, look at them, and upload them. They are
      composed from the shipped experiment files, so a changed experiment
      changes the picture. See "Updating the store listing" in RELEASE.md;
      uploading is not releasing, and on Play a commit sends the listing
      for review.

  - id: store-texts
    area: release
    tier: T3
    platforms: [android, ios]
    status: active
    optional: true
    manual: true
    description: >-
      Only if the listing text changed in phyphox-translation, or a
      language gained a translation or a store listing: sync the titles and
      descriptions from the store PO files, checking that nothing overruns
      Play's limits (30/80/4000 characters) after translation. That
      repository is read from and never written to. Apple's subtitle and
      keywords have no PO source and stay manual.

  - id: permissions-fresh-os
    area: ui
    tier: T3
    platforms: [android, ios]
    status: active
    manual: true
    description: >-
      On a factory-fresh install, walk the permission dialogs - location,
      microphone, camera, Bluetooth - and confirm each is asked for at the
      right moment with a comprehensible reason, and that declining leaves
      the app usable. Once per major OS release rather than every release:
      the wording and the timing are the OS's, and they change with it.