diff --git a/.claude/skills/film/SKILL.md b/.claude/skills/film/SKILL.md new file mode 100644 index 0000000..4bf57ff --- /dev/null +++ b/.claude/skills/film/SKILL.md @@ -0,0 +1,101 @@ +--- +name: film +description: >- + Making the videos: the website's teaser (Tools/film/teaser.rb) and the App + Store app previews for iPhone, iPad, Mac and Vision Pro + (Tools/film/previews.rb) — recording UI tests on the simulators and on this + Mac, cutting them to Apple's rules, checking the result, and handing the + films over for music and upload. Load this before reshooting either, before + changing what a film shows, and whenever a film run fails or a film looks + wrong. Still pictures are the `screenshots` skill. +--- + +# Films + +Two kinds, one set of tools in `Tools/film/`. The scripts are UI tests. The +Ruby records them, cuts out the waiting, and edits. **`Tools/film/README.md` is +the why** — every rule, number and trap is explained there, so read it before +changing anything. This file is the how. + +```bash +ruby Tools/film/teaser.rb # the website's teaser, ~15 min +ruby Tools/film/previews.rb # iphone ipad mac vision, ~5 min each +ruby Tools/film/previews.rb ipad mac # only the ones named +ruby Tools/film/previews.rb --compose # re-cut the last recordings, no shooting +``` + +Every film comes out English, silent and uncommitted, in `$TMPDIR/tortoise-teaser/` +or `$TMPDIR/tortoise-previews/`. The maintainer adds the music by hand. The +previews are uploaded by hand as well: fastlane's deliver does not take them. + +## Before a run + +- **One rig at a time.** The films share DerivedData with the screenshot rigs; + check `pgrep -f 'shots.rb|film/'` first. +- **Tell the maintainer before the Mac preview, and wait for them.** It drives + this Mac's real pointer for three or four minutes, the screen has to stay + unlocked, and macOS asks for the password before a UI test may take the + pointer (`automationmodetool` says "requires user authentication"). The + maintainer chose to type it each time rather than switch the check off, so + say when the dialog is coming, and say again when the Mac is free. The + simulator films need nothing from them. +- A run changes the Simulator's `ConnectHardwareKeyboard` and the simulators' + language, and puts both back when it ends. After an interrupted run, check + `defaults read com.apple.iphonesimulator ConnectHardwareKeyboard` is back to + what it was (0 here). + +## Changing what a film shows + +- **The story is the test.** `TeaserTests`, `AppPreviewTests` (iPhone, iPad) + and `MacPreviewTests`. Pauses only give the app time to settle. The pace is + set by the cut, and `linger(_:)` is how a script asks for a moment to be held. +- **A preview starts from a document that is already nearly a program**, + written in `PREVIEWS` in `previews.rb`. Keep a preview to 15–30s: a short + cut holds its last frame, and a long one stops the run. Vision Pro has no + script, only launch arguments (sample, sheet, speed) in the same table. +- **What a preview may show is Apple's call, not ours**: the screen as + captured, with no zoom, backdrop, cards or captions. The README's "The + previews" section lists the rules. The teaser has none of those limits. + +## After a run + +Nothing counts as done until someone has watched it, and that someone is the +maintainer. Movement cannot be judged from stills: a sheet of frames has +already passed legs as walking that barely moved, and the maintainer caught it +on playback. + +1. **Check the file against the spec.** Run `ffprobe -show_entries + stream=codec_name,width,height,r_frame_rate,level:format=duration`. It should + show H.264, the device's exact size, 30fps, 15–30s and an AAC track. The + sizes are iPhone 886×1920, iPad 1600×1200, Mac 1920×1080 and Vision Pro + 3840×2160 (Level 5.1; the others 4.0). +2. **Look at a contact sheet** for the story and for anything wrong in a frame, + such as a home screen, a keyboard, or a sheet that never opened: + `ffmpeg -i f.mp4 -vf fps=1,scale=320:-1,tile=6x4 -frames:v 1 sheet.png`. +3. **Spot-check the touches** at full size over a second or two around a + press. The ring should land on the control as it lights, not after it. +4. **Hand the films over.** Copy them out of `$TMPDIR` to somewhere the + maintainer can open, give the links, and say what cannot be judged from + stills. For previews, remind them of two things at upload: the poster frame + defaults to 5s, which is mid-story, so pick the finished drawing; and + Vision Pro's Level 5.1 file has to be checked against what App Store Connect + accepts. + +## When a run fails + +- **A simulator script stopped.** The reason is in the result bundle: + `xcrun xcresulttool get test-results tests --path /test.xcresult`, then + the "Failure Message" nodes. `failure.txt` beside it is the screen the test + last saw, with frames. Read it before guessing. +- **The Mac script stopped.** Its runner is sandboxed, so there is no + `failure.txt`. Export the attachments (`xcresulttool export attachments`): + XCTest's own UI hierarchy dumps and screen recording are among them. + "Timed out while enabling automation mode" means the password dialog went + unanswered. +- **Vision Pro gives `TBNotReady` every time.** Something is recording the + simulator while the app launches. The recorder must start after `TBReady`. +- **A film cuts to the home screen, or its touches drift.** The conversion has + lost `-fflags +igndts`. +- **A cut is far longer than expected.** Something keeps drawing frames: + usually a blinking caret outside a `typing` span, or on the Mac a recording + judged by frames rather than by content. diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md index 3066829..1753b65 100644 --- a/.claude/skills/release/SKILL.md +++ b/.claude/skills/release/SKILL.md @@ -169,7 +169,10 @@ size of an Apple TV one, and that collision is a real trap further down. pass every reshoot has to end with, and the traps that produce a picture of the wrong thing. Nothing reaches App Store Connect without going through it: every source this project shoots from writes an alpha channel, which Apple refuses. -The text is `appstore/metadata//`, one file per field — **except +**App previews are the one piece of the listing that fastlane does not +carry.** deliver uploads screenshots but not previews, so they are made by the +`film` skill and put up by hand in App Store Connect, and nothing in +`appstore/` holds them. The text is `appstore/metadata//`, one file per field — **except visionOS**, which is pushed from `appstore/metadata-visionos/` instead (#53). That split is not tidiness: the App Store shows a Vision Pro shopper the visionOS description and nothing else, and the app is a different product diff --git a/.claude/skills/screenshots/SKILL.md b/.claude/skills/screenshots/SKILL.md index b058651..266c92a 100644 --- a/.claude/skills/screenshots/SKILL.md +++ b/.claude/skills/screenshots/SKILL.md @@ -7,9 +7,9 @@ description: >- (Tools/ipad-shots.rb, Tools/macos-shots.rb, Tools/visionos-shots.rb), the flatten-and-optimise pass (Tools/screenshots.rb), what a sendable capture has to be, and the traps that - make a screenshot tool fail silently. Also the website's teaser video - (Tools/teaser/). Load this before reshooting, before adding a shot or a - platform, and whenever a capture looks wrong. + make a screenshot tool fail silently. Load this before reshooting, before + adding a shot or a platform, and whenever a capture looks wrong. Videos — + the website's teaser and the App Store previews — are the `film` skill. --- # Screenshots @@ -259,19 +259,12 @@ Seven ways the Mac differs from the iPad, all handled but all worth knowing: what makes 1280×800pt — 2560×1600px — reproducible. Keep that `defaultSize`: a capture at any other size would need cropping or resampling. -## The teaser video +## The films -The website's two-minute video is made the same way: `ruby Tools/teaser/teaser.rb` -records `TortoiseBlocksUITests/TeaserTests.swift` on the 11-inch iPad simulator -and cuts it into a silent 1080p film; music goes on by hand afterwards. The -test is the script, and it logs every press, caption and camera move for the -editor to work from. `Tools/teaser/README.md` has why each piece is as it is. -It shares DerivedData with the rigs above, so it counts as one of them: one at -a time. - -Judge it the way the stills are judged, but on the film: a contact sheet -shows what is in it (`ffmpeg -i teaser.mp4 -vf fps=1/3,scale=320:-1,tile=6x9`), -and only playing it shows whether it moves well. +The website's teaser and the App Store previews are made the same way as the +iPad and Mac captures, with UI tests pressing and a script recording, and they +share DerivedData with these rigs. So they count as rigs: one at a time. How +to make them is the `film` skill. ## Judging the result diff --git a/App/Views/Viewer/CLAUDE.md b/App/Views/Viewer/CLAUDE.md index f31f028..bc61dbc 100644 --- a/App/Views/Viewer/CLAUDE.md +++ b/App/Views/Viewer/CLAUDE.md @@ -90,7 +90,11 @@ tortoise also turns, and a turn changes the silhouette of every leg from frame to frame; a sheet of consecutive frames looked like legs moving when they barely were, and it was the maintainer who saw that they were not. Holding two opposite moments of the cycle and comparing them is the honest still; the -honest check is `-TBPlay YES -TBSpeed 1` (×0.2) and `simctl io recordVideo`. +honest check is `-TBPlay YES -TBSpeed 1` (×0.2) and `simctl io recordVideo` — +started once the app is up, never before the launch: a recording running +through it keeps the immersive space from opening at all (`TBNotReady`, every +time). To catch a drawing from its first line, `-TBPlay ` holds it +until the recorder has started, and `TBPlaying` says when it went. **The visionOS simulator shows all of this** — paper, drawing and tortoise — and this note said the opposite for a while, which is worth keeping as a correction rather than an edit. The symptom was real: a blank sheet, a nil diff --git a/App/Views/Viewer/ViewerWindow.swift b/App/Views/Viewer/ViewerWindow.swift index 4087b2c..2e3533d 100644 --- a/App/Views/Viewer/ViewerWindow.swift +++ b/App/Views/Viewer/ViewerWindow.swift @@ -99,8 +99,9 @@ // put the drawing down // -TBSample square | star | spiral | tree // -TBDraw <0…1> run the drawing that far and stop - // -TBPlay YES then keep playing from there, for a - // recording of the tortoise walking + // -TBPlay YES| then keep playing from there, for a + // recording of the tortoise walking — + // after seconds, if a number // -TBSpeed the transport's speed, 1…10 (5 is ×1) // -TBSheet s,r,d the sheet's side, how far ahead of the // eyes it lands, and how far below them @@ -116,7 +117,12 @@ // which a still cannot show: the check is a recording // (`simctl io recordVideo`) of a drawing that keeps playing, // usually at level 1 (×0.2), where a step lasts long enough to - // see the legs change feet. + // see the legs change feet. The delay is for a recording that + // has to catch the first line: `simctl io recordVideo` running + // while the app launches stops the immersive space opening at + // all (`TBNotReady`, every time), so a recording can only start + // once the sheet is up — and by then an undelayed drawing is + // already under way. `TBPlaying` marks the moment it starts. // // `-TBSheet` is the framing one, and it exists because // nothing in the simulator can reach out and pinch the sheet @@ -178,9 +184,10 @@ openWindow(id: ViewerModel.programWindowID) openWindow(id: ViewerModel.codeWindowID) + let play = Self.value(of: "-TBPlay", in: arguments) await settle( drawingTo: Self.value(of: "-TBDraw", in: arguments).flatMap(Double.init), - playing: arguments.contains("-TBPlay")) + playing: arguments.contains("-TBPlay") ? play.flatMap(Double.init) ?? 0 : nil) } } @@ -235,7 +242,7 @@ /// all, and from the outside that is indistinguishable from one that is /// merely slow. Told which it is, the script can relaunch instead of /// filing a picture of an empty room. - private func settle(drawingTo fraction: Double?, playing: Bool) async { + private func settle(drawingTo fraction: Double?, playing delay: Double?) async { let clock = ContinuousClock() let deadline = clock.now + .seconds(20) while model.runner.player.currentTortoiseState == nil, clock.now < deadline { @@ -251,8 +258,12 @@ let clamped = min(max(fraction, 0), 1) model.runner.seek(to: Int((Double(commands - 1) * clamped).rounded())) } - if playing { model.runner.player.isPaused = false } Self.shoot.notice("TBReady") + if let delay { + try? await Task.sleep(for: .seconds(delay)) + model.runner.player.isPaused = false + Self.shoot.notice("TBPlaying") + } } /// Only ever written to under `-TBPlace`, and read only by the diff --git a/CLAUDE.md b/CLAUDE.md index 8c80a96..ca0513b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,16 +31,21 @@ pkill -x TortoiseBlocks; open ~/Library/Developer/Xcode/DerivedData/TortoiseBloc # `-TBSheet side,reach,drop` frames it (metres). The sheet is aimed at the # camera, so no recentring — but a run occasionally comes up without it, so # look at the capture. The walk is motion, so it is checked on a recording: -# `-TBPlay YES` keeps the drawing playing and `-TBSpeed 1` slows it to ×0.2. +# `-TBPlay YES` keeps the drawing playing and `-TBSpeed 1` slows it to ×0.2; +# start recordVideo only once the app is up — running it through the launch +# stops the immersive space opening (`-TBPlay ` delays the drawing). xcrun simctl install ~/Library/Developer/Xcode/DerivedData/TortoiseBlocks-*/Build/Products/Debug-xrsimulator/TortoiseBlocks.app xcrun simctl launch space.hiraku.tortoiseblocks \ -TBPlace YES -TBSample star -TBDraw 1 -TBSheet 0.5,0.95,0.42 xcrun simctl io screenshot shot.png # 3840x2160, with an alpha channel xcrun simctl io recordVideo walk.mov # Ctrl-C to stop -# The website's teaser video: records TeaserTests on the iPad simulator, then -# cuts it to a silent 1080p film (Tools/teaser/README.md). ~15 minutes. -ruby Tools/teaser/teaser.rb # --compose re-cuts the last recording +# The films (Tools/film/README.md): the website's teaser, and the App Store +# previews for iphone / ipad / mac / vision. Silent, for music added by hand; +# `--compose` re-cuts the last recording. The Mac one drives this Mac's real +# pointer and asks for the password first. +ruby Tools/film/teaser.rb # ~15 minutes +ruby Tools/film/previews.rb [ipad …] # ~5 minutes each # The App Store listing (appstore/). The check needs no key and no bundle; # the other two need ASC_ISSUER_ID / ASC_KEY_ID / ASC_PRIVATE_KEY_PATH. @@ -455,6 +460,7 @@ the four files it describes. **Releasing, the store listing and the website are in the `release` skill.** Tags, Xcode Cloud, TestFlight, `appstore/`, fastlane, and `site/`. **Making the pictures is the `screenshots` skill** — the capture rigs for iPad, Mac and Vision Pro, the pass every reshoot ends with (`ruby Tools/screenshots.rb`), and the traps that hand back a perfectly well-made capture of the wrong thing. +**Making the videos is the `film` skill** — the website's teaser and the App Store previews, from UI tests, cut to Apple's rules. **Localization**: `en` is the source language; Japanese (kid-friendly hiragana) lives in `App/Localizable.xcstrings`. Palette titles are diff --git a/Tools/film/README.md b/Tools/film/README.md new file mode 100644 index 0000000..d9c9a2e --- /dev/null +++ b/Tools/film/README.md @@ -0,0 +1,202 @@ +# The films + +Two kinds of video are made here, from scripts, rather than filmed by hand: +the website's teaser, and the App Store's previews. + +```bash +ruby Tools/film/teaser.rb # the teaser: record on the iPad simulator, then compose +ruby Tools/film/teaser.rb --compose # compose the last recording again + +ruby Tools/film/previews.rb # every preview: iphone, ipad, mac, vision +ruby Tools/film/previews.rb ipad mac # only the ones named +ruby Tools/film/previews.rb --compose # compose the last recordings again +``` + +Everything comes out silent, English and ready for music, which is added by +hand. The films land in a work directory the scripts print +(`$TMPDIR/tortoise-teaser/`, `$TMPDIR/tortoise-previews/`), next to the raw +recordings. None of it is committed: each film is megabytes, and fastlane's +deliver uploads screenshots but not previews, so App Store Connect takes the +previews by hand. + +The pieces: + +- **The scripts are UI tests.** `TeaserTests` and `AppPreviewTests` (iPhone, + iPad), both on `FilmTestCase`, and `MacPreviewTests` on its own. Each one + lists every press, when each caption comes up, and where the camera looks, + in the order they happen. Changing the story means editing the test. A + script writes a log of what it did and when (`events.jsonl`); it decides + nothing about the film. +- **`film.rb` is the crew.** It records a simulator while a test runs, cuts the + recording down to what moves, and draws the touches. +- **`teaser.rb` and `previews.rb` are the editors.** The teaser gets a + backdrop, a camera, captions and cards. The previews get only what Apple + allows (see below). +- **`text.swift`** renders the teaser's captions and titles in SF Pro Rounded. + ImageMagick cannot ask the system's variable font for a weight. +- **`window-recorder.swift`** records the Mac's window with ScreenCaptureKit. + +It needs Xcode, `ffmpeg` and ImageMagick (`magick`). The teaser takes about a +quarter of an hour, nearly all of it recording. The previews take about five +minutes each. Don't run either while a screenshot rig is running: they share +DerivedData (see the `screenshots` skill). + +## The previews + +Apple's rules are what shape these, and they are stricter than the teaser's +(developer.apple.com/app-store/app-previews/, and the preview specifications +in App Store Connect's help): + +- **The screen as captured.** No zooming into the UI, so there is no camera. + Nothing that is not the device either: no backdrop, no cards. The Mac's + desktop is the exception, and it is the same drawn plate the screenshots + stand on. Graphics that show where to touch are allowed, so the touches are + drawn as in the teaser. By the maintainer's choice, there are no captions. +- **15 to 30 seconds.** A cut that comes out short holds its last frame. One + that comes out long stops the run. +- **One exact size per device class**, 30fps at most, H.264 High, and an audio + track, which is required even when it is silence. iPhone is 886×1920, iPad + 1600×1200, Mac 1920×1080 and Vision Pro 3840×2160. + +Fifteen seconds is too short to build a program from nothing and still watch +it draw. So each script starts from a document that already holds most of +one, written in `PREVIEWS` in `previews.rb`. + +**The iPad is shot on the 13-inch, not the teaser's 11-inch.** A preview is +4:3, which only the 13-inch is. The 11-inch would have to be cropped, and a +preview may not be. + +**Vision Pro's is H.264 Level 5.1, not the 4.0 the specification names.** +Level 4.0 stops at 8,192 macroblocks, and 3840×2160 is 32,400. Whether App +Store Connect accepts it is checked at upload. + +**On a phone, the number pad is pressed; on an iPad, the value is typed.** A +phone's pad is what a phone is used with, so the touches drawn over its keys +are the real ones. A phone showed its pad under the hardware-keyboard +preference in one run and not in the next, so `setNumber` looks for the pad +rather than assuming either. Keys pressed one after another leave no +stillness to find the touch by, so a key's touch is taken as `KEY_LAG` +(0.26s, measured) before the tap returned. + +**Vision Pro is launched, not scripted: a simulator takes no input.** The +preview is the viewer doing what it is for, the star drawn on the table. The +launch arguments are the screenshot rig's, plus `-TBPlay 6`. **Recording +starts after the sheet is up, never before the launch.** With `simctl io +recordVideo` running while the app launches, the immersive space does not open +at all. The windows come up and the app reports `TBNotReady`, every time, +whether or not the simulator was restarted. So the app waits six seconds +before it plays: the sheet comes up, the recorder starts, and the drawing +begins at `TBPlaying`, whose log timestamp is where the cut starts. + +**The Mac is recorded on this Mac**, by ScreenCaptureKit, while +`MacPreviewTests` drives the real pointer. It wants the machine left alone for +a few minutes with the screen unlocked, and **macOS asks for the password** +before it lets a UI test take the pointer. That is the automation mode +`automationmodetool` reports. Nothing is drawn over the Mac's film: the +recorder keeps the pointer and its clicks. Four things stood in the way: + +- *The test runner is sandboxed.* It cannot write where the driver can read, + and the driver cannot write into its container. So the log comes back as + an attachment in the result bundle. The recorder watches for the window to + appear instead of waiting for a signal, and the test gives it an + eight-second head start once the document is open. +- *A window recorded as itself wears macOS's purple "sharing this window" + control* where its close, minimise and zoom buttons belong, in every frame. + It also answers to the window's title, which made the test's window query + ambiguous. So the recorder takes the display with every other application + removed, the wallpaper included, and crops it to the window. Outside the + window's corners is then black, and one frame's corners are flood-filled to + make the mask. +- *That recording writes frames whether or not anything changed*, so stillness + is judged by content: a frame counts as a change only if `mpdecimate` + keeps it (`Cut.new(raw, by_content: true)`). +- *A row's chips fold into the row on a Mac.* On iOS "Number 90" is a button + inside "Turn Right, Number 90". On a Mac the row is the one element, so the + chip is placed by measurement: 41pt of padding and icon, then the label in + the system font. A screenshot of the row was tried first and found nothing, + because the runner's screenshots do not show the app's window. + +## Things that look like mistakes and are not + +**The teaser is shot on the 11-inch iPad**, not the 13-inch the App Store +captures use. It has the same layout in fewer points, so every block is about +a quarter larger in a 1080p frame. At 13 inches the palette's text came out a +few pixels tall. + +**Most of every recording is thrown away.** A UI test is slow: every press +waits for the app to go idle, finding an element takes a snapshot, and a +typed digit takes seconds. The teaser's raw run is almost four minutes. The +simulator writes a frame only when the screen changes, so the gaps between +frames are exactly the still stretches. Each one is cut down to `HEAD` +seconds, except where the log asks for time: a caption to read, a camera move +to finish, the moment around a touch, and `linger` (a finished drawing, the +code at the end). The tests' own `pause`s therefore don't set the pace. They +only give the app time to settle. Two refinements. While a number is being +typed, the field's caret blinks, and every blink is a frame, so the test logs +the typing span and only the keys inside it are kept. And a drawing longer +than `LONG_DRAWING` plays at `FAST`×, because the spiral takes nine seconds at +the app's own tempo. + +**The recording is converted with `-fflags +igndts`.** The recorder writes +decode times that drift away from the presentation times, by eleven seconds +by the end of one phone recording. Without the flag the conversion followed +the wrong clock, and the film cut to the home screen before the drawing had +finished. + +**A touch is found in the recording, not read off the log.** A tap is logged +when it returns, and that is after the app has gone idle again: 0.1–0.3s after +the touch on an iPad, up to 1.4s on a phone, where a sheet has to finish +moving first. XCUITest touches only once the app is idle, so the touch is the +first frame after the tap was called (`t0`) that breaks a stillness of +`QUIET`. It is not the burst of frames the tap returned in: a settled sheet can +still draw one late frame, and that frame is not the touch. + +**A drop that did not land is cut out.** A drag in the simulator now and then +parts the rows and inserts nothing. The test checks that the block arrived, +tries again if not, and logs the failed take as a `cut`, which the film leaves +out. + +**The teaser's camera is part of the log, not the edit.** `camera(.blocks)` in +the test records a rectangle in screen points. The film eases to it, zooming so +that the rectangle fills the frame. The zoom is `zoompan` on the +full-resolution source, so the text stays sharp at 2×. + +**Presses are coordinates, so nothing scrolls for them.** +`XCUIElement.tap()` scrolls its element into view first. A coordinate tap +below the fold presses whatever is there. On the 11-inch iPad that includes +Repeat, and the first run put a Start Fill where the repeat should have been. +`palette(_:)` scrolls the palette with a finger until the entry is on screen. +Coordinates are used anyway because the log needs the point that was pressed. + +**The palette entry is the leftmost button with that label.** "Forward" is +also the transport's step button. It is a separate element with the same name, +and a query that walks its matches one by one lost one between counting and +fetching. + +**A hardware keyboard is attached for the run.** The scripts switch the +Simulator's `ConnectHardwareKeyboard` preference on and restart the device to +pick it up. Xcode 27 has no Simulator.app to attach one by hand. The +preferences are exported first and imported back when the run ends. Without +it, an iPad's number field raises the full on-screen keyboard over half the +screen. The maintainer's own recording used a pointer and a keyboard, so none +shows there either. The value is typed after a tap in the middle of the field, +which puts the caret at the end, so the old digits go by backspace. `⌘A` is +not used: selecting brought iPadOS 27's small number keypad up over the field, +and sometimes the full keyboard under it. + +**"Drawing finished" means the scrubber settled after it moved or had time +to.** It does not require a new end value: changing an angle leaves the step +count alone, so the run ends on exactly the value the previous one did, and +waiting for a different value waited forever. It does not require seeing the +scrubber move either. A press returns only once the app is idle, which once +took nine seconds, and an eight-step triangle is drawn in 0.8. On a phone +there is no scrubber at all until the run is under way. + +**The recording is sideways and variable-rate.** `simctl io recordVideo` +writes the framebuffer as it is held (portrait, with a landscape app on its +side) and only when something changes. `transpose=2` stands it up, and `fps=30` +fills the still stretches before anything is cut. + +**The system language is English for the run**, as well as the app's, because +the status bar writes its date in the system's language. It is put back +afterwards, as `ipad-shots.rb` does. diff --git a/Tools/film/film.rb b/Tools/film/film.rb new file mode 100644 index 0000000..8570c9f --- /dev/null +++ b/Tools/film/film.rb @@ -0,0 +1,481 @@ +# frozen_string_literal: true + +# What the teaser (`teaser.rb`) and the App Store previews (`previews.rb`) +# share: recording a UI test on a simulator, cutting the recording down to +# what moves, and drawing the touches the simulator does not show. +# `README.md` has why each piece is the way it is. + +require "fileutils" +require "json" +require "open3" +require "pathname" +require "securerandom" + +module Film + ROOT = Pathname.new(__dir__).parent.parent + BUNDLE_ID = "space.hiraku.tortoiseblocks" + FPS = 30 + + # Time, in seconds. A still screen is cut down to HEAD, except around these. + HEAD = 0.22 + TAP_BEFORE = 0.35 + TAP_AFTER = 0.25 + # When a touch cannot be found in the recording, it is taken to have landed + # this long before the test logged it — the iPad's typical 0.1–0.26s. A + # phone's taps return far later, after a sheet has finished moving, which is + # why the recording is asked first (`Cut#touch`). + TAP_LEAD = 0.22 + # A gap between frames at least this long is the screen standing still. + QUIET = 0.25 + # A key on a phone's number pad lights this long before the tap returns, + # measured over six presses at 0.26s each. + KEY_LAG = 0.26 + # A typed key has landed by the time it is logged. + KEY_BEFORE = 0.35 + KEY_AFTER = 0.15 + CAPTION_HOLD = 1.0 + CAMERA_HOLD = 0.8 + DRAG_AFTER = 0.3 + # A drawing longer than this plays at FAST× — the teaser's spiral takes nine + # seconds at the app's own tempo. + LONG_DRAWING = 3.0 + FAST = 2 + + module_function + + def run(*command) + system(*command) || abort("failed: #{command.join(' ')[0, 400]}") + end + + def simctl(*arguments) + IO.popen(["xcrun", "simctl", *arguments], err: %i[child out], &:read) + end + + # `a+b+c+…` as a balanced tree of sums. ffmpeg's expression parser recurses + # once per `+`, and a flat sum of the ~120 pieces a cut has fails to parse + # ("Cannot allocate memory"); balanced, it is seven deep. + def sum(terms) + return terms.first if terms.size == 1 + + half = terms.size / 2 + "(#{sum(terms[0, half])}+#{sum(terms[half..-1])})" + end + + def dims(file) + `magick identify -format '%w %h' '#{file}'`.split.map(&:to_i) + end + + # --------------------------------------------------------------------------- + # The simulator. + + # The newest runtime that has the device — for iOS 27.0 or later, the same + # floor as the screenshot rigs: 26.5 does not open a document handed to it + # by URL. + def device(name, os: "iOS", minimum: "27") + json = JSON.parse(simctl("list", "devices", "available", "-j")) + candidates = json["devices"].flat_map do |runtime, list| + runtime.include?("SimRuntime.#{os}-") ? list.select { |d| d["name"] == name }.map { |d| [runtime, d] } : [] + end + abort("No #{name} simulator.") if candidates.empty? + runtime, chosen = candidates.max_by { |r, _| r } + version = runtime[/#{os}-(\d+)-(\d+)/, 0].to_s.sub("#{os}-", "").tr("-", ".") + abort("#{name} is only on #{os} #{version}; #{minimum} or newer is needed.") if version < minimum + puts "#{name}: #{os} #{version}, #{chosen['udid']}" + if chosen["state"] != "Booted" + simctl("boot", chosen["udid"]) + simctl("bootstatus", chosen["udid"]) + end + chosen["udid"] + end + + def restart(udid) + simctl("shutdown", udid) + simctl("boot", udid) + simctl("bootstatus", udid) + end + + # English, a hardware keyboard, 9:41 and light — then a restart, which the + # first two need. Everything goes back when the process ends. + # + # The system is told the language as well as the app because the status bar + # writes its date in the system's. The keyboard is the Simulator's + # preference — Xcode 27 has no Simulator.app to attach one from — and on an + # iPad it is what keeps a number field from raising half a screen of keys. A + # phone showed its number pad under it in one run and not in the next, so + # `FilmTestCase.setNumber` looks for the pad rather than assuming either. + def prepare(udid, work) + preferences = work / "simulator-preferences-#{udid}.plist" + run("defaults", "export", "com.apple.iphonesimulator", preferences.to_s) + at_exit { system("defaults", "import", "com.apple.iphonesimulator", preferences.to_s) } + run("defaults", "write", "com.apple.iphonesimulator", "ConnectHardwareKeyboard", "-bool", "true") + + languages = simctl("spawn", udid, "defaults", "read", "-g", "AppleLanguages") + .scan(/[A-Za-z]{2,3}(?:-[A-Za-z0-9]+)*/) + locale = simctl("spawn", udid, "defaults", "read", "-g", "AppleLocale").strip + unless languages.first == "en-US" && locale == "en_US" + at_exit do + simctl("spawn", udid, "defaults", "write", "-g", "AppleLanguages", "-array", *languages) + simctl("spawn", udid, "defaults", "write", "-g", "AppleLocale", locale) + end + simctl("spawn", udid, "defaults", "write", "-g", "AppleLanguages", "-array", "en-US") + simctl("spawn", udid, "defaults", "write", "-g", "AppleLocale", "en_US") + end + restart(udid) + simctl("status_bar", udid, "override", + "--time", "9:41", "--batteryState", "charged", "--batteryLevel", "100", + "--wifiMode", "active", "--wifiBars", "3", "--cellularMode", "notSupported") + at_exit { simctl("status_bar", udid, "clear") } + simctl("ui", udid, "appearance", "light") + end + + # A document as the app writes one. `blocks` is the frozen wire format, with + # the ids left out: each block gets a fresh one here. + def document(title, blocks) + identify = lambda do |block| + kind = block.transform_values do |payload| + next payload unless payload.is_a?(Hash) && payload["body"] + + payload.merge("body" => payload["body"].map(&identify)) + end + { "id" => SecureRandom.uuid.upcase, "kind" => kind } + end + JSON.pretty_generate({ "blocks" => blocks.map(&identify), "schemaVersion" => 1, "title" => title }) + end + + # Starts `simctl io recordVideo` and returns when it has. The recording's + # own clock starts at "Recording started", and the test's log is in wall + # time, so the moment it says so is what lines the two up. + def start_recording(udid, movie) + FileUtils.rm_f(movie) + _, output, recorder = Open3.popen2e("xcrun", "simctl", "io", udid, "recordVideo", + "--codec=h264", "--force", movie.to_s) + while (line = output.gets) + return [recorder, Time.now.to_f] if line.include?("Recording started") + end + abort("the recorder never started") + end + + def stop_recording(recorder) + Process.kill("INT", recorder.pid) + recorder.value + end + + # Records one UI test into `raw`: the movie, the test's log, and the moment + # the recording started. + # + # `documents` maps a title to its blocks. An iPad is handed its document by + # URL from the device's tmp, which survives the reinstall preparing a test + # run does, and the app is uninstalled first so an import is never renamed + # `-1`. A phone cannot be handed one by URL, so its documents go in the app's + # own folder — installed first so the folder exists — and nothing is + # uninstalled, which would take them with it. + def record_test(udid:, test:, documents:, raw:, work:, phone: false) + prepare(udid, work) + common = [ + "-project", (ROOT / "TortoiseBlocks.xcodeproj").to_s, + "-scheme", "TortoiseBlocks", + "-destination", "id=#{udid}", + "-only-testing:TortoiseBlocksUITests/#{test}", + "-parallel-testing-enabled", "NO", + # A failed UI test otherwise spends ten minutes in `simctl diagnose`. + "-collect-test-diagnostics", "never" + ] + # Built first, so the recording is not minutes of a compiler. + run("xcodebuild", "build-for-testing", *common, "-quiet") + + if phone + settings = JSON.parse(`xcodebuild -project "#{ROOT}/TortoiseBlocks.xcodeproj" -scheme TortoiseBlocks \ + -destination "id=#{udid}" -showBuildSettings -json 2>/dev/null`).first["buildSettings"] + simctl("install", udid, (Pathname.new(settings["BUILT_PRODUCTS_DIR"]) / settings["FULL_PRODUCT_NAME"]).to_s) + seed = Pathname.new(simctl("get_app_container", udid, BUNDLE_ID, "data").strip) / "Documents" + log = Pathname.new(Dir.home) / "Library/Developer/CoreSimulator/Devices" / udid / "data/tmp/tbfilm/events.jsonl" + else + simctl("uninstall", udid, BUNDLE_ID) + seed = Pathname.new(Dir.home) / "Library/Developer/CoreSimulator/Devices" / udid / "data/tmp/tbfilm" + FileUtils.rm_rf(seed) + log = seed / "events.jsonl" + end + FileUtils.mkdir_p(seed) + FileUtils.mkdir_p(log.dirname) + documents.each { |title, blocks| (seed / "#{title}.tortoise").write(document(title, blocks)) } + + FileUtils.rm_rf(raw) + FileUtils.mkdir_p(raw) + recorder, started = start_recording(udid, raw / "raw.mov") + environment = { "TEST_RUNNER_TB_DOCUMENTS" => seed.to_s, "TEST_RUNNER_TB_EVENTS" => log.to_s } + passed = system(environment, "xcodebuild", "test-without-building", *common, + "-resultBundlePath", (raw / "test.xcresult").to_s, "-quiet") + stop_recording(recorder) + + FileUtils.cp(log, raw / "events.jsonl") if log.exist? + failure = Pathname.new("#{log}.failure.txt") + FileUtils.cp(failure, raw / "failure.txt") if failure.exist? + (raw / "recording.json").write(JSON.generate({ "started" => started })) + abort("#{test} stopped part-way; #{raw / 'failure.txt'} has the screen it was looking at.") unless passed + end + + # --------------------------------------------------------------------------- + # The cut. + + # A recording and its log, read back, with the recording cut down to what + # moves: the first HEAD of every still stretch, all of every moving one, and + # whatever each logged event needs around it. `map` takes recording time to + # film time; `select` is the ffmpeg expression that keeps those frames. + class Cut + attr_reader :events, :points, :kept, :total, :started + + # `by_content` is for recordings that write frames whether or not + # anything changed — ScreenCaptureKit on a display, which the Mac's is. + # There a frame counts as a change only if `mpdecimate` keeps it. + def initialize(raw, by_content: false) + @raw = raw + @frames = + if by_content + `ffmpeg -hide_banner -fflags +igndts -i '#{raw / 'raw.mov'}' -vf scale=640:-1,mpdecimate,showinfo -f null - 2>&1` + .scan(/pts_time:([\d.]+)/).flatten.map(&:to_f).sort + else + `ffprobe -v error -select_streams v -show_entries frame=pts_time -of csv=p=0 '#{raw / 'raw.mov'}'` + .split.map { |f| f.delete(",").to_f }.sort + end + @events = raw.join("events.jsonl").readlines.map { |line| JSON.parse(line) } + @started = JSON.parse(raw.join("recording.json").read)["started"] + screen = @events.find { |e| e["type"] == "screen" } + @points = [screen["w"].to_f, screen["h"].to_f] + @t_start = at(@events.find { |e| e["type"] == "start" }) + @t_end = at(@events.find { |e| e["type"] == "end" }) + @typing = of("typing").map { |e| [e["from"] - @started, at(e)] } + cut + end + + def at(event) + event["t"] - @started + end + + def of(type) + @events.select { |e| e["type"] == type } + end + + # When a tap's touch landed, in recording time: the first frame, after + # the tap was called, that breaks a stillness. XCUITest waits for the app to + # go idle before it touches, so the touch is what ends the quiet — and the + # log's own time, written once the app is idle again, can be a second and + # a half later on a phone, where a sheet has to finish moving first. + # + # Not "the burst the tap returned in": a sheet that has settled can still + # draw one late frame, and that frame is not the touch. + # + # Keys pressed one after another on a number pad are the exception: the + # last key's release is still drawing when the next goes down, so there is + # no stillness to break. Those return a steady KEY_LAG after the press. + def touch(e) + t = at(e) + return t - KEY_LAG if @typing.any? { |a, b| t > a && t < b } + + floor = e["t0"] ? e["t0"] - @started : t - 2.5 + landed = @frames.each_cons(2).find { |earlier, later| later > floor && later <= t + 0.02 && later - earlier >= QUIET } + landed ? landed[1] : t - TAP_LEAD + end + + # A drag's phases, in recording time, worked back from when it returned. + def gesture(e) + distance = Math.hypot(e["x2"] - e["x"], e["y2"] - e["y"]) + release = at(e) - 0.05 + move_end = release - e["linger"] + move_start = move_end - distance / e["velocity"] + { press: move_start - e["hold"], move_start: move_start, move_end: move_end, release: release } + end + + # Recording time → film time. A moment that was cut lands where the cut is. + def map(t) + @kept.each_with_index do |(a, b, speed), i| + return @offsets[i] if t < a + return @offsets[i] + (t - a) / speed if t <= b + end + @total + end + + # The whole film, in whole frames. + def length + (@total * FPS).floor / FPS.to_f + end + + def select + Film.sum(@kept.map do |a, b, speed| + piece = "between(t,#{a.round(4)},#{b.round(4)})" + speed == 1 ? piece : "#{piece}*not(mod(n,#{speed}))" + end) + end + + private + + def cut + # While a number is being typed the caret blinks, and every blink is a + # frame; those stretches count as still, and only the keys are kept. + changes = @frames.select { |t| t > @t_start && t < @t_end } + .reject { |t| @typing.any? { |a, b| t > a && t < b } } + + windows = [] + ([@t_start] + changes + [@t_end]).each_cons(2) { |x, y| windows << [x, [y, x + HEAD].min] } + @events.each do |e| + t = at(e) + case e["type"] + when "caption" then windows << [t, t + CAPTION_HOLD] + when "camera" then windows << [t, t + CAMERA_HOLD] + when "tap" + landed = touch(e) + windows << [landed - TAP_BEFORE, landed + TAP_AFTER] + when "key" then windows << [t - KEY_BEFORE, t + KEY_AFTER] + when "drag" + g = gesture(e) + windows << [g[:press] - 0.15, g[:release] + DRAG_AFTER] + when "linger" then windows << [t, t + e["seconds"]] + end + end + windows = windows.map { |a, b| [a.clamp(@t_start, @t_end), b.clamp(@t_start, @t_end)] } + .reject { |a, b| b <= a }.sort + kept = [] + windows.each do |a, b| + if kept.any? && a <= kept.last[1] + 1.5 / FPS + kept.last[1] = [kept.last[1], b].max + else + kept << [a, b] + end + end + + # Takes the script threw away — a drop that did not land, tried again. + of("cut").each do |e| + a = e["from"] - @started + b = at(e) + kept = kept.flat_map do |x, y| + next [[x, y]] if y <= a || x >= b + + [[x, [y, a].min], [[x, b].max, y]].reject { |p, q| q - p < 1e-3 } + end + end + + # Long drawings, from the press of play to the moment they finished (the + # linger `playDrawing` logs), are sped up; each piece carries its speed. + drawings = [] + @events.each_with_index do |e, i| + next unless e["type"] == "linger" + + press = @events[0...i].reverse.find { |p| p["type"] == "tap" } + next unless press && at(e) - at(press) > LONG_DRAWING + + drawings << [touch(press) + TAP_AFTER, at(e)] + end + @kept = kept.flat_map do |a, b| + cuts = drawings.flatten.select { |t| t > a && t < b }.sort + ([a] + cuts + [b]).each_cons(2).map do |x, y| + fast = drawings.any? { |p, q| x >= p && y <= q } + [x, y, fast ? FAST : 1] + end + end + + @offsets = [] + @total = 0.0 + @kept.each do |a, b, speed| + @offsets << @total + @total += (b - a) / speed + end + end + end + + # The recording the right way up, at a constant rate: the simulator writes a + # frame only when the screen changes, and holds a landscape screen on its + # side. Kept beside the raw movie and remade only when that changes. + def upright(raw, landscape:) + movie = raw / "upright.mp4" + return movie if movie.exist? && movie.mtime > raw.join("raw.mov").mtime + + # `igndts`: the recorder writes decode times that drift from the + # presentation times — eleven seconds apart by the end of one phone + # recording — and without it the conversion followed the wrong clock, so + # the film cut to the home screen before the drawing had finished. + filters = [landscape ? "transpose=2" : nil, "fps=#{FPS}"].compact.join(",") + run("ffmpeg", "-v", "error", "-y", "-fflags", "+igndts", "-i", raw.join("raw.mov").to_s, "-vf", filters, + "-c:v", "libx264", "-preset", "fast", "-crf", "10", "-pix_fmt", "yuv420p", movie.to_s) + movie + end + + # --------------------------------------------------------------------------- + # Touches. + + # The finger, `radius` pixels: a tap is a clip (it lands, then a ring spreads + # as it lifts); a held finger, for drags, is a still. + # + # PNG32 on every frame, the empty last one included — written as greyscale + # it has no alpha, and overlays as a black square. + def fingers(layers, radius) + size = (radius * 5).round + size += 1 if size.odd? + c = size / 2 + frames = layers / "finger" + FileUtils.rm_rf(frames) + FileUtils.mkdir_p(frames) + 12.times do |i| + p = i / 11.0 + dot_alpha = i < 7 ? 0.6 : 0.6 * (1 - (i - 6) / 5.0) + ring_r = radius * (1 + 1.25 * p) + ring_alpha = 0.9 * (1 - p) + dot_r = radius * (1 - [p * 2, 1].min / 6.0) + stroke = [radius / 12.0, 1.5].max + run("magick", "-size", "#{size}x#{size}", "xc:none", + "-fill", "rgba(255,255,255,#{dot_alpha.round(3)})", + "-stroke", "rgba(60,60,67,#{(dot_alpha * 0.8).round(3)})", "-strokewidth", stroke.round(1).to_s, + "-draw", "circle #{c},#{c} #{(c + dot_r).round(1)},#{c}", + "-fill", "none", "-stroke", "rgba(255,255,255,#{ring_alpha.round(3)})", + "-strokewidth", (stroke * 2).round(1).to_s, + "-draw", "circle #{c},#{c} #{(c + ring_r).round(1)},#{c}", "PNG32:#{frames / format('%02d.png', i)}") + end + run("magick", "-size", "#{size}x#{size}", "xc:none", "PNG32:#{frames / '12.png'}") + tap = layers / "tap.mov" + run("ffmpeg", "-v", "error", "-y", "-framerate", FPS.to_s, "-i", (frames / "%02d.png").to_s, + "-c:v", "png", tap.to_s) + held = layers / "held.png" + run("magick", "-size", "#{size}x#{size}", "xc:none", + "-fill", "rgba(255,255,255,0.6)", "-stroke", "rgba(60,60,67,0.5)", + "-strokewidth", ([radius / 12.0, 1.5].max).round(1).to_s, + "-draw", "circle #{c},#{c} #{c + radius},#{c}", "PNG32:#{held}") + { tap: tap, held: held, half: c } + end + + # Adds every logged touch to a filter graph: `inputs` and `filters` grow, + # and the label of the last stage is returned. `to_frame.(x, y, t)` places a + # screen point in the frame at film time t. + def touches(cut, finger, to_frame, inputs, filters, last) + index = inputs.count("-i") + cut.of("tap").each_with_index do |e, i| + moment = cut.map(cut.touch(e) - 1.0 / FPS) + x, y = to_frame.call(e["x"], e["y"], moment) + inputs.push("-i", finger[:tap].to_s) + filters << "[#{index}:v]format=rgba,setpts=PTS-STARTPTS+#{moment.round(3)}/TB[t#{i}]" + filters << "[#{last}][t#{i}]overlay=#{(x - finger[:half]).round}:#{(y - finger[:half]).round}:" \ + "eof_action=pass[vt#{i}]" + last = "vt#{i}" + index += 1 + end + cut.of("drag").each_with_index do |e, i| + g = cut.gesture(e).transform_values { |t| cut.map(t) } + x1, y1 = to_frame.call(e["x"], e["y"], g[:press]).map { |v| v - finger[:half] } + x2, y2 = to_frame.call(e["x2"], e["y2"], g[:release]).map { |v| v - finger[:half] } + span = [g[:move_end] - g[:move_start], 0.01].max + u = "clip((t-#{g[:move_start].round(3)})/#{span.round(3)},0,1)" + inputs.push("-loop", "1", "-i", finger[:held].to_s) + filters << "[#{index}:v]format=rgba[h#{i}]" + filters << "[#{last}][h#{i}]overlay=x='#{x1.round(1)}+(#{(x2 - x1).round(1)})*#{u}':" \ + "y='#{y1.round(1)}+(#{(y2 - y1).round(1)})*#{u}':eval=frame:shortest=1:" \ + "enable='between(t,#{g[:press].round(3)},#{g[:release].round(3)})'[vd#{i}]" + last = "vd#{i}" + index += 1 + # The lift, from the tap's own animation: its second half. + inputs.push("-i", finger[:tap].to_s) + filters << "[#{index}:v]format=rgba,trim=start=#{(6.0 / FPS).round(3)}," \ + "setpts=PTS-STARTPTS+#{g[:release].round(3)}/TB[r#{i}]" + filters << "[#{last}][r#{i}]overlay=#{x2.round}:#{y2.round}:eof_action=pass[vr#{i}]" + last = "vr#{i}" + index += 1 + end + last + end +end diff --git a/Tools/film/previews.rb b/Tools/film/previews.rb new file mode 100644 index 0000000..6a4552f --- /dev/null +++ b/Tools/film/previews.rb @@ -0,0 +1,415 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Makes the App Store app previews: fifteen-odd seconds per device, English, +# no captions, silent — the music goes on by hand, as it does for the teaser. +# +# ruby Tools/film/previews.rb # every device, record and compose +# ruby Tools/film/previews.rb ipad # only the ones named +# ruby Tools/film/previews.rb --compose # compose the last recordings again +# +# The films land in the work directory printed at the end. They are not +# committed — each is megabytes, and the listing takes them by hand: fastlane's +# deliver uploads screenshots but not previews. +# +# **Apple's rules are what shape these**, and they are stricter than the +# teaser's (developer.apple.com/app-store/app-previews/ and the preview +# specifications in App Store Connect's help): +# +# - The screen as captured: no zooming into the UI, so there is no camera, +# and nothing that is not the device — no backdrop, no cards. Graphics that +# show where to touch are allowed, so the touches are drawn as in the +# teaser. +# - 15 to 30 seconds. A cut that comes out short holds its last frame. +# - One exact size per device class, 30fps at most, H.264 High, and an audio +# track, which is required even when it is silence. +# +# `TortoiseBlocksUITests/AppPreviewTests.swift` holds the scripts, one test per +# device. Each starts from a document already holding most of its program, +# written here in PREVIEWS: fifteen seconds is too short to build one from +# nothing and still watch it draw. + +require "time" +require "tmpdir" +require_relative "film" + +WORK = Pathname.new(Dir.tmpdir) / "tortoise-previews" +FPS = Film::FPS +SHORTEST = 15.5 +LONGEST = 30.0 + +# Blocks in the frozen wire format (`BlockCodableTests` has every kind). +def forward(steps) + { "forward" => { "literal" => steps } } +end + +def turn_right(degrees) + { "turnRight" => { "literal" => degrees } } +end + +def pen_width(width) + { "penWidth" => { "literal" => width } } +end + +PREVIEWS = { + # 4:3, which only the 13-inch iPad is; the 11-inch would have to be cropped. + "ipad" => { + device: "iPad Pro 13-inch (M5)", test: "AppPreviewTests/testIPad", + landscape: true, size: [1600, 1200], + documents: { + "My Drawing" => [ + pen_width(6), + { "repeat" => { "count" => { "literal" => 5 }, "body" => [forward(150), turn_right(90)] } } + ] + } + }, + # 886×1920 is a hair taller than the 6.9-inch screen, so a few rows go. + "iphone" => { + device: "iPhone 17 Pro Max", test: "AppPreviewTests/testIPhone", + landscape: false, size: [886, 1920], phone: true, + documents: { + "My Drawing" => [ + pen_width(6), { "penColor" => "orange" }, + { "repeat" => { "count" => { "literal" => 5 }, "body" => [] } } + ] + } + }, + # The Mac's own window, recorded where it runs: this Mac, not a simulator. + # The test drives the real pointer, so this one wants the machine left alone + # for the three minutes it takes, with the screen unlocked. + "mac" => { + mac: true, test: "MacPreviewTests/testMac", size: [1920, 1080], + plate: "appstore/screenshot-sources/macos-plate-en.png", + documents: { + "My Drawing" => [ + pen_width(6), + { "repeat" => { "count" => { "literal" => 5 }, "body" => [forward(150), turn_right(90)] } } + ] + } + }, + # No script: a simulator takes no input, so the preview is the viewer doing + # what it is for — the star being drawn on the table, with the blocks and the + # code in their windows. The launch arguments are the screenshot rig's. + "vision" => { + device: "Apple Vision Pro", os: "xrOS", size: [3840, 2160], + sample: "star", sheet: "0.7,1.2,0.38", + # 0.5 / level seconds a step: the star's 74 steps take 14.2s at 2.6, and + # the legs have time to be seen changing feet. + speed: 2.6, length: 16.5, + # How long after the sheet is up the drawing starts: time for the + # recorder, which may not be running while the app launches. + wait: 6 + } +}.freeze + +def run(*command) + Film.run(*command) +end + +# --------------------------------------------------------------------------- +# Vision Pro: launched, not scripted. + +# The fraction of the frame that is strongly saturated: the windows put a +# palette of pastels in the picture and an empty room has none (the +# screenshot rig's measure; a real capture is 0.07–0.09, a room 0.01–0.02). +def ink(image) + IO.popen(["magick", image.to_s, "-colorspace", "HSL", "-channel", "G", "-separate", "+channel", + "-threshold", "40%", "-format", "%[fx:mean]", "info:"], &:read).to_f +end + +# When this launch logged `marker`, by the log's own clock, or nil if it said +# `TBNotReady` instead or nothing at all in time. +def logged_at(udid, pid, marker, timeout: 40) + deadline = Time.now + timeout + while Time.now < deadline + lines = IO.popen(["xcrun", "simctl", "spawn", udid, "log", "show", "--last", "3m", "--style", "ndjson", + "--predicate", "processID == #{pid} AND subsystem == \"#{Film::BUNDLE_ID}\""], &:read) + lines.each_line do |line| + entry = JSON.parse(line) rescue next + message = entry["eventMessage"].to_s + return nil if message.include?("TBNotReady") + return Time.strptime(entry["timestamp"], "%Y-%m-%d %H:%M:%S.%N%z").to_f if message.include?(marker) + end + sleep(0.5) + end + nil +end + +# **Recording starts after the sheet is up, never before the launch.** With +# `simctl io recordVideo` running while the app launches, the immersive space +# does not open at all — the windows come up and the app reports `TBNotReady`, +# every time, restarted simulator or not. So the app is told to wait +# (`-TBPlay `) before it plays: the sheet comes up, the recorder +# starts, and the drawing begins in front of it at `TBPlaying`. +def record_vision(profile, raw) + udid = Film.device(profile[:device], os: profile[:os], minimum: "26") + run("xcodebuild", "-project", (Film::ROOT / "TortoiseBlocks.xcodeproj").to_s, "-scheme", "TortoiseBlocks", + "-destination", "id=#{udid}", "-quiet", "build") + settings = JSON.parse(`xcodebuild -project "#{Film::ROOT}/TortoiseBlocks.xcodeproj" -scheme TortoiseBlocks \ + -destination "id=#{udid}" -showBuildSettings -json 2>/dev/null`).first["buildSettings"] + app = Pathname.new(settings["BUILT_PRODUCTS_DIR"]) / settings["FULL_PRODUCT_NAME"] + + FileUtils.rm_rf(raw) + FileUtils.mkdir_p(raw) + 3.times do |attempt| + # Reinstalled every time: visionOS restores an app's windows, and a launch + # that inherits the last one's opens a second set on top. + Film.simctl("terminate", udid, Film::BUNDLE_ID) + Film.simctl("uninstall", udid, Film::BUNDLE_ID) + Film.simctl("install", udid, app.to_s) + sleep(2) + launched = Film.simctl("launch", udid, Film::BUNDLE_ID, + "-TBPlace", "YES", "-TBSample", profile[:sample], "-TBDraw", "0", + "-TBPlay", profile[:wait].to_s, "-TBSpeed", profile[:speed].to_s, + "-TBSheet", profile[:sheet], "-AppleLanguages", "(en)", "-AppleLocale", "en_US") + pid = launched[/:\s*(\d+)/, 1] + unless pid && logged_at(udid, pid.to_i, "TBReady") + warn(" no sheet on attempt #{attempt + 1}, relaunching") + next + end + + recorder, started = Film.start_recording(udid, raw / "raw.mov") + playing = logged_at(udid, pid.to_i, "TBPlaying", timeout: profile[:wait] + 10) + sleep(profile[:length] + 1) if playing + Film.stop_recording(recorder) + Film.simctl("terminate", udid, Film::BUNDLE_ID) + unless playing && playing > started + warn(" the drawing started before the recorder on attempt #{attempt + 1}, relaunching") + next + end + + (raw / "recording.json").write(JSON.generate({ "started" => started, "playing" => playing })) + probe = raw / "probe.png" + Film.run("ffmpeg", "-v", "error", "-y", "-fflags", "+igndts", "-i", (raw / "raw.mov").to_s, + "-ss", (playing - started + profile[:length] - 1).round(3).to_s, "-frames:v", "1", probe.to_s) + measured = ink(probe) + return if measured >= 0.04 + + warn(" an empty room on attempt #{attempt + 1} (ink #{measured.round(4)}), relaunching") + end + abort("vision never came up with a sheet after three attempts") +end + +def compose_vision(profile, raw) + times = JSON.parse((raw / "recording.json").read) + width, height = profile[:size] + film = WORK / "vision.mp4" + # Level 5.1, not the 4.0 the specification names for H.264: 4.0 stops at + # 8,192 macroblocks, and 3840×2160 is 32,400. Vision Pro's is the one size + # 4.0 cannot carry. + run("ffmpeg", "-v", "error", "-y", "-fflags", "+igndts", "-i", (raw / "raw.mov").to_s, + "-f", "lavfi", "-i", "anullsrc=channel_layout=stereo:sample_rate=48000", + "-ss", (times["playing"] - times["started"]).round(3).to_s, "-t", profile[:length].to_s, + "-map", "0:v", "-map", "1:a", + "-vf", "fps=#{FPS},scale=#{width}:#{height}:flags=lanczos,format=yuv420p", + "-c:v", "libx264", "-profile:v", "high", "-level:v", "5.1", "-pix_fmt", "yuv420p", + "-b:v", "30M", "-maxrate", "40M", "-bufsize", "80M", "-preset", "slow", + "-c:a", "aac", "-b:a", "256k", "-ar", "48000", "-ac", "2", + "-movflags", "+faststart", film.to_s) + seconds = `ffprobe -v error -show_entries format=duration -of csv=p=0 '#{film}'`.to_f + puts format("%s — %dx%d, %.1fs", film, width, height, seconds) +end + +# --------------------------------------------------------------------------- +# The Mac: scripted, and recorded by ScreenCaptureKit rather than simctl. + +# The menu bar the plates were drawn with, in the plate's pixels, and the +# shadow the screenshots give the window (`Tools/macos-shots.rb`). +PLATE_MENU_BAR = 48 +MARGIN = 36 + +def record_mac(profile, raw) + recorder = WORK / "window-recorder" + run("xcrun", "swiftc", "-O", "-parse-as-library", (Pathname.new(__dir__) / "window-recorder.swift").to_s, + "-o", recorder.to_s) + seed = Pathname.new(Dir.mktmpdir("tbfilm")) + profile[:documents].each { |title, blocks| (seed / "#{title}.tortoise").write(Film.document(title, blocks)) } + common = [ + "-project", (Film::ROOT / "TortoiseBlocks.xcodeproj").to_s, "-scheme", "TortoiseBlocks", + "-destination", "platform=macOS", "-only-testing:TortoiseBlocksUITests/#{profile[:test]}", + "-parallel-testing-enabled", "NO", "-collect-test-diagnostics", "never" + ] + run("xcodebuild", "build-for-testing", *common, "-quiet") + + # The window at the app's own `defaultSize`, 1280×800pt: macOS restores a + # saved frame in preference to it, so the saved state goes first, as the + # screenshot rig does. + FileUtils.rm_rf(Pathname.new(Dir.home) / "Library/Saved Application State/#{Film::BUNDLE_ID}.savedState") + system("defaults", "delete", Film::BUNDLE_ID, out: File::NULL, err: File::NULL) + + FileUtils.rm_rf(raw) + FileUtils.mkdir_p(raw) + # The runner is sandboxed and can hand nothing over while it runs: the + # recorder waits for the window on its own, the test gives it a head start, + # and the log comes back afterwards as an attachment in the result bundle. + input, output, capture = Open3.popen2(recorder.to_s, Film::BUNDLE_ID, "My Drawing", (raw / "raw.mov").to_s) + waiting = Thread.new { output.gets.to_s[/started ([\d.]+)/, 1]&.to_f } + passed = system({ "TEST_RUNNER_TB_DOCUMENTS" => seed.to_s }, "xcodebuild", "test-without-building", *common, + "-resultBundlePath", (raw / "test.xcresult").to_s, "-quiet") + input.close + capture.value + started = waiting.value + + exported = raw / "attachments" + FileUtils.mkdir_p(exported) + run("xcrun", "xcresulttool", "export", "attachments", "--path", (raw / "test.xcresult").to_s, + "--output-path", exported.to_s, out: File::NULL, err: File::NULL) + JSON.parse((exported / "manifest.json").read).flat_map { |test| test["attachments"] }.each do |attachment| + name = attachment["suggestedHumanReadableName"].to_s + target = if name.start_with?("events") then raw / "events.jsonl" + elsif name.start_with?("failure") then raw / "failure.txt" + end + FileUtils.cp(exported / attachment["exportedFileName"], target) if target + end + abort("the window recorder never started") unless started + (raw / "recording.json").write(JSON.generate({ "started" => started })) + FileUtils.rm_rf(seed) + abort("#{profile[:test]} stopped part-way; #{raw / 'failure.txt'} has the window it was looking at.") unless passed +end + +# The window, cut like the others, on the screenshots' desktop: the plate +# scaled to the frame's width with its bottom cropped, the window scaled to +# sit under the menu bar with its shadow. Nothing is drawn over it — the +# recorder kept the pointer and its clicks. +def compose_mac(profile, raw) + layers = raw / "layers" + FileUtils.mkdir_p(layers) + cut = Film::Cut.new(raw, by_content: true) + source = Film.upright(raw, landscape: false) + width, height = profile[:size] + source_w, source_h = `ffprobe -v error -select_streams v -show_entries stream=width,height -of csv=p=0 '#{source}'` + .strip.split(",").map(&:to_i) + + plate = layers / "plate.png" + plate_w, = Film.dims(Film::ROOT / profile[:plate]) + plate_scale = width.to_f / plate_w + run("magick", (Film::ROOT / profile[:plate]).to_s, "-resize", "#{width}x", "-gravity", "north", + "-crop", "#{width}x#{height}+0+0", "+repage", plate.to_s) + menu_bar = (PLATE_MENU_BAR * plate_scale).round + window_h = height - menu_bar - 2 * MARGIN + window_w = (source_w * window_h.to_f / source_h).round + window_w += 1 if window_w.odd? + x = (width - window_w) / 2 + y = menu_bar + MARGIN + + # Outside its rounded corners the recorded window is black: flood-filled + # away from each corner of one frame, which gives the corners macOS drew. + still = layers / "still.png" + run("ffmpeg", "-v", "error", "-y", "-i", source.to_s, "-frames:v", "1", still.to_s) + mask = layers / "mask.png" + corners = [[0, 0], [source_w - 1, 0], [0, source_h - 1], [source_w - 1, source_h - 1]] + run("magick", still.to_s, "-alpha", "set", "-fuzz", "6%", "-fill", "none", + *corners.flat_map { |cx, cy| ["-draw", "color #{cx},#{cy} floodfill"] }, + "-alpha", "extract", "-resize", "#{window_w}x#{window_h}!", mask.to_s) + shadow = layers / "shadow.png" + run("magick", "-size", "#{window_w}x#{window_h}", "xc:black", mask.to_s, "-alpha", "off", + "-compose", "CopyOpacity", "-composite", "-background", "black", + "-shadow", "55x#{(30 * plate_scale).round}+0+#{(22 * plate_scale).round}", shadow.to_s) + offset = IO.popen(["magick", "identify", "-format", "%[fx:page.x] %[fx:page.y]", shadow.to_s], &:read) + .split.map(&:to_i) + + length = cut.length + abort(format("mac runs %.1fs; a preview is at most %ds.", length, LONGEST)) if length > LONGEST + hold = [SHORTEST - length, 0].max + inputs = ["-loop", "1", "-i", plate.to_s, "-loop", "1", "-i", shadow.to_s, "-i", source.to_s, + "-loop", "1", "-i", mask.to_s] + filters = [ + "[2:v]select='#{cut.select}',setpts=N/(#{FPS}*TB),scale=#{window_w}:#{window_h}:flags=lanczos,format=rgba[w0]", + "[3:v]format=gray[m]", + "[w0][m]alphamerge[w]", + "[0:v][1:v]overlay=#{x + offset[0]}:#{y + offset[1]}[bg]", + "[bg][w]overlay=#{x}:#{y}:shortest=1,format=yuv420p" + + (hold > 0.05 ? ",tpad=stop_mode=clone:stop_duration=#{hold.round(3)}" : "") + "[v]" + ] + script = raw / "filters.txt" + script.write(filters.join(";\n")) + film = WORK / "mac.mp4" + run("ffmpeg", "-v", "error", "-y", *inputs, + "-f", "lavfi", "-i", "anullsrc=channel_layout=stereo:sample_rate=48000", + "-/filter_complex", script.to_s, "-map", "[v]", "-map", "4:a", + "-t", (length + hold).round(3).to_s, "-r", FPS.to_s, + "-c:v", "libx264", "-profile:v", "high", "-level:v", "4.0", "-pix_fmt", "yuv420p", + "-b:v", "10M", "-maxrate", "12M", "-bufsize", "24M", "-preset", "slow", + "-c:a", "aac", "-b:a", "256k", "-ar", "48000", "-ac", "2", + "-movflags", "+faststart", film.to_s) + seconds = `ffprobe -v error -show_entries format=duration -of csv=p=0 '#{film}'`.to_f + puts format("%s — %dx%d, %.1fs (%.1fs cut), %d clicks", film, width, height, seconds, length, cut.of("tap").size) +end + +# --------------------------------------------------------------------------- +# iPhone and iPad: scripted. + +def compose(name, profile) + raw = WORK / name + layers = raw / "layers" + FileUtils.mkdir_p(layers) + cut = Film::Cut.new(raw) + source = Film.upright(raw, landscape: profile[:landscape]) + width, height = profile[:size] + + # Scaled to cover the frame and centred, so a source a touch off the frame's + # shape loses a few rows rather than gaining bars. + source_w, source_h = `ffprobe -v error -select_streams v -show_entries stream=width,height -of csv=p=0 '#{source}'` + .strip.split(",").map(&:to_i) + scale = [width.to_f / source_w, height.to_f / source_h].max + scaled_w = (source_w * scale).round + scaled_h = (source_h * scale).round + scaled_w += 1 if scaled_w.odd? + scaled_h += 1 if scaled_h.odd? + crop_x = (scaled_w - width) / 2 + crop_y = (scaled_h - height) / 2 + pixels = source_w / cut.points[0] * scale + to_frame = ->(x, y, _t) { [x * pixels - crop_x, y * pixels - crop_y] } + finger = Film.fingers(layers, (20 * pixels).round) + + length = cut.length + abort(format("%s runs %.1fs; a preview is at most %ds.", name, length, LONGEST)) if length > LONGEST + hold = [SHORTEST - length, 0].max + + inputs = ["-i", source.to_s] + filters = ["[0:v]select='#{cut.select}',setpts=N/(#{FPS}*TB)," \ + "scale=#{scaled_w}:#{scaled_h}:flags=lanczos,crop=#{width}:#{height}:#{crop_x}:#{crop_y}," \ + "format=rgba[v0]"] + last = Film.touches(cut, finger, to_frame, inputs, filters, "v0") + filters << "[#{last}]format=yuv420p" + (hold.positive? ? ",tpad=stop_mode=clone:stop_duration=#{hold.round(3)}" : "") + "[v]" + script = raw / "filters.txt" + script.write(filters.join(";\n")) + + film = WORK / "#{name}.mp4" + # Level 4.0 is the ceiling the specification names, and it covers every + # size here: the largest, 1600×1200, is 7,500 macroblocks of its 8,192. + run("ffmpeg", "-v", "error", "-y", *inputs, + "-f", "lavfi", "-i", "anullsrc=channel_layout=stereo:sample_rate=48000", + "-/filter_complex", script.to_s, "-map", "[v]", "-map", "#{inputs.count('-i')}:a", + "-t", (length + hold).round(3).to_s, "-r", FPS.to_s, + "-c:v", "libx264", "-profile:v", "high", "-level:v", "4.0", "-pix_fmt", "yuv420p", + "-b:v", "10M", "-maxrate", "12M", "-bufsize", "24M", "-preset", "slow", + "-c:a", "aac", "-b:a", "256k", "-ar", "48000", "-ac", "2", + "-movflags", "+faststart", film.to_s) + seconds = `ffprobe -v error -show_entries format=duration -of csv=p=0 '#{film}'`.to_f + puts format("%s — %dx%d, %.1fs (%.1fs cut%s), %d taps, %d drags", + film, width, height, seconds, length, hold > 0.05 ? format(", %.1fs held", hold) : "", + cut.of("tap").size, cut.of("drag").size) +end + +wanted = ARGV.reject { |a| a.start_with?("-") } +chosen = wanted.empty? ? PREVIEWS : PREVIEWS.select { |name, _| wanted.include?(name) } +abort("Nothing matches #{wanted.join(', ')}; there are #{PREVIEWS.keys.join(', ')}.") if chosen.empty? + +FileUtils.mkdir_p(WORK) +chosen.each do |name, profile| + raw = WORK / name + if profile[:mac] + record_mac(profile, raw) unless ARGV.include?("--compose") + compose_mac(profile, raw) + elsif profile[:test] + unless ARGV.include?("--compose") + Film.record_test(udid: Film.device(profile[:device]), test: profile[:test], raw: raw, + work: WORK, documents: profile[:documents], phone: profile[:phone] || false) + end + compose(name, profile) + else + record_vision(profile, raw) unless ARGV.include?("--compose") + compose_vision(profile, raw) + end +end diff --git a/Tools/film/teaser.rb b/Tools/film/teaser.rb new file mode 100644 index 0000000..8a10bfd --- /dev/null +++ b/Tools/film/teaser.rb @@ -0,0 +1,215 @@ +#!/usr/bin/env ruby +# frozen_string_literal: true + +# Makes the teaser video for the website: a child's-eye walk from one line to +# a spiral of stars, shot on the iPad simulator and cut to 1920×1080. +# +# ruby Tools/film/teaser.rb # record, then compose +# ruby Tools/film/teaser.rb --compose # compose the last recording again +# +# Silent on purpose: the music is laid on afterwards, by hand. The film lands +# in the work directory printed at the end, with the raw recording beside it. +# +# `TortoiseBlocksUITests/TeaserTests.swift` is the script — every press, when +# each caption comes up and where the camera looks. `film.rb` records it and +# cuts it down to what moves; this frames it: the backdrop, the camera, the +# captions and the cards at either end. The README beside this file says why +# each piece is the way it is. + +require "tmpdir" +require_relative "film" + +WORK = Pathname.new(Dir.tmpdir) / "tortoise-teaser" +RAW = WORK / "raw" +# The 11-inch rather than the 13-inch the App Store captures use: the same +# layout in fewer points, so every block is a quarter larger in the frame. +DEVICE_NAME = "iPad Pro 11-inch (M5)" + +W = 1920 +H = 1080 +FPS = Film::FPS +# The previous video's backdrop, left to right. +GREEN = "#39FE92" +CYAN = "#42D2F7" +INK = "1c1c1e" +# The iPad's screen in the frame: clear of the top and bottom by a little. +MARGIN = 40 +RADIUS = 26 +CAMERA_EASE = 0.7 +TITLE = 3.5 +ENDING = 5.0 +FADE = 0.6 + +def run(*command) + Film.run(*command) +end + +def compose + text_tool = WORK / "text" + run("xcrun", "swiftc", "-O", (Pathname.new(__dir__) / "text.swift").to_s, "-o", text_tool.to_s) + text = lambda do |file, string, size, weight, colour| + run(text_tool.to_s, file.to_s, string, size.to_s, weight, colour) + end + layers = WORK / "layers" + FileUtils.mkdir_p(layers) + + cut = Film::Cut.new(RAW) + points = cut.points + footage = cut.length + source = Film.upright(RAW, landscape: true) + puts format("kept %.1fs, in %d pieces", cut.total, cut.kept.size) + + # The screen in the frame, and the camera over it. + screen_h = H - 2 * MARGIN + screen_w = (screen_h * points[0] / points[1]).round + screen_w += 1 if screen_w.odd? + screen_x = (W - screen_w) / 2 + screen_y = MARGIN + scale = screen_h / points[1] + + cues = cut.of("camera").map do |e| + x, y, w, h = e["rect"] + zoom = [[points[0] / w, points[1] / h].min, 1.0].max + { at: cut.map(cut.at(e)), to: [x + w / 2.0, y + h / 2.0, zoom] } + end + cues = [{ at: -10.0, to: [points[0] / 2, points[1] / 2, 1.0] }] + cues + smooth = ->(u) { u = u.clamp(0.0, 1.0); u * u * (3 - 2 * u) } + camera_at = lambda do |t| + value = cues.first[:to].dup + cues.each_cons(2) do |a, b| + k = smooth.call((t - b[:at]) / CAMERA_EASE) + 3.times { |i| value[i] += (b[:to][i] - a[:to][i]) * k } + end + value + end + # The same curve, for zoompan, as a function of the film's time. + camera_expression = lambda do |i| + terms = ["(#{cues.first[:to][i].round(4)})"] + cues.each_cons(2) do |a, b| + delta = b[:to][i] - a[:to][i] + next if delta.abs < 1e-6 + + u = "clip((it-#{b[:at].round(3)})/#{CAMERA_EASE},0,1)" + terms << "(#{delta.round(5)})*#{u}*#{u}*(3-2*#{u})" + end + Film.sum(terms) + end + to_frame = lambda do |x, y, t| + cx, cy, z = camera_at.call(t) + left = (cx - points[0] / z / 2).clamp(0, points[0] - points[0] / z) + top = (cy - points[1] / z / 2).clamp(0, points[1] - points[1] / z) + [screen_x + (x - left) * z * scale, screen_y + (y - top) * z * scale] + end + + # Stills: the backdrop, the screen's corners and shadow. + background = layers / "background.png" + run("magick", "-size", "#{H}x#{W}", "gradient:#{GREEN}-#{CYAN}", "-rotate", "-90", background.to_s) + mask = layers / "mask.png" + run("magick", "-size", "#{screen_w}x#{screen_h}", "xc:black", "-fill", "white", + "-draw", "roundrectangle 0,0 #{screen_w - 1},#{screen_h - 1} #{RADIUS},#{RADIUS}", mask.to_s) + shadow = layers / "shadow.png" + run("magick", "-size", "#{W}x#{H}", "xc:none", "-fill", "rgba(0,70,50,0.30)", + "-draw", "roundrectangle #{screen_x},#{screen_y + 12} #{screen_x + screen_w},#{screen_y + screen_h + 12} #{RADIUS},#{RADIUS}", + "-blur", "0x20", shadow.to_s) + finger = Film.fingers(layers, 24) + + # Captions: dark text on a white pill, low over the screen, each up until + # the next one. + captions = cut.of("caption") + caption_layers = captions.each_with_index.map do |caption, i| + words = layers / "caption-text-#{i}.png" + text.call(words, caption["text"], 46, "bold", INK) + tw, th = Film.dims(words) + pill_w = tw + 24 + pill_h = th - 4 + pad = 24 + file = layers / "caption-#{i}.png" + run("magick", "-size", "#{pill_w + 2 * pad}x#{pill_h + 2 * pad}", "xc:none", + "-fill", "rgba(0,0,0,0.22)", + "-draw", "roundrectangle #{pad},#{pad + 4} #{pad + pill_w - 1},#{pad + pill_h + 3} #{pill_h / 2},#{pill_h / 2}", + "-blur", "0x9", + "-fill", "rgba(255,255,255,0.96)", + "-draw", "roundrectangle #{pad},#{pad} #{pad + pill_w - 1},#{pad + pill_h - 1} #{pill_h / 2},#{pill_h / 2}", + words.to_s, "-gravity", "center", "-geometry", "+0-2", "-composite", "PNG32:#{file}") + from = cut.map(cut.at(caption)) + to = i + 1 < captions.size ? cut.map(cut.at(captions[i + 1])) : footage + puts format(" %5.1fs %4.1fs %s", TITLE - FADE + from, to - from, caption["text"]) + { file: file, from: from, to: to } + end + + # The footage: cut, zoomed, framed, captioned, touched. + zx = camera_expression.call(0) + zy = camera_expression.call(1) + zz = camera_expression.call(2) + inputs = ["-loop", "1", "-i", background.to_s, "-i", source.to_s, + "-loop", "1", "-i", mask.to_s, "-loop", "1", "-i", shadow.to_s] + filters = [] + filters << "[1:v]select='#{cut.select}',setpts=N/(#{FPS}*TB)," \ + "zoompan=z='#{zz}':x='clip((#{zx})*iw/#{points[0]}-iw/zoom/2,0,iw-iw/zoom)':" \ + "y='clip((#{zy})*ih/#{points[1]}-ih/zoom/2,0,ih-ih/zoom)':d=1:s=#{screen_w}x#{screen_h}:fps=#{FPS}," \ + "format=rgba[scr0]" + filters << "[2:v]format=gray[mask]" + filters << "[scr0][mask]alphamerge[scr]" + filters << "[0:v][3:v]overlay=0:0[bg]" + filters << "[bg][scr]overlay=#{screen_x}:#{screen_y}:shortest=1[v0]" + last = "v0" + caption_layers.each_with_index do |layer, i| + index = inputs.count("-i") + inputs.push("-loop", "1", "-i", layer[:file].to_s) + w, h = Film.dims(layer[:file]) + filters << "[#{index}:v]format=rgba,fade=t=in:st=#{layer[:from].round(3)}:d=0.25:alpha=1," \ + "fade=t=out:st=#{(layer[:to] - 0.25).round(3)}:d=0.25:alpha=1[c#{i}]" + filters << "[#{last}][c#{i}]overlay=#{(W - w) / 2}:#{screen_y + screen_h - h - 18}:shortest=1[vc#{i}]" + last = "vc#{i}" + end + last = Film.touches(cut, finger, to_frame, inputs, filters, last) + footage_file = WORK / "footage.mp4" + script = WORK / "filters.txt" + script.write(filters.join(";\n")) + run("ffmpeg", "-v", "error", "-y", *inputs, "-/filter_complex", script.to_s, "-map", "[#{last}]", + "-t", footage.round(3).to_s, "-r", FPS.to_s, + "-c:v", "libx264", "-preset", "medium", "-crf", "16", "-pix_fmt", "yuv420p", footage_file.to_s) + + # The cards: the icon and the name first; the icon, the name and where it + # runs at the end. + icon = Film::ROOT / "docs" / "Icon.png" + name = layers / "name.png" + text.call(name, "Tortoise Blocks", 104, "heavy", INK) + devices = layers / "devices.png" + text.call(devices, "iPad · iPhone · Mac · Apple Vision Pro", 44, "semibold", "3a3a3c") + title = layers / "title.png" + run("magick", background.to_s, + "(", icon.to_s, "-resize", "340x340", ")", "-gravity", "north", "-geometry", "+0+250", "-composite", + name.to_s, "-gravity", "north", "-geometry", "+0+620", "-composite", title.to_s) + ending = layers / "ending.png" + run("magick", background.to_s, + "(", icon.to_s, "-resize", "300x300", ")", "-gravity", "north", "-geometry", "+0+215", "-composite", + name.to_s, "-gravity", "north", "-geometry", "+0+545", "-composite", + devices.to_s, "-gravity", "north", "-geometry", "+0+705", "-composite", ending.to_s) + + film = WORK / "teaser.mp4" + graph = [ + "[0:v]fps=#{FPS},format=yuv420p,settb=AVTB[a]", + "[1:v]fps=#{FPS},format=yuv420p,settb=AVTB[b]", + "[2:v]fps=#{FPS},format=yuv420p,settb=AVTB[c]", + "[a][b]xfade=transition=fade:duration=#{FADE}:offset=#{(TITLE - FADE).round(3)}[ab]", + "[ab][c]xfade=transition=fade:duration=#{FADE}:offset=#{(TITLE + footage - 2 * FADE).round(3)}[v]" + ].join(";") + run("ffmpeg", "-v", "error", "-y", + "-loop", "1", "-t", TITLE.to_s, "-i", title.to_s, + "-i", footage_file.to_s, + "-loop", "1", "-t", ENDING.to_s, "-i", ending.to_s, + "-filter_complex", graph, "-map", "[v]", + "-c:v", "libx264", "-preset", "slow", "-crf", "17", "-pix_fmt", "yuv420p", + "-movflags", "+faststart", film.to_s) + length = `ffprobe -v error -show_entries format=duration -of csv=p=0 '#{film}'`.to_f + puts format("%s — %.1fs, %d taps, %d drags, %d captions", + film, length, cut.of("tap").size, cut.of("drag").size, captions.size) +end + +FileUtils.mkdir_p(WORK) +unless ARGV.include?("--compose") + Film.record_test(udid: Film.device(DEVICE_NAME), test: "TeaserTests", raw: RAW, work: WORK, + documents: { "My Drawing" => [] }) +end +compose diff --git a/Tools/teaser/text.swift b/Tools/film/text.swift similarity index 100% rename from Tools/teaser/text.swift rename to Tools/film/text.swift diff --git a/Tools/film/window-recorder.swift b/Tools/film/window-recorder.swift new file mode 100644 index 0000000..e48bd65 --- /dev/null +++ b/Tools/film/window-recorder.swift @@ -0,0 +1,151 @@ +// Records one window with ScreenCaptureKit — the pointer and its clicks +// included — until its standard input closes. `previews.rb` compiles and runs +// it for the Mac's preview. +// +// window-recorder <out.mov> +// +// It waits for the window to appear, then prints `started <unix time>` once +// the file is being written: the test's log is in wall time, and that line is +// what lines the two up. +// +// The window alone rather than the screen: nothing else on the machine reaches +// the picture, the window can sit behind another and still be recorded, and +// the desktop around it comes from the same drawn plate the screenshots use. +import AVFoundation +import AppKit +import ScreenCaptureKit + +final class Recording: NSObject, SCRecordingOutputDelegate, @unchecked Sendable { + private let lock = NSLock() + private var startWaiters: [CheckedContinuation<Void, Never>] = [] + private var finishWaiters: [CheckedContinuation<Void, Never>] = [] + private var didStart = false + private var didFinish = false + + func recordingOutputDidStartRecording(_ recordingOutput: SCRecordingOutput) { + print("started \(Date().timeIntervalSince1970)") + fflush(stdout) + resume(&startWaiters, flag: \.didStart) + } + + func recordingOutputDidFinishRecording(_ recordingOutput: SCRecordingOutput) { + resume(&finishWaiters, flag: \.didFinish) + } + + func recordingOutput(_ recordingOutput: SCRecordingOutput, didFailWithError error: any Error) { + FileHandle.standardError.write(Data("recording failed: \(error)\n".utf8)) + exit(1) + } + + func started() async { await wait(on: \.startWaiters, flag: \.didStart) } + func finished() async { await wait(on: \.finishWaiters, flag: \.didFinish) } + + private func resume( + _ waiters: inout [CheckedContinuation<Void, Never>], + flag: ReferenceWritableKeyPath<Recording, Bool> + ) { + lock.lock() + self[keyPath: flag] = true + let pending = waiters + waiters = [] + lock.unlock() + for waiter in pending { waiter.resume() } + } + + private func wait( + on waiters: ReferenceWritableKeyPath<Recording, [CheckedContinuation<Void, Never>]>, + flag: KeyPath<Recording, Bool> + ) async { + await withCheckedContinuation { continuation in + lock.lock() + if self[keyPath: flag] { + lock.unlock() + continuation.resume() + } + else { + self[keyPath: waiters].append(continuation) + lock.unlock() + } + } + } +} + +@main +struct WindowRecorder { + static func main() async throws { + let arguments = CommandLine.arguments + guard arguments.count == 4 else { + FileHandle.standardError.write( + Data("usage: window-recorder <bundle id> <title fragment> <out.mov>\n".utf8)) + exit(64) + } + let (bundle, title, out) = (arguments[1], arguments[2], URL(fileURLWithPath: arguments[3])) + // A command-line tool has no window-server connection until something + // asks for one, and a stream started without it stops the process on + // an assertion (`CGS_REQUIRE_INIT`). + _ = await MainActor.run { NSApplication.shared } + + // Waits for the window: the driver starts this beside the test, and the + // document it records is not open yet. + var found: SCWindow? + let deadline = Date().addingTimeInterval(180) + while found == nil, Date() < deadline { + let content = try await SCShareableContent.excludingDesktopWindows( + false, onScreenWindowsOnly: true) + found = content.windows.first(where: { + $0.owningApplication?.bundleIdentifier == bundle + && ($0.title ?? "").contains(title) + }) + if found == nil { try await Task.sleep(for: .milliseconds(250)) } + } + guard let window = found else { + FileHandle.standardError.write(Data("no window of \(bundle) titled \(title)\n".utf8)) + exit(1) + } + + // The display with every other application taken out — the wallpaper + // included, so outside the window's corners is black — cropped to the + // window. Not the window by name: a window that is shared as itself + // wears macOS's purple "sharing this window" control where its close, + // minimise and zoom buttons belong, in every frame. + let content = try await SCShareableContent.excludingDesktopWindows( + false, onScreenWindowsOnly: true) + guard let display = content.displays.first(where: { $0.frame.intersects(window.frame) }) + else { + FileHandle.standardError.write(Data("no display shows the window\n".utf8)) + exit(1) + } + let filter = SCContentFilter( + display: display, + excludingApplications: content.applications.filter { $0.bundleIdentifier != bundle }, + exceptingWindows: []) + let configuration = SCStreamConfiguration() + let scale = CGFloat(filter.pointPixelScale) + configuration.sourceRect = CGRect( + x: window.frame.minX - display.frame.minX, y: window.frame.minY - display.frame.minY, + width: window.frame.width, height: window.frame.height) + configuration.width = Int(window.frame.width * scale) + configuration.height = Int(window.frame.height * scale) + configuration.showsCursor = true + configuration.showMouseClicks = true + configuration.minimumFrameInterval = CMTime(value: 1, timescale: 60) + + let settings = SCRecordingOutputConfiguration() + settings.outputURL = out + settings.outputFileType = .mov + settings.videoCodecType = .h264 + let recording = Recording() + let output = SCRecordingOutput(configuration: settings, delegate: recording) + let stream = SCStream(filter: filter, configuration: configuration, delegate: nil) + try stream.addRecordingOutput(output) + try await stream.startCapture() + await recording.started() + + // Until the driver closes standard input. + await Task.detached { _ = FileHandle.standardInput.readDataToEndOfFile() }.value + // The stream may have stopped already: it ends with the window, and the + // test quits the app before the driver says stop. + try? await stream.stopCapture() + await recording.finished() + } +} diff --git a/Tools/teaser/README.md b/Tools/teaser/README.md deleted file mode 100644 index 84b372b..0000000 --- a/Tools/teaser/README.md +++ /dev/null @@ -1,107 +0,0 @@ -# The teaser video - -The website's video — a program built by trial and error, from one line to a -spiral of stars — is made here, not filmed by hand. - -```bash -ruby Tools/teaser/teaser.rb # record on the iPad simulator, then compose -ruby Tools/teaser/teaser.rb --compose # compose the last recording again -``` - -The film comes out silent, 1920×1080 at 30fps, about two minutes, in the -work directory the script prints (`$TMPDIR/tortoise-teaser/teaser.mp4`), with -the raw recording beside it. Music is added afterwards, by hand. - -Three pieces: - -- **`TortoiseBlocksUITests/TeaserTests.swift` is the script.** Every press, - when each caption comes up, and where the camera looks, in the order they - happen. Changing the story is editing this file. The test writes a log of - what it did and when (`events.jsonl`); it does not decide anything about the - film. -- **`teaser.rb` is the crew and the editor.** It records the simulator while - the test runs, then cuts, frames, zooms, captions and draws the touches from - the log. -- **`text.swift`** renders captions and titles in SF Pro Rounded, because - ImageMagick cannot ask the system's variable font for a weight. - -It needs Xcode, `ffmpeg` and ImageMagick (`magick`). A full run takes about a -quarter of an hour, nearly all of it the recording; composing again takes two -minutes. Don't run it while a screenshot rig is -running: they share DerivedData (see the `screenshots` skill). - -## Things that look like mistakes and are not - -**The shoot is on the 11-inch iPad**, not the 13-inch one the App Store -captures use. It has the same layout in fewer points, so every block is about -a quarter larger in a 1080p frame. At 13 inches the palette's text came out a -few pixels tall. - -**Most of the recording is thrown away.** A UI test is slow: every press -waits for the app to go idle, finding an element takes a snapshot, and a -typed digit takes seconds. The raw run is almost four minutes. The simulator -writes a frame only when the screen changes, so the gaps between frames are -exactly the still stretches. Each one is cut down to `HEAD` seconds, except -where the log asks for time: a caption to read, a camera move to finish, the -moment around a touch, and `linger` (a finished drawing, the code at the end). -The test's own `pause`s therefore don't set the pace. They only give the app -time to settle. Two refinements: while a number is being typed the field's -caret blinks, and every blink is a frame, so the test logs the typing span and -only the keys inside it are kept. And a drawing longer than `LONG_DRAWING` -plays at `FAST`×: the spiral takes nine seconds at the app's own tempo. - -**A drop that did not land is cut out.** A drag in the simulator now and then -parts the rows and inserts nothing. The test checks that the block arrived, -tries again if not, and logs the failed take as a `cut`, which the film leaves -out. - -**Touches are drawn afterwards.** The simulator's recording shows no finger, -and the log already knows where each press went. A tap is logged just after it -returns, and the screen reacts 0.1–0.26s before that (measured against the -first changed frame), so the finger is drawn landing `TAP_LEAD` earlier. A drag -is worked back from when it returned: its hold, its travel at the velocity it -was given, and its linger. - -**The camera is part of the log, not the edit.** `camera(.blocks)` in the test -records a rectangle in screen points. The film eases to it, zooming so that -the rectangle fills the frame. The zoom is `zoompan` on the full-resolution -source, so the text stays sharp at 2×. - -**Presses are coordinates, so nothing scrolls for them.** `XCUIElement.tap()` -scrolls its element into view first; a coordinate tap below the fold presses -whatever is there. On the 11-inch iPad that includes Repeat, and the first run -put a Start Fill where the repeat should have been. `palette(_:)` scrolls the -palette with a finger until the entry is on screen. Coordinates are used anyway -because the log needs the point that was pressed. - -**The palette entry is the leftmost button with that label.** "Forward" is -also the transport's step button. It is a separate element with the same name, -and a query that walks its matches one by one lost one between counting and -fetching. - -**A hardware keyboard is attached for the run**, by switching the Simulator's -`ConnectHardwareKeyboard` preference on and restarting the device to pick it -up. Xcode 27 has no Simulator.app to attach one by hand. The -preferences are exported first and imported back when the run ends. Without -it, a number field raises the full on-screen keyboard over half the screen. The maintainer's own recording -used a pointer and a keyboard, so none shows there either. The value is typed -after a tap in the middle of the field, which puts the caret at the end, so -the old digits go by backspace. `⌘A` is not used: selecting brought iPadOS -27's small number keypad up over the field, and sometimes the full keyboard -under it. - -**"Drawing finished" means the scrubber settled after it moved or had time -to.** It does not require a new end value: changing an angle leaves the step -count alone, so the run ends on exactly the value the previous one did, and -waiting for a different value waited forever. It does not require seeing the -scrubber move either: a press returns only once the app is idle, which once -took nine seconds, and an eight-step triangle is drawn in 0.8. - -**The recording is sideways and variable-rate.** `simctl io recordVideo` -writes the framebuffer as it is held (portrait, the landscape app on its side) -and only when something changes. `transpose=2` stands it up, and `fps=30` -fills the still stretches before anything is cut. - -**The system language is English for the run** as well as the app, because -the status bar writes its date in the system's language. It is put back -afterwards, as `ipad-shots.rb` does. diff --git a/Tools/teaser/teaser.rb b/Tools/teaser/teaser.rb deleted file mode 100644 index c431a28..0000000 --- a/Tools/teaser/teaser.rb +++ /dev/null @@ -1,544 +0,0 @@ -#!/usr/bin/env ruby -# frozen_string_literal: true - -# Makes the teaser video for the website: a child's-eye walk from one line to -# a spiral of stars, shot on the iPad simulator and cut to 1920×1080. -# -# ruby Tools/teaser/teaser.rb # record, then compose -# ruby Tools/teaser/teaser.rb --compose # compose the last recording again -# -# Silent on purpose: the music is laid on afterwards, by hand. The film lands -# in the work directory printed at the end, with the raw recording beside it. -# -# `TortoiseBlocksUITests/TeaserTests.swift` is the script — every press, when -# each caption comes up and where the camera looks. This is the camera crew -# and the editor. The README beside this file says why each piece is the way -# it is; the short version of the four things that are not tidiness: -# -# - **The recording is mostly waiting**, and the cut is what removes it. The -# simulator writes a frame only when the screen changes, so the gaps between -# frames *are* the still stretches; each is cut down to HEAD seconds, except -# where the script asked for time (a caption to read, a camera move, a -# finished drawing). -# - **The simulator records the framebuffer as it is held**: portrait, with the -# landscape app on its side. `transpose=2` stands it up. -# - **Typing needs a hardware keyboard**, or a number field brings up half a -# screen of keys. It is the Simulator's preference, so it is switched on for -# the run, the device restarted to pick it up, and the preferences put back. -# - **Touches are drawn afterwards**, from the script's own log. The simulator -# shows none, and the log knows exactly where each press went. - -require "fileutils" -require "json" -require "open3" -require "pathname" -require "tmpdir" - -ROOT = Pathname.new(__dir__).parent.parent -WORK = Pathname.new(Dir.tmpdir) / "tortoise-teaser" -RAW = WORK / "raw" -BUNDLE_ID = "space.hiraku.tortoiseblocks" -# The 11-inch rather than the 13-inch the App Store captures use: the same -# layout in fewer points, so every block is a quarter larger in the frame. -DEVICE_NAME = "iPad Pro 11-inch (M5)" - -W = 1920 -H = 1080 -FPS = 30 -# The current video's backdrop, left to right. -GREEN = "#39FE92" -CYAN = "#42D2F7" -INK = "1c1c1e" - -# The iPad's screen in the frame: clear of the top and bottom by a little. -MARGIN = 40 -RADIUS = 26 - -# Time, in seconds. A still screen is cut down to HEAD, except around these. -HEAD = 0.22 -TAP_BEFORE = 0.35 -TAP_AFTER = 0.25 -# The press lands this long before the test logs it: measured against the -# first changed frame, 0.1–0.26s. -TAP_LEAD = 0.22 -# A typed key has landed by the time it is logged. -KEY_BEFORE = 0.35 -KEY_AFTER = 0.15 -CAPTION_HOLD = 1.0 -CAMERA_HOLD = 0.8 -CAMERA_EASE = 0.7 -# A drawing longer than this plays at FAST× — the spiral takes nine seconds. -LONG_DRAWING = 3.0 -FAST = 2 -DRAG_AFTER = 0.3 -TITLE = 3.5 -ENDING = 5.0 -FADE = 0.6 - -def run(*command) - system(*command) || abort("failed: #{command.join(' ')[0, 400]}") -end - -def simctl(*arguments) - IO.popen(["xcrun", "simctl", *arguments], err: %i[child out], &:read) -end - -# `a+b+c+…` as a balanced tree of sums. ffmpeg's expression parser recurses -# once per `+`, and a flat sum of the ~120 pieces a cut has fails to parse -# ("Cannot allocate memory"); balanced, it is seven deep. -def sum(terms) - return terms.first if terms.size == 1 - - half = terms.size / 2 - "(#{sum(terms[0, half])}+#{sum(terms[half..-1])})" -end - -def dims(file) - `magick identify -format '%w %h' '#{file}'`.split.map(&:to_i) -end - -# --------------------------------------------------------------------------- -# Recording. - -# The newest iOS runtime that has the device — 27.0 or later, the same floor -# as the screenshot rig: 26.5 does not open a document handed to it by URL. -def device - json = JSON.parse(simctl("list", "devices", "available", "-j")) - candidates = json["devices"].flat_map do |runtime, list| - runtime.include?("iOS") ? list.select { |d| d["name"] == DEVICE_NAME }.map { |d| [runtime, d] } : [] - end - abort("No #{DEVICE_NAME} simulator.") if candidates.empty? - runtime, chosen = candidates.max_by { |r, _| r } - version = runtime[/iOS-(\d+)-(\d+)/, 0].to_s.sub("iOS-", "").tr("-", ".") - abort("#{DEVICE_NAME} is only on iOS #{version}; 27.0 or newer is needed.") if version < "27" - [chosen["udid"], chosen["state"]] -end - -def restart(udid) - simctl("shutdown", udid) - simctl("boot", udid) - simctl("bootstatus", udid) -end - -def record - udid, state = device - puts "device #{udid}" - if state != "Booted" - simctl("boot", udid) - simctl("bootstatus", udid) - end - - # A hardware keyboard, so a number field raises no on-screen keyboard. It is - # the Simulator's preference — Xcode 27 has no Simulator.app to attach one - # from — so it is written first, the device restarted below to pick it up, - # and the preferences put back afterwards. - preferences = WORK / "simulator-preferences.plist" - run("defaults", "export", "com.apple.iphonesimulator", preferences.to_s) - at_exit { system("defaults", "import", "com.apple.iphonesimulator", preferences.to_s) } - run("defaults", "write", "com.apple.iphonesimulator", "ConnectHardwareKeyboard", "-bool", "true") - - # English, said to the system as well as the app: the status bar writes the - # date in the system's language. Whatever it was goes back afterwards. - languages = simctl("spawn", udid, "defaults", "read", "-g", "AppleLanguages").scan(/[A-Za-z]{2,3}(?:-[A-Za-z0-9]+)*/) - locale = simctl("spawn", udid, "defaults", "read", "-g", "AppleLocale").strip - unless languages.first == "en-US" && locale == "en_US" - at_exit do - simctl("spawn", udid, "defaults", "write", "-g", "AppleLanguages", "-array", *languages) - simctl("spawn", udid, "defaults", "write", "-g", "AppleLocale", locale) - end - simctl("spawn", udid, "defaults", "write", "-g", "AppleLanguages", "-array", "en-US") - simctl("spawn", udid, "defaults", "write", "-g", "AppleLocale", "en_US") - end - restart(udid) - simctl("status_bar", udid, "override", - "--time", "9:41", "--batteryState", "charged", "--batteryLevel", "100", - "--wifiMode", "active", "--wifiBars", "3", "--cellularMode", "notSupported") - simctl("ui", udid, "appearance", "light") - - # An empty document to build in, where the test can hand it to the app: the - # device's tmp survives the reinstall that preparing a test run does. - seed = Pathname.new(Dir.home) / "Library/Developer/CoreSimulator/Devices" / udid / "data/tmp/tbteaser" - FileUtils.rm_rf(seed) - FileUtils.mkdir_p(seed) - (seed / "My Drawing.tortoise").write( - JSON.pretty_generate({ "blocks" => [], "schemaVersion" => 1, "title" => "My Drawing" }) - ) - events = seed / "events.jsonl" - simctl("uninstall", udid, BUNDLE_ID) - - common = [ - "-project", (ROOT / "TortoiseBlocks.xcodeproj").to_s, - "-scheme", "TortoiseBlocks", - "-destination", "id=#{udid}", - "-only-testing:TortoiseBlocksUITests/TeaserTests", - "-parallel-testing-enabled", "NO", - # A failed UI test otherwise spends ten minutes in `simctl diagnose`. - "-collect-test-diagnostics", "never" - ] - # Built first, so the recording is not four minutes of a compiler. - run("xcodebuild", "build-for-testing", *common, "-quiet") - - FileUtils.rm_rf(RAW) - FileUtils.mkdir_p(RAW) - movie = RAW / "raw.mov" - _, output, recorder = Open3.popen2e("xcrun", "simctl", "io", udid, "recordVideo", "--codec=h264", "--force", movie.to_s) - started = nil - while (line = output.gets) - next unless line.include?("Recording started") - - # The recording's own clock starts here, and the test's log is in wall - # time, so this is what lines the two up. - started = Time.now.to_f - break - end - abort("the recorder never started") unless started - - result = RAW / "teaser.xcresult" - environment = { "TEST_RUNNER_TB_DOCUMENTS" => seed.to_s, "TEST_RUNNER_TB_EVENTS" => events.to_s } - passed = system(environment, "xcodebuild", "test-without-building", *common, - "-resultBundlePath", result.to_s, "-quiet") - Process.kill("INT", recorder.pid) - recorder.value - simctl("status_bar", udid, "clear") - - FileUtils.cp(events, RAW / "events.jsonl") - failure = Pathname.new("#{events}.failure.txt") - FileUtils.cp(failure, RAW / "failure.txt") if failure.exist? - (RAW / "recording.json").write(JSON.generate({ "started" => started })) - abort("The script stopped part-way; #{RAW / 'failure.txt'} has the screen it was looking at.") unless passed -end - -# --------------------------------------------------------------------------- -# Composing. - -def compose - text_tool = WORK / "text" - run("xcrun", "swiftc", "-O", (Pathname.new(__dir__) / "text.swift").to_s, "-o", text_tool.to_s) - text = lambda do |file, string, size, weight, colour| - run(text_tool.to_s, file.to_s, string, size.to_s, weight, colour) - end - layers = WORK / "layers" - FileUtils.mkdir_p(layers) - - events = RAW.join("events.jsonl").readlines.map { |line| JSON.parse(line) } - started = JSON.parse(RAW.join("recording.json").read)["started"] - at = ->(event) { event["t"] - started } - screen = events.find { |e| e["type"] == "screen" } - points = [screen["w"].to_f, screen["h"].to_f] - t_start = at.call(events.find { |e| e["type"] == "start" }) - t_end = at.call(events.find { |e| e["type"] == "end" }) - - # The source the right way up, at a constant rate. - cfr = WORK / "upright.mp4" - unless cfr.exist? && cfr.mtime > RAW.join("raw.mov").mtime - run("ffmpeg", "-v", "error", "-y", "-i", RAW.join("raw.mov").to_s, "-vf", "transpose=2,fps=#{FPS}", - "-c:v", "libx264", "-preset", "fast", "-crf", "10", "-pix_fmt", "yuv420p", cfr.to_s) - end - # While a number is being typed the caret blinks, and every blink is a - # frame; those stretches count as still, and only the keys are kept. - typing = events.select { |e| e["type"] == "typing" }.map { |e| [e["from"] - started, at.call(e)] } - changes = `ffprobe -v error -select_streams v -show_entries frame=pts_time -of csv=p=0 '#{RAW / 'raw.mov'}'` - .split.map { |s| s.delete(",").to_f }.select { |t| t > t_start && t < t_end } - .reject { |t| typing.any? { |a, b| t > a && t < b } } - - # A drag's phases, in recording time, worked back from when it returned. - gesture = lambda do |e| - distance = Math.hypot(e["x2"] - e["x"], e["y2"] - e["y"]) - release = at.call(e) - 0.05 - move_end = release - e["linger"] - move_start = move_end - distance / e["velocity"] - { press: move_start - e["hold"], move_start: move_start, move_end: move_end, release: release } - end - - # What stays: the first HEAD of every still stretch, all of every moving - # one, and whatever each event needs around it. - windows = [] - ([t_start] + changes + [t_end]).each_cons(2) { |x, y| windows << [x, [y, x + HEAD].min] } - events.each do |e| - t = at.call(e) - case e["type"] - when "caption" then windows << [t, t + CAPTION_HOLD] - when "camera" then windows << [t, t + CAMERA_HOLD] - when "tap" then windows << [t - TAP_BEFORE, t + TAP_AFTER] - when "key" then windows << [t - KEY_BEFORE, t + KEY_AFTER] - when "drag" - g = gesture.call(e) - windows << [g[:press] - 0.15, g[:release] + DRAG_AFTER] - when "linger" then windows << [t, t + e["seconds"]] - end - end - windows = windows.map { |a, b| [a.clamp(t_start, t_end), b.clamp(t_start, t_end)] }.reject { |a, b| b <= a }.sort - kept = [] - windows.each do |a, b| - if kept.any? && a <= kept.last[1] + 1.5 / FPS - kept.last[1] = [kept.last[1], b].max - else - kept << [a, b] - end - end - # Takes the script threw away — a drop that did not land, tried again. - events.select { |e| e["type"] == "cut" }.each do |e| - a = e["from"] - started - b = at.call(e) - kept = kept.flat_map do |x, y| - next [[x, y]] if y <= a || x >= b - - [[x, [y, a].min], [[x, b].max, y]].reject { |p, q| q - p < 1e-3 } - end - end - # Long drawings, from the press of play to the moment they finished (the - # linger `playDrawing` logs), are sped up; each piece carries its speed. - drawings = [] - events.each_with_index do |e, i| - next unless e["type"] == "linger" - - press = events[0...i].reverse.find { |p| p["type"] == "tap" } - next unless press && at.call(e) - at.call(press) > LONG_DRAWING - - drawings << [at.call(press) + TAP_AFTER, at.call(e)] - end - kept = kept.flat_map do |a, b| - cuts = drawings.flatten.select { |t| t > a && t < b }.sort - ([a] + cuts + [b]).each_cons(2).map do |x, y| - fast = drawings.any? { |p, q| x >= p && y <= q } - [x, y, fast ? FAST : 1] - end - end - offsets = [] - total = 0.0 - kept.each do |a, b, speed| - offsets << total - total += (b - a) / speed - end - # Recording time → film time. A moment that was cut lands where the cut is. - map = lambda do |t| - kept.each_with_index do |(a, b, speed), i| - return offsets[i] if t < a - return offsets[i] + (t - a) / speed if t <= b - end - total - end - footage = (total * FPS).floor / FPS.to_f - puts format("kept %.1fs of %.1fs, in %d pieces", total, t_end - t_start, kept.size) - - # The screen in the frame, and the camera over it. - screen_h = H - 2 * MARGIN - screen_w = (screen_h * points[0] / points[1]).round - screen_w += 1 if screen_w.odd? - screen_x = (W - screen_w) / 2 - screen_y = MARGIN - scale = screen_h / points[1] - - cues = events.select { |e| e["type"] == "camera" }.map do |e| - x, y, w, h = e["rect"] - zoom = [[points[0] / w, points[1] / h].min, 1.0].max - { at: map.call(at.call(e)), to: [x + w / 2.0, y + h / 2.0, zoom] } - end - cues = [{ at: -10.0, to: [points[0] / 2, points[1] / 2, 1.0] }] + cues - smooth = ->(u) { u = u.clamp(0.0, 1.0); u * u * (3 - 2 * u) } - camera_at = lambda do |t| - value = cues.first[:to].dup - cues.each_cons(2) do |a, b| - k = smooth.call((t - b[:at]) / CAMERA_EASE) - 3.times { |i| value[i] += (b[:to][i] - a[:to][i]) * k } - end - value - end - # The same curve, for zoompan, as a function of the film's time. - camera_expression = lambda do |i| - terms = ["(#{cues.first[:to][i].round(4)})"] - cues.each_cons(2) do |a, b| - delta = b[:to][i] - a[:to][i] - next if delta.abs < 1e-6 - - u = "clip((it-#{b[:at].round(3)})/#{CAMERA_EASE},0,1)" - terms << "(#{delta.round(5)})*#{u}*#{u}*(3-2*#{u})" - end - sum(terms) - end - to_frame = lambda do |x, y, t| - cx, cy, z = camera_at.call(t) - left = (cx - points[0] / z / 2).clamp(0, points[0] - points[0] / z) - top = (cy - points[1] / z / 2).clamp(0, points[1] - points[1] / z) - [screen_x + (x - left) * z * scale, screen_y + (y - top) * z * scale] - end - - # Stills: the backdrop, the screen's corners and shadow, the finger. - background = layers / "background.png" - run("magick", "-size", "#{H}x#{W}", "gradient:#{GREEN}-#{CYAN}", "-rotate", "-90", background.to_s) - mask = layers / "mask.png" - run("magick", "-size", "#{screen_w}x#{screen_h}", "xc:black", "-fill", "white", - "-draw", "roundrectangle 0,0 #{screen_w - 1},#{screen_h - 1} #{RADIUS},#{RADIUS}", mask.to_s) - shadow = layers / "shadow.png" - run("magick", "-size", "#{W}x#{H}", "xc:none", "-fill", "rgba(0,70,50,0.30)", - "-draw", "roundrectangle #{screen_x},#{screen_y + 12} #{screen_x + screen_w},#{screen_y + screen_h + 12} #{RADIUS},#{RADIUS}", - "-blur", "0x20", shadow.to_s) - - finger = 120 - c = finger / 2 - frames = layers / "finger" - FileUtils.rm_rf(frames) - FileUtils.mkdir_p(frames) - # A tap: the finger lands, then a ring spreads as it lifts. PNG32 on every - # frame, the empty last one included — written as greyscale it has no alpha - # and overlays as a black square. - 12.times do |i| - p = i / 11.0 - dot_alpha = i < 7 ? 0.6 : 0.6 * (1 - (i - 6) / 5.0) - ring_r = 24 + 30 * p - ring_alpha = 0.9 * (1 - p) - dot_r = 24 - 4 * [p * 2, 1].min - run("magick", "-size", "#{finger}x#{finger}", "xc:none", - "-fill", "rgba(255,255,255,#{dot_alpha.round(3)})", - "-stroke", "rgba(60,60,67,#{(dot_alpha * 0.8).round(3)})", "-strokewidth", "2", - "-draw", "circle #{c},#{c} #{c + dot_r},#{c}", - "-fill", "none", "-stroke", "rgba(255,255,255,#{ring_alpha.round(3)})", "-strokewidth", "4", - "-draw", "circle #{c},#{c} #{c + ring_r},#{c}", "PNG32:#{frames / format('%02d.png', i)}") - end - run("magick", "-size", "#{finger}x#{finger}", "xc:none", "PNG32:#{frames / '12.png'}") - tap_movie = layers / "tap.mov" - run("ffmpeg", "-v", "error", "-y", "-framerate", FPS.to_s, "-i", (frames / "%02d.png").to_s, - "-c:v", "png", tap_movie.to_s) - held = layers / "held.png" - run("magick", "-size", "#{finger}x#{finger}", "xc:none", - "-fill", "rgba(255,255,255,0.6)", "-stroke", "rgba(60,60,67,0.5)", "-strokewidth", "2", - "-draw", "circle #{c},#{c} #{c + 24},#{c}", "PNG32:#{held}") - - # Captions: dark text on a white pill, low over the screen, each up until - # the next one. - captions = events.select { |e| e["type"] == "caption" } - caption_layers = captions.each_with_index.map do |caption, i| - words = layers / "caption-text-#{i}.png" - text.call(words, caption["text"], 46, "bold", INK) - tw, th = dims(words) - pill_w = tw + 24 - pill_h = th - 4 - pad = 24 - file = layers / "caption-#{i}.png" - run("magick", "-size", "#{pill_w + 2 * pad}x#{pill_h + 2 * pad}", "xc:none", - "-fill", "rgba(0,0,0,0.22)", - "-draw", "roundrectangle #{pad},#{pad + 4} #{pad + pill_w - 1},#{pad + pill_h + 3} #{pill_h / 2},#{pill_h / 2}", - "-blur", "0x9", - "-fill", "rgba(255,255,255,0.96)", - "-draw", "roundrectangle #{pad},#{pad} #{pad + pill_w - 1},#{pad + pill_h - 1} #{pill_h / 2},#{pill_h / 2}", - words.to_s, "-gravity", "center", "-geometry", "+0-2", "-composite", "PNG32:#{file}") - from = map.call(at.call(caption)) - to = i + 1 < captions.size ? map.call(at.call(captions[i + 1])) : footage - puts format(" %5.1fs %4.1fs %s", TITLE - FADE + from, to - from, caption["text"]) - { file: file, from: from, to: to } - end - - taps = events.select { |e| e["type"] == "tap" }.map do |e| - moment = map.call(at.call(e) - TAP_LEAD) - x, y = to_frame.call(e["x"], e["y"], moment) - { at: moment, x: (x - c).round, y: (y - c).round } - end - drags = events.select { |e| e["type"] == "drag" }.map do |e| - g = gesture.call(e).transform_values { |t| map.call(t) } - x1, y1 = to_frame.call(e["x"], e["y"], g[:press]) - x2, y2 = to_frame.call(e["x2"], e["y2"], g[:release]) - g.merge(x1: x1 - c, y1: y1 - c, x2: x2 - c, y2: y2 - c) - end - - # The footage: cut, zoomed, framed, captioned, touched. - select = sum(kept.map do |a, b, speed| - piece = "between(t,#{a.round(4)},#{b.round(4)})" - speed == 1 ? piece : "#{piece}*not(mod(n,#{speed}))" - end) - zx = camera_expression.call(0) - zy = camera_expression.call(1) - zz = camera_expression.call(2) - inputs = ["-loop", "1", "-i", background.to_s, "-i", cfr.to_s, - "-loop", "1", "-i", mask.to_s, "-loop", "1", "-i", shadow.to_s] - filters = [] - filters << "[1:v]select='#{select}',setpts=N/(#{FPS}*TB)," \ - "zoompan=z='#{zz}':x='clip((#{zx})*iw/#{points[0]}-iw/zoom/2,0,iw-iw/zoom)':" \ - "y='clip((#{zy})*ih/#{points[1]}-ih/zoom/2,0,ih-ih/zoom)':d=1:s=#{screen_w}x#{screen_h}:fps=#{FPS}," \ - "format=rgba[scr0]" - filters << "[2:v]format=gray[mask]" - filters << "[scr0][mask]alphamerge[scr]" - filters << "[0:v][3:v]overlay=0:0[bg]" - filters << "[bg][scr]overlay=#{screen_x}:#{screen_y}:shortest=1[v0]" - last = "v0" - index = 4 - caption_layers.each_with_index do |layer, i| - inputs += ["-loop", "1", "-i", layer[:file].to_s] - w, h = dims(layer[:file]) - filters << "[#{index}:v]format=rgba,fade=t=in:st=#{layer[:from].round(3)}:d=0.25:alpha=1," \ - "fade=t=out:st=#{(layer[:to] - 0.25).round(3)}:d=0.25:alpha=1[c#{i}]" - filters << "[#{last}][c#{i}]overlay=#{(W - w) / 2}:#{screen_y + screen_h - h - 18}:shortest=1[vc#{i}]" - last = "vc#{i}" - index += 1 - end - taps.each_with_index do |tap, i| - inputs += ["-i", tap_movie.to_s] - filters << "[#{index}:v]format=rgba,setpts=PTS-STARTPTS+#{tap[:at].round(3)}/TB[t#{i}]" - filters << "[#{last}][t#{i}]overlay=#{tap[:x]}:#{tap[:y]}:eof_action=pass[vt#{i}]" - last = "vt#{i}" - index += 1 - end - drags.each_with_index do |d, i| - inputs += ["-loop", "1", "-i", held.to_s] - span = [d[:move_end] - d[:move_start], 0.01].max - u = "clip((t-#{d[:move_start].round(3)})/#{span.round(3)},0,1)" - x = "#{d[:x1].round(1)}+(#{(d[:x2] - d[:x1]).round(1)})*#{u}" - y = "#{d[:y1].round(1)}+(#{(d[:y2] - d[:y1]).round(1)})*#{u}" - filters << "[#{index}:v]format=rgba[h#{i}]" - filters << "[#{last}][h#{i}]overlay=x='#{x}':y='#{y}':eval=frame:shortest=1:" \ - "enable='between(t,#{d[:press].round(3)},#{d[:release].round(3)})'[vd#{i}]" - last = "vd#{i}" - index += 1 - # The lift, from the tap's own animation: its second half. - inputs += ["-i", tap_movie.to_s] - filters << "[#{index}:v]format=rgba,trim=start=#{(6.0 / FPS).round(3)}," \ - "setpts=PTS-STARTPTS+#{d[:release].round(3)}/TB[r#{i}]" - filters << "[#{last}][r#{i}]overlay=#{d[:x2].round}:#{d[:y2].round}:eof_action=pass[vr#{i}]" - last = "vr#{i}" - index += 1 - end - footage_file = WORK / "footage.mp4" - script = WORK / "filters.txt" - script.write(filters.join(";\n")) - run("ffmpeg", "-v", "error", "-y", *inputs, "-/filter_complex", script.to_s, "-map", "[#{last}]", - "-t", footage.round(3).to_s, "-r", FPS.to_s, - "-c:v", "libx264", "-preset", "medium", "-crf", "16", "-pix_fmt", "yuv420p", footage_file.to_s) - - # The cards: the icon and the name first; the icon, the name and where it - # runs at the end. - icon = ROOT / "docs" / "Icon.png" - name = layers / "name.png" - text.call(name, "Tortoise Blocks", 104, "heavy", INK) - devices = layers / "devices.png" - text.call(devices, "iPad · iPhone · Mac · Apple Vision Pro", 44, "semibold", "3a3a3c") - title = layers / "title.png" - run("magick", background.to_s, - "(", icon.to_s, "-resize", "340x340", ")", "-gravity", "north", "-geometry", "+0+250", "-composite", - name.to_s, "-gravity", "north", "-geometry", "+0+620", "-composite", title.to_s) - ending = layers / "ending.png" - run("magick", background.to_s, - "(", icon.to_s, "-resize", "300x300", ")", "-gravity", "north", "-geometry", "+0+215", "-composite", - name.to_s, "-gravity", "north", "-geometry", "+0+545", "-composite", - devices.to_s, "-gravity", "north", "-geometry", "+0+705", "-composite", ending.to_s) - - film = WORK / "teaser.mp4" - graph = [ - "[0:v]fps=#{FPS},format=yuv420p,settb=AVTB[a]", - "[1:v]fps=#{FPS},format=yuv420p,settb=AVTB[b]", - "[2:v]fps=#{FPS},format=yuv420p,settb=AVTB[c]", - "[a][b]xfade=transition=fade:duration=#{FADE}:offset=#{(TITLE - FADE).round(3)}[ab]", - "[ab][c]xfade=transition=fade:duration=#{FADE}:offset=#{(TITLE + footage - 2 * FADE).round(3)}[v]" - ].join(";") - run("ffmpeg", "-v", "error", "-y", - "-loop", "1", "-t", TITLE.to_s, "-i", title.to_s, - "-i", footage_file.to_s, - "-loop", "1", "-t", ENDING.to_s, "-i", ending.to_s, - "-filter_complex", graph, "-map", "[v]", - "-c:v", "libx264", "-preset", "slow", "-crf", "17", "-pix_fmt", "yuv420p", - "-movflags", "+faststart", film.to_s) - length = `ffprobe -v error -show_entries format=duration -of csv=p=0 '#{film}'`.to_f - puts format("%s — %.1fs, %d taps, %d drags, %d captions", film, length, taps.size, drags.size, captions.size) -end - -FileUtils.mkdir_p(WORK) -record unless ARGV.include?("--compose") -compose diff --git a/TortoiseBlocksUITests/AppPreviewTests.swift b/TortoiseBlocksUITests/AppPreviewTests.swift new file mode 100644 index 0000000..90fb835 --- /dev/null +++ b/TortoiseBlocksUITests/AppPreviewTests.swift @@ -0,0 +1,69 @@ +// iPhone and iPad only, like everything the films are made of: see +// `FilmTestCase`. +#if os(iOS) + + import XCTest + + /// The App Store previews: fifteen-odd seconds each, one per device, of a + /// program changing until it draws something — the teaser's story, told + /// short. `ruby Tools/film/previews.rb` records and cuts them. + /// + /// **Nothing here asks for a camera or a caption.** Apple's rules for a + /// preview are stricter than the website's: the footage is the screen as + /// captured, not zoomed into, and these carry no text by the maintainer's + /// choice. So each script starts from a document that already holds most + /// of a program — fifteen seconds is not enough to build one from nothing + /// and still show it drawing. + final class AppPreviewTests: FilmTestCase { + /// A square becomes a star, and the star turns orange. + /// + /// The document is `Pen Width 6`, then `Repeat 5 { Forward 150, + /// Turn Right 90 }` — thick lines, so they survive the 13-inch screen + /// being scaled down to 1600×1200, and already five times round, so a + /// single number turns the square into a star (the fifth side of a + /// square only retraces the first). + @MainActor + func testIPad() throws { + try film("My Drawing", landscape: true) { + pause(0.8) + try playDrawing(holding: 0.7) + try setNumber("Number 90", to: "144") + try playDrawing(holding: 0.7) + try drag("Pen Color", above: "Number 5", bringing: "Color blue") + pause(0.8) + try tap(button("Color blue")) + pause(0.8) + try tap(button("orange")) + pause(0.8) + try playDrawing(holding: 2) + } + } + + /// The two blocks a star is made of, from the palette, into a repeat + /// that is waiting for them — then the one number that makes it a star. + /// + /// The document is `Pen Width 6`, `Pen Color orange` and an empty + /// `Repeat 5`. Its "Add Here" makes the repeat the palette's target, so + /// the forward and the turn land inside it; the palette is a sheet on a + /// phone, raised for each block and gone again once one is added. + @MainActor + func testIPhone() throws { + try film("My Drawing", landscape: false) { + pause(0.8) + // A `Toggle` in the button style, which XCUITest calls a + // switch. + try tap(find("Add Here", type: .switch) { $0.minY < $1.minY }) + pause(0.8) + for title in ["Forward", "Turn Right"] { + try tap(frame(of: app.buttons["square.grid.2x2"].firstMatch)) + pause(0.9) + try tap(palette(title, top: 120)) + pause(0.9) + } + try setNumber("Number 90", to: "144") + try playDrawing(holding: 2) + } + } + } + +#endif diff --git a/TortoiseBlocksUITests/FilmTestCase.swift b/TortoiseBlocksUITests/FilmTestCase.swift new file mode 100644 index 0000000..7e01630 --- /dev/null +++ b/TortoiseBlocksUITests/FilmTestCase.swift @@ -0,0 +1,441 @@ +// iPhone and iPad only: the films are shot on the simulators, and orientation, +// `XCUIDevice.system` and the keyboard keys this leans on do not exist on a +// Mac, which builds this target too (for `ScreenshotTests`). +#if os(iOS) + + import XCTest + + /// What the films' scripts are made of — the teaser (`TeaserTests`) and the + /// App Store previews (`AppPreviewTests`), which `Tools/film/` records and + /// cuts. + /// + /// A script presses, and writes down when and where each press happened, + /// when each caption belongs, where the camera should look and which + /// moments are worth holding, so the driver can draw the touches and cut + /// the recording from the log. It decides nothing about the film itself. + /// + /// Everything is addressed in points on the screen as the app is held, + /// which is what both `XCUIElement.frame` and the log use; the driver maps + /// them onto the recording. + class FilmTestCase: XCTestCase { + var app: XCUIApplication! + private var events: FileHandle? + + override func setUp() { + continueAfterFailure = false + } + + /// Opens `document` and runs `script` over it, logging as it goes. + /// + /// On an iPad the document is handed over by URL from the device's + /// tmp, which survives the reinstall a test run does. A phone cannot be + /// handed one at all (see `ScreenshotTests.openFromBrowser`), so there + /// the driver seeds the app's own folder and this walks the document + /// browser to it. + @MainActor + func film(_ document: String, landscape: Bool, script: () throws -> Void) throws { + let documents = try environment("TB_DOCUMENTS") + let log = try environment("TB_EVENTS") + FileManager.default.createFile(atPath: log, contents: nil) + events = FileHandle(forWritingAtPath: log) + defer { try? events?.close() } + + app = XCUIApplication() + app.launchArguments = ["-AppleLanguages", "(en)", "-AppleLocale", "en_US"] + app.launch() + // After the launch, not before: a device turned while no app is in + // front comes back portrait when one arrives. + if landscape { XCUIDevice.shared.orientation = .landscapeLeft } + + if UIDevice.current.userInterfaceIdiom == .phone { + try openFromBrowser("\(document).tortoise") + } + else { + XCUIDevice.shared.system.open( + URL(fileURLWithPath: documents).appendingPathComponent("\(document).tortoise")) + } + + let play = app.buttons["play.fill"].firstMatch + XCTAssertTrue(play.waitForExistence(timeout: 30), "the document never opened") + record(["type": "screen", "w": app.frame.width, "h": app.frame.height]) + pause(2) + record(["type": "start"]) + + // A run that stops half-way leaves the screen as it last saw it + // beside the log, so it says what *was* there and not only what was + // missing. + do { + try script() + } + catch { + try? app.debugDescription.write( + toFile: log + ".failure.txt", atomically: true, encoding: .utf8) + throw error + } + + record(["type": "end"]) + pause(1) + } + + /// Browse → the locations → the app's folder → the file, climbing only + /// the rungs still ahead: the browser remembers where it was left. + @MainActor + private func openFromBrowser(_ file: String) throws { + let tabs = app.tabBars.firstMatch + guard tabs.waitForExistence(timeout: 30) else { throw FilmError.missing("the browser") } + tabs.buttons.element(boundBy: 2).tap() + if tapCell(file) { return } + if !tapCell("Tortoise Blocks") { + // By index: the locations' identifiers carry their localized + // names, so nothing about them is language-stable. + let location = app.cells.matching( + NSPredicate(format: "identifier BEGINSWITH 'DOC.sidebar.item.'") + ).element(boundBy: 1) + guard location.waitForExistence(timeout: 15) else { + throw FilmError.missing("On My iPhone") + } + location.tap() + guard tapCell("Tortoise Blocks", timeout: 15) else { + throw FilmError.missing("the app's folder") + } + } + guard tapCell(file, timeout: 15) else { throw FilmError.missing(file) } + } + + @MainActor + private func tapCell(_ prefix: String, timeout: TimeInterval = 5) -> Bool { + let cell = app.cells.matching(NSPredicate(format: "identifier BEGINSWITH %@", prefix)) + .firstMatch + guard cell.waitForExistence(timeout: timeout) else { return false } + cell.tap() + return true + } + + // MARK: - The hands + + /// A palette entry by its title, scrolled on screen first if it is not. + /// + /// The title is not unique: the transport's step button is "Forward" + /// too. Of every button with that label this is the one furthest left, + /// since the palette is the first column — read from one snapshot, + /// because walking a query's matches one by one raced the UI and lost an + /// element between counting and fetching it. + /// + /// **Taps here are coordinates, so nothing scrolls for them.** An + /// element's own `tap()` scrolls it into view; a coordinate below the + /// fold is simply pressed, and on the 11-inch iPad that is where Repeat + /// is — the first run put a Start Fill where the repeat should have + /// been. `top` is where the list starts: under the toolbar on an iPad, + /// under the sheet's grabber on a phone. + @MainActor + func palette(_ title: String, top: CGFloat = 150) throws -> CGRect { + let bottom = app.frame.height - 40 + var frame = try find(title, type: .button) { $0.minX < $1.minX } + for _ in 0..<4 where frame.minY < top || frame.maxY > bottom { + let distance = min(max(frame.midY - (top + bottom) / 2, -420), 420) + let from = CGPoint(x: frame.midX, y: distance > 0 ? bottom - 60 : top + 60) + swipe(from: from, to: CGPoint(x: from.x, y: from.y - distance)) + pause(0.5) + frame = try find(title, type: .button) { $0.minX < $1.minX } + } + return frame + } + + /// A button anywhere, by its accessibility label; the topmost if there + /// are several. + @MainActor + func button(_ label: String) throws -> CGRect { + try find(label, type: .button) { $0.minY < $1.minY } + } + + @MainActor + func find( + _ label: String, type: XCUIElement.ElementType, within timeout: TimeInterval = 10, + preferring order: (CGRect, CGRect) -> Bool + ) throws -> CGRect { + let deadline = Date().addingTimeInterval(timeout) + repeat { + var found: [CGRect] = [] + func walk(_ node: XCUIElementSnapshot) { + if node.elementType == type, node.label == label, !node.frame.isEmpty { + found.append(node.frame) + } + node.children.forEach(walk) + } + walk(try app.snapshot()) + if let best = found.min(by: order) { return best } + Thread.sleep(forTimeInterval: 0.3) + } while Date() < deadline + throw FilmError.missing(label) + } + + @MainActor + func frame(of element: XCUIElement) throws -> CGRect { + guard element.waitForExistence(timeout: 10) else { + throw FilmError.missing(element.description) + } + return element.frame + } + + @MainActor + func coordinate(_ point: CGPoint) -> XCUICoordinate { + app.coordinate(withNormalizedOffset: .zero) + .withOffset(CGVector(dx: point.x, dy: point.y)) + } + + /// Taps the middle of `frame`. The log gets the moment the call began as + /// well as when it returned: the touch lands somewhere between, after + /// the app has gone idle, and the driver finds it in the recording. + @MainActor + func tap(_ frame: CGRect) throws { + let point = CGPoint(x: frame.midX, y: frame.midY) + let started = Date() + coordinate(point).tap() + record([ + "type": "tap", "x": point.x, "y": point.y, "t0": started.timeIntervalSince1970, + ]) + } + + /// Drags a palette entry into the program, dropping it in the gap above + /// the row the button `row` sits on, and makes sure it arrived: a block + /// wearing `bringing` has to turn up. + /// + /// Aimed a little *above* the gap's line: a gap is hit-tested over a + /// whole row's pitch, but the row below it paints later and wins right + /// at its own top edge — and a container's header is a drop target of + /// its own that puts the block inside. + /// + /// **A drop in the simulator does not always land.** The rows part, the + /// finger lets go, and nothing is inserted — seen once in five runs, + /// with the same gesture that works every other time. So a drag that + /// brought nothing is tried again, and the failed one is logged as a + /// `cut`: the driver takes it out of the film, so what is seen is the + /// take that worked. + @MainActor + func drag(_ title: String, above row: String, bringing label: String) throws { + for _ in 0..<3 { + let from = try palette(title) + let target = try button(row) + let start = CGPoint(x: from.midX, y: from.midY) + let end = CGPoint(x: target.minX + 60, y: target.minY - 22) + let hold = 0.7 + let velocity: CGFloat = 500 + let linger = 0.5 + let started = Date() + coordinate(start).press( + forDuration: hold, thenDragTo: coordinate(end), + withVelocity: XCUIGestureVelocity(velocity), thenHoldForDuration: linger) + if (try? find(label, type: .button, within: 3) { $0.minY < $1.minY }) != nil { + record([ + "type": "drag", "x": start.x, "y": start.y, "x2": end.x, "y2": end.y, + "t0": started.timeIntervalSince1970, "hold": hold, "linger": linger, + "velocity": velocity, + ]) + return + } + record(["type": "cut", "from": started.timeIntervalSince1970]) + pause(0.5) + } + throw FilmError.missing("\(label) after dragging \(title) in") + } + + /// Scrolls with a finger: pressed only briefly, so it is a scroll and + /// not the long press that lifts a block, and held at the end so it does + /// not fling on past where it was aimed. + @MainActor + func swipe(from start: CGPoint, to end: CGPoint) { + let velocity: CGFloat = 900 + let started = Date() + coordinate(start).press( + forDuration: 0.05, thenDragTo: coordinate(end), + withVelocity: XCUIGestureVelocity(velocity), thenHoldForDuration: 0.15) + record([ + "type": "drag", "x": start.x, "y": start.y, "x2": end.x, "y2": end.y, + "t0": started.timeIntervalSince1970, "hold": 0.05, "linger": 0.15, + "velocity": velocity, + ]) + } + + /// Opens a number chip, puts in the new value and closes it again. + /// + /// **Pressed on the number pad when one comes up, typed when none + /// does.** A pad is what a phone is used with, so the touches drawn over + /// its keys are the real ones. On an iPad the driver attaches a hardware + /// keyboard, because otherwise a number field raises half a screen of + /// keys — the maintainer's own recording was made with a pointer and a + /// keyboard. Whether a phone shows its pad under the same preference + /// was not settled: one run did and the next did not, so this looks + /// rather than assumes. `⌘A` is not used for typing: selecting brought + /// iPadOS 27's number keypad up over the field and, now and then, the + /// full keyboard under it. + /// + /// Either way the field is tapped in the middle, which puts the caret + /// after the digits, so the old value goes by backspace. + @MainActor + func setNumber(_ chip: String, to value: String) throws { + try tap(button(chip)) + pause(0.8) + try tap(frame(of: app.popovers.textFields.firstMatch)) + let editing = Date() + pause(0.6) + let digits = chip.split(separator: " ").last.map(\.count) ?? 0 + let pad = + UIDevice.current.userInterfaceIdiom == .phone + && (try? key("delete", within: 2)) != nil + if pad { + for _ in 0..<digits { + try tap(key("delete")) + pause(0.25) + } + for character in value { + try tap(key(String(character))) + pause(0.25) + } + } + else { + app.typeText(String(repeating: XCUIKeyboardKey.delete.rawValue, count: digits)) + record(["type": "key"]) + pause(0.3) + for character in value { + app.typeText(String(character)) + record(["type": "key"]) + pause(0.2) + } + } + pause(0.5) + // While the field has focus its caret blinks, and a blink is a new + // frame: without this the driver would keep every one of them. + record(["type": "typing", "from": editing.timeIntervalSince1970]) + dismissPopover() + pause(0.7) + } + + /// A key on the on-screen number pad, by its label or its identifier, + /// either case: delete is `delete` by identifier on one keyboard and + /// `Delete` on the next, and its label is in the system's language. + @MainActor + func key(_ name: String, within timeout: TimeInterval = 10) throws -> CGRect { + let deadline = Date().addingTimeInterval(timeout) + repeat { + var found: CGRect? + func walk(_ node: XCUIElementSnapshot) { + if found == nil, node.elementType == .key, + [node.label, node.identifier].contains(where: { + $0.caseInsensitiveCompare(name) == .orderedSame + }), + !node.frame.isEmpty + { + found = node.frame + } + node.children.forEach(walk) + } + walk(try app.snapshot()) + if let found { return found } + Thread.sleep(forTimeInterval: 0.3) + } while Date() < deadline + throw FilmError.missing("the \(name) key") + } + + /// Closes a popover by tapping outside it — not drawn as a touch: it is + /// the tap nobody watching needs to see. + @MainActor + func dismissPopover() { + let region = app.otherElements["PopoverDismissRegion"] + if region.exists { + region.tap() + } + else { + coordinate(CGPoint(x: 400, y: 740)).tap() + } + } + + /// Presses play and waits for the drawing to finish: the scrubber's + /// value has settled, and it either moved or had time to. + /// + /// Moving is not required, because a short drawing can be over before + /// the press returns — XCUITest waits for the app to go idle, which took + /// nine seconds once, and an eight-step triangle takes 0.8. Nor is + /// ending somewhere new: changing an angle leaves the step count alone, + /// so the run ends on exactly the value the last one did. + @MainActor + func playDrawing(holding seconds: TimeInterval = 1.1) throws { + // A phone's transport is in the sheet the press raises, so there + // is no scrubber to read until the run is under way. + let scrubber = app.sliders.firstMatch + var last = scrubber.exists ? scrubber.value.map { String(describing: $0) } : nil + let pressed = Date() + try tap(frame(of: app.buttons["play.fill"].firstMatch)) + var moved = false + var settled = 0 + let deadline = Date().addingTimeInterval(60) + while Date() < deadline { + Thread.sleep(forTimeInterval: 0.2) + let now = scrubber.exists ? scrubber.value.map { String(describing: $0) } : nil + if now == last { + settled += 1 + } + else { + settled = 0 + moved = true + } + last = now + if settled >= 4, moved || Date().timeIntervalSince(pressed) > 3 { + linger(seconds) + return + } + } + throw FilmError.stalled + } + + // MARK: - The log + + func caption(_ text: String) { + record(["type": "caption", "text": text]) + } + + /// Where the camera should be looking from here on, in screen points. + /// Only the teaser has a camera; an App Store preview may not zoom. + func camera(_ rect: CGRect) { + record(["type": "camera", "rect": [rect.minX, rect.minY, rect.width, rect.height]]) + } + + /// Asks the driver to keep this moment on screen for `seconds` even + /// though nothing moves: the driver cuts every still stretch short + /// otherwise, and a finished drawing is the one thing worth looking at + /// for a while. + func linger(_ seconds: TimeInterval) { + record(["type": "linger", "seconds": seconds]) + } + + func pause(_ seconds: TimeInterval) { + Thread.sleep(forTimeInterval: seconds) + } + + func record(_ event: [String: Any]) { + var event = event + event["t"] = Date().timeIntervalSince1970 + guard let data = try? JSONSerialization.data(withJSONObject: event) else { return } + events?.write(data + Data("\n".utf8)) + } + + private func environment(_ name: String) throws -> String { + guard let value = ProcessInfo.processInfo.environment[name], !value.isEmpty else { + throw XCTSkip("\(name) is not set — run this through Tools/film/") + } + return value + } + } + + enum FilmError: Error, CustomStringConvertible { + case missing(String) + case stalled + + var description: String { + switch self { + case .missing(let what): "nothing on screen matches \(what)" + case .stalled: "the drawing never finished" + } + } + } + +#endif diff --git a/TortoiseBlocksUITests/MacPreviewTests.swift b/TortoiseBlocksUITests/MacPreviewTests.swift new file mode 100644 index 0000000..d81a946 --- /dev/null +++ b/TortoiseBlocksUITests/MacPreviewTests.swift @@ -0,0 +1,329 @@ +// The Mac only: the iPhone and iPad previews are `AppPreviewTests`, on top of +// `FilmTestCase`, which leans on things a Mac does not have. +#if os(macOS) + + import XCTest + + /// The Mac's App Store preview: the iPad's story — a square becomes a star, + /// the star turns orange — then the same program as Swift. + /// + /// `ruby Tools/film/previews.rb mac` records the window with + /// ScreenCaptureKit while this runs, pointer and clicks included, so + /// nothing is drawn over it afterwards: on a Mac the pointer *is* the touch. + /// The log is the same as the other films' (`FilmTestCase`), in screen + /// points, which is what `XCUIElement.frame` is on a Mac. + /// + /// **The runner is sandboxed**, so nothing here touches a file the driver + /// can see: the log goes back as an attachment in the result bundle, and + /// instead of waiting for a signal the test gives the recorder — which is + /// watching for the window to appear — a few seconds' start once the + /// document is open. + final class MacPreviewTests: XCTestCase { + private var app: XCUIApplication! + private var window: XCUIElement! + private var events = Data() + + override func setUp() { + continueAfterFailure = false + } + + /// The document is `Pen Width 6`, then `Repeat 5 { Forward 150, + /// Turn Right 90 }` — the iPad's. + @MainActor + func testMac() throws { + let documents = try environment("TB_DOCUMENTS") + defer { + let attachment = XCTAttachment( + data: events, uniformTypeIdentifier: "public.plain-text") + attachment.name = "events.jsonl" + attachment.lifetime = .keepAlways + add(attachment) + } + + app = XCUIApplication() + app.launchArguments = [ + "-AppleLanguages", "(en)", "-AppleLocale", "en_US", + "-NSQuitAlwaysKeepsWindows", "NO", + ] + app.launch() + // Whatever macOS restored is closed, so the one window is ours. + for restored in app.windows.allElementsBoundByIndex where restored.exists { + restored.buttons[XCUIIdentifierCloseWindow].click() + } + XCUIDevice.shared.system.open( + URL(fileURLWithPath: documents).appendingPathComponent("My Drawing.tortoise")) + // The one with the transport, not merely the one with the title: + // once the recorder has the window, macOS hangs its "sharing this + // window" control off it, and that answers to the same title. + window = + app.windows.matching(NSPredicate(format: "title == %@", "My Drawing.tortoise")) + .containing(.button, identifier: "play.fill").firstMatch + XCTAssertTrue(window.waitForExistence(timeout: 30), "the document never opened") + XCTAssertTrue( + window.buttons["play.fill"].waitForExistence(timeout: 30), + "the document has no transport") + let bounds = window.frame + record([ + "type": "screen", "w": bounds.width, "h": bounds.height, "x": bounds.minX, + "y": bounds.minY, + ]) + + // The recorder's head start: it polls for the window four times a + // second and takes about half a second to begin writing. + pause(8) + record(["type": "start"]) + + do { + pause(0.8) + try playDrawing(holding: 0.7) + try setNumber(inRow: "Turn Right,", to: "144") + try playDrawing(holding: 0.7) + try drag("Pen Color", above: "Number 5", bringing: "Pen Color,") + pause(0.6) + try click(chip(inRow: "Pen Color,")) + pause(0.8) + try click(button("orange")) + pause(0.8) + try playDrawing(holding: 1.2) + try click(frame(of: window.radioGroups.firstMatch.radioButtons.element(boundBy: 1))) + pause(0.8) + linger(2.5) + pause(3) + } + catch { + let screen = XCTAttachment(string: app.debugDescription) + screen.name = "failure.txt" + screen.lifetime = .keepAlways + add(screen) + throw error + } + record(["type": "end"]) + pause(1) + } + + // MARK: - The hands + + /// The leftmost button with this label: the palette is the first + /// column, and "Forward" is the transport's step button too. + @MainActor + private func palette(_ title: String) throws -> CGRect { + try find(title) { $0.minX < $1.minX } + } + + /// The topmost button with this label. + @MainActor + private func button(_ label: String) throws -> CGRect { + try find(label) { $0.minY < $1.minY } + } + + /// Read from one snapshot of the window — walking a query's matches one + /// by one raced the UI on iPad and lost an element between counting and + /// fetching it. A label ending in a comma matches as a prefix: that is + /// how a row is named on a Mac, where its chips fold into it + /// ("Turn Right, Number 90"). + @MainActor + private func find( + _ label: String, within timeout: TimeInterval = 10, + preferring order: (CGRect, CGRect) -> Bool + ) throws -> CGRect { + let matches: (String) -> Bool = + label.hasSuffix(",") ? { $0.hasPrefix(label) } : { $0 == label } + let deadline = Date().addingTimeInterval(timeout) + repeat { + var found: [CGRect] = [] + func walk(_ node: XCUIElementSnapshot) { + if node.elementType == .button, matches(node.label), !node.frame.isEmpty { + found.append(node.frame) + } + node.children.forEach(walk) + } + walk(try window.snapshot()) + // A popover is not inside the window on a Mac: the colour + // swatches live there. + let popover = app.popovers.firstMatch + if found.isEmpty, popover.exists { walk(try popover.snapshot()) } + if let best = found.min(by: order) { return best } + pause(0.3) + } while Date() < deadline + throw MacFilmError.missing(label) + } + + @MainActor + private func frame(of element: XCUIElement) throws -> CGRect { + guard element.waitForExistence(timeout: 10) else { + throw MacFilmError.missing(element.description) + } + return element.frame + } + + /// Where a row's first chip is: the row's leading edge, then a fixed + /// run of padding, icon slot and gaps, then the label, measured in the + /// font the row draws it in. + /// + /// **A Mac folds a row's chips into the row.** On iOS "Number 90" is a + /// button of its own inside "Turn Right, Number 90"; on a Mac the row is + /// the one element and the chip has no frame to ask for. Looking for it + /// in a screenshot of the row was tried and found nothing — the + /// runner's screenshots do not show the app's window — so it is + /// measured instead: 41pt before the label, taken from a recording + /// ("Turn Right", 62.4pt wide, puts its chip 103.5pt in), and 10pt past + /// its start, which is inside a chip whatever it holds and survives a + /// running row being drawn 3% larger. + @MainActor + private func chip(inRow row: String) throws -> CGRect { + let frame = try button(row) + let label = String(row.dropLast()) + let width = (label as NSString).size(withAttributes: [ + .font: NSFont.systemFont(ofSize: NSFont.systemFontSize) + ]).width + let x = frame.minX + 41 + width + 10 + return CGRect(x: x - 1, y: frame.midY - 1, width: 2, height: 2) + } + + /// A point on the screen, as a coordinate in the window. + @MainActor + private func coordinate(_ point: CGPoint) -> XCUICoordinate { + let origin = window.frame.origin + return window.coordinate(withNormalizedOffset: .zero) + .withOffset(CGVector(dx: point.x - origin.x, dy: point.y - origin.y)) + } + + /// Clicks the middle of `frame`. `tap()` is not an option on a Mac: it + /// compiles, runs and presses nothing (see `ScreenshotTests.press`). + @MainActor + private func click(_ frame: CGRect) throws { + let point = CGPoint(x: frame.midX, y: frame.midY) + let started = Date() + coordinate(point).click() + record([ + "type": "tap", "x": point.x, "y": point.y, "t0": started.timeIntervalSince1970, + ]) + } + + /// Drags a palette entry into the gap above the row that `row` sits + /// on, and tries again if the block did not arrive — logging the failed + /// take as a `cut` for the driver to leave out, as on iPad. + @MainActor + private func drag(_ title: String, above row: String, bringing label: String) throws { + for _ in 0..<3 { + let from = try palette(title) + let target = try button(row) + let start = CGPoint(x: from.midX, y: from.midY) + let end = CGPoint(x: target.minX + 60, y: target.minY - 16) + let hold = 0.3 + let velocity: CGFloat = 600 + let linger = 0.4 + let started = Date() + coordinate(start).click( + forDuration: hold, thenDragTo: coordinate(end), + withVelocity: XCUIGestureVelocity(velocity), thenHoldForDuration: linger) + if (try? find(label, within: 3) { $0.minY < $1.minY }) != nil { + record([ + "type": "drag", "x": start.x, "y": start.y, "x2": end.x, "y2": end.y, + "t0": started.timeIntervalSince1970, "hold": hold, "linger": linger, + "velocity": velocity, + ]) + return + } + record(["type": "cut", "from": started.timeIntervalSince1970]) + pause(0.5) + } + throw MacFilmError.missing("\(label) after dragging \(title) in") + } + + /// Opens the number chip on a row, types the new value, and closes it + /// with Return and Escape — a Mac has a keyboard, and that is what it is + /// used with. + @MainActor + private func setNumber(inRow row: String, to value: String) throws { + try click(chip(inRow: row)) + pause(0.8) + try click(frame(of: app.popovers.textFields.firstMatch)) + let editing = Date() + pause(0.4) + window.typeKey("a", modifierFlags: .command) + record(["type": "key"]) + pause(0.3) + for character in value { + window.typeText(String(character)) + record(["type": "key"]) + pause(0.2) + } + window.typeText("\r") + record(["type": "key"]) + pause(0.5) + // The caret blinks while the field has focus, and each blink is a + // frame; the driver keeps only the keys inside this span. + record(["type": "typing", "from": editing.timeIntervalSince1970]) + window.typeKey(.escape, modifierFlags: []) + pause(0.7) + } + + /// Presses play and waits for the scrubber — a number here, not the + /// spoken string iOS gives — to have moved, or had time to, and settle. + @MainActor + private func playDrawing(holding seconds: TimeInterval) throws { + let scrubber = window.sliders.firstMatch + var last = scrubber.exists ? scrubber.value.map { String(describing: $0) } : nil + let pressed = Date() + try click(frame(of: window.buttons["play.fill"])) + var moved = false + var settled = 0 + let deadline = Date().addingTimeInterval(60) + while Date() < deadline { + pause(0.2) + let now = scrubber.exists ? scrubber.value.map { String(describing: $0) } : nil + if now == last { + settled += 1 + } + else { + settled = 0 + moved = true + } + last = now + if settled >= 4, moved || Date().timeIntervalSince(pressed) > 3 { + linger(seconds) + return + } + } + throw MacFilmError.stalled + } + + // MARK: - The log + + private func linger(_ seconds: TimeInterval) { + record(["type": "linger", "seconds": seconds]) + } + + private func pause(_ seconds: TimeInterval) { + Thread.sleep(forTimeInterval: seconds) + } + + private func record(_ event: [String: Any]) { + var event = event + event["t"] = Date().timeIntervalSince1970 + guard let data = try? JSONSerialization.data(withJSONObject: event) else { return } + events.append(data + Data("\n".utf8)) + } + + private func environment(_ name: String) throws -> String { + guard let value = ProcessInfo.processInfo.environment[name], !value.isEmpty else { + throw XCTSkip("\(name) is not set — run this through Tools/film/previews.rb") + } + return value + } + + private enum MacFilmError: Error, CustomStringConvertible { + case missing(String) + case stalled + + var description: String { + switch self { + case .missing(let what): "nothing in the window matches \(what)" + case .stalled: "the drawing never finished" + } + } + } + } + +#endif diff --git a/TortoiseBlocksUITests/TeaserTests.swift b/TortoiseBlocksUITests/TeaserTests.swift index 203ab9f..e48daf7 100644 --- a/TortoiseBlocksUITests/TeaserTests.swift +++ b/TortoiseBlocksUITests/TeaserTests.swift @@ -1,56 +1,15 @@ -// iPad only: the teaser is shot on the iPad simulator, and orientation, -// `XCUIDevice.system` and the keyboard keys this leans on do not exist on a -// Mac, which builds this target too (for `ScreenshotTests`). +// iPad only, like everything the films are made of: see `FilmTestCase`. #if os(iOS) import XCTest - /// Performs the teaser video's script on an iPad, slowly enough to watch. - /// - /// The driver records the simulator's screen while this runs and cuts the - /// recording into the video; this does the pressing, and writes down when and - /// where each press happened, when each caption belongs and where the camera - /// should look, so the driver can draw the touches, lay the captions over the - /// right moments and zoom in on the part of the screen that matters. - /// - /// Everything is addressed in points on the landscape screen, which is what - /// both `XCUIElement.frame` and the log use; the driver maps them onto the - /// recording. - final class TeaserTests: XCTestCase { - private var events: FileHandle? - private var app: XCUIApplication! - - override func setUp() { - continueAfterFailure = false - } - + /// The website's teaser: one program, built by trial and error, from a + /// line to a spiral of stars — played slowly enough to watch, on the + /// 11-inch iPad. `ruby Tools/film/teaser.rb` records and cuts it. + final class TeaserTests: FilmTestCase { @MainActor func testTeaser() throws { - let documents = try environment("TB_DOCUMENTS") - let log = try environment("TB_EVENTS") - FileManager.default.createFile(atPath: log, contents: nil) - events = FileHandle(forWritingAtPath: log) - defer { try? events?.close() } - - app = XCUIApplication() - app.launchArguments = ["-AppleLanguages", "(en)", "-AppleLocale", "en_US"] - app.launch() - XCUIDevice.shared.orientation = .landscapeLeft - - let document = URL(fileURLWithPath: documents) - .appendingPathComponent("My Drawing.tortoise") - XCUIDevice.shared.system.open(document) - - let play = app.buttons["play.fill"] - XCTAssertTrue(play.waitForExistence(timeout: 30), "the document never opened") - record(["type": "screen", "w": app.frame.width, "h": app.frame.height]) - pause(2) - record(["type": "start"]) - - // A run that stops half-way leaves the screen as it last saw it - // beside the log, so it says what *was* there and not only what was - // missing. - do { + try film("My Drawing", landscape: true) { try firstSteps() try repeating() try findingTheStar() @@ -58,14 +17,6 @@ try growingTheLine() try showingTheCode() } - catch { - try? app.debugDescription.write( - toFile: log + ".failure.txt", atomically: true, encoding: .utf8) - throw error - } - - record(["type": "end"]) - pause(1) } // MARK: - The script @@ -176,7 +127,7 @@ pause(0.8) try drag("Pen Width", above: "Number 5", bringing: "Number 2") pause(0.8) - try setNumber("Number 2", to: "6", camera: nil) + try setNumber("Number 2", to: "6") camera(.all) pause(1) try playDrawing() @@ -201,7 +152,7 @@ // Still the palette's target, so this goes inside the repeat too. try tap(palette("Add to Box")) pause(0.8) - try setNumber("Number 5", to: "30", camera: nil) + try setNumber("Number 5", to: "30") camera(.all) pause(1) try playDrawing() @@ -252,277 +203,7 @@ @MainActor private func camera(_ shot: Shot) { - let rect = shot.rect(on: app.frame) - record([ - "type": "camera", - "rect": [rect.minX, rect.minY, rect.width, rect.height], - ]) - } - - // MARK: - The hands - - /// A palette entry by its title, scrolled on screen first if it is not. - /// - /// The title is not unique: the transport's step button is "Forward" too. - /// Of every button with that label this is the one furthest left, since - /// the palette is the first column — read from one snapshot, because - /// walking a query's matches one by one raced the UI and lost an element - /// between counting and fetching it. - /// - /// **Taps here are coordinates, so nothing scrolls for them.** An element's - /// own `tap()` scrolls it into view; a coordinate below the fold is simply - /// pressed, and on the 11-inch iPad that is where Repeat is — the first run - /// put a Start Fill where the repeat should have been. - @MainActor - private func palette(_ title: String) throws -> CGRect { - let top: CGFloat = 150 - let bottom = app.frame.height - 40 - var frame = try find(title, type: .button) { $0.minX < $1.minX } - for _ in 0..<4 where frame.minY < top || frame.maxY > bottom { - let distance = min(max(frame.midY - (top + bottom) / 2, -420), 420) - let from = CGPoint(x: frame.midX, y: distance > 0 ? bottom - 60 : top + 60) - swipe(from: from, to: CGPoint(x: from.x, y: from.y - distance)) - pause(0.5) - frame = try find(title, type: .button) { $0.minX < $1.minX } - } - return frame - } - - /// A button anywhere, by its accessibility label; the topmost if there - /// are several. - @MainActor - private func button(_ label: String) throws -> CGRect { - try find(label, type: .button) { $0.minY < $1.minY } - } - - @MainActor - private func find( - _ label: String, type: XCUIElement.ElementType, within timeout: TimeInterval = 10, - preferring order: (CGRect, CGRect) -> Bool - ) throws -> CGRect { - let deadline = Date().addingTimeInterval(timeout) - repeat { - var found: [CGRect] = [] - func walk(_ node: XCUIElementSnapshot) { - if node.elementType == type, node.label == label, !node.frame.isEmpty { - found.append(node.frame) - } - node.children.forEach(walk) - } - walk(try app.snapshot()) - if let best = found.min(by: order) { return best } - Thread.sleep(forTimeInterval: 0.3) - } while Date() < deadline - throw TeaserError.missing(label) - } - - @MainActor - private func frame(of element: XCUIElement) throws -> CGRect { - guard element.waitForExistence(timeout: 10) else { - throw TeaserError.missing(element.description) - } - return element.frame - } - - @MainActor - private func coordinate(_ point: CGPoint) -> XCUICoordinate { - app.coordinate(withNormalizedOffset: .zero) - .withOffset(CGVector(dx: point.x, dy: point.y)) - } - - @MainActor - private func tap(_ frame: CGRect) throws { - let point = CGPoint(x: frame.midX, y: frame.midY) - coordinate(point).tap() - record(["type": "tap", "x": point.x, "y": point.y]) - } - - /// Drags a palette entry into the program, dropping it in the gap above - /// the row the button `row` sits on, and makes sure it arrived: a block - /// wearing `bringing` has to turn up. - /// - /// Aimed a little *above* the gap's line: a gap is hit-tested over a whole - /// row's pitch, but the row below it paints later and wins right at its - /// own top edge — and a container's header is a drop target of its own - /// that puts the block inside. - /// - /// **A drop in the simulator does not always land.** The rows part, the - /// finger lets go, and nothing is inserted — seen once in five runs, with - /// the same gesture that works every other time. So a drag that brought - /// nothing is tried again, and the failed one is logged as a `cut`: the - /// driver takes it out of the film, so what is seen is the take that - /// worked. (The retry has not been needed since it was written, so the - /// cut has only been checked by reading.) - @MainActor - private func drag(_ title: String, above row: String, bringing label: String) throws { - for _ in 0..<3 { - let from = try palette(title) - let target = try button(row) - let start = CGPoint(x: from.midX, y: from.midY) - let end = CGPoint(x: target.minX + 60, y: target.minY - 22) - let hold = 0.7 - let velocity: CGFloat = 500 - let linger = 0.5 - let started = Date() - coordinate(start).press( - forDuration: hold, thenDragTo: coordinate(end), - withVelocity: XCUIGestureVelocity(velocity), thenHoldForDuration: linger) - if (try? find(label, type: .button, within: 3) { $0.minY < $1.minY }) != nil { - record([ - "type": "drag", "x": start.x, "y": start.y, "x2": end.x, "y2": end.y, - "t0": started.timeIntervalSince1970, "hold": hold, "linger": linger, - "velocity": velocity, - ]) - return - } - record(["type": "cut", "from": started.timeIntervalSince1970]) - pause(0.5) - } - throw TeaserError.missing("\(label) after dragging \(title) in") - } - - /// Scrolls with a finger: pressed only briefly, so it is a scroll and not - /// the long press that lifts a block, and held at the end so it does not - /// fling on past where it was aimed. - @MainActor - private func swipe(from start: CGPoint, to end: CGPoint) { - let velocity: CGFloat = 900 - let started = Date() - coordinate(start).press( - forDuration: 0.05, thenDragTo: coordinate(end), - withVelocity: XCUIGestureVelocity(velocity), thenHoldForDuration: 0.15) - record([ - "type": "drag", "x": start.x, "y": start.y, "x2": end.x, "y2": end.y, - "t0": started.timeIntervalSince1970, "hold": 0.05, "linger": 0.15, - "velocity": velocity, - ]) - } - - /// Opens a number chip, types the new value and closes it again. - /// - /// Typed on the hardware keyboard the driver attaches, so no on-screen - /// keyboard comes up — the way the maintainer's own recording was made, - /// with a pointer and a keyboard. The field is tapped in the middle, which - /// puts the caret after the digits, so the old value goes by backspace; - /// `⌘A` is not used, because selecting brought iPadOS 27's number keypad - /// up over the field and, now and then, the full keyboard under it. - @MainActor - private func setNumber(_ chip: String, to value: String, camera shot: Shot? = .program) - throws - { - if let shot { camera(shot) } - try tap(button(chip)) - pause(0.8) - try tap(frame(of: app.popovers.textFields.firstMatch)) - let editing = Date() - pause(0.6) - let digits = chip.split(separator: " ").last.map(\.count) ?? 0 - app.typeText(String(repeating: XCUIKeyboardKey.delete.rawValue, count: digits)) - record(["type": "key"]) - pause(0.3) - for character in value { - app.typeText(String(character)) - record(["type": "key"]) - pause(0.2) - } - pause(0.5) - // While the field has focus its caret blinks, and a blink is a new - // frame: without this the driver would keep every one of them. - record(["type": "typing", "from": editing.timeIntervalSince1970]) - dismissPopover() - pause(0.7) - } - - /// Closes a popover by tapping outside it — not drawn as a touch: it is - /// the tap nobody watching needs to see. - @MainActor - private func dismissPopover() { - let region = app.otherElements["PopoverDismissRegion"] - if region.exists { - region.tap() - } - else { - coordinate(CGPoint(x: 400, y: 740)).tap() - } - } - - /// Presses play and waits for the drawing to finish: the scrubber's value - /// has settled, and it either moved or had time to. - /// - /// Moving is not required, because a short drawing can be over before the - /// press returns — XCUITest waits for the app to go idle, which took nine - /// seconds once, and an eight-step triangle takes 0.8. Nor is ending - /// somewhere new: changing an angle leaves the step count alone, so the - /// run ends on exactly the value the last one did. - @MainActor - private func playDrawing() throws { - let scrubber = app.sliders.firstMatch - var last = scrubber.value.map { String(describing: $0) } - let pressed = Date() - try tap(frame(of: app.buttons["play.fill"])) - var moved = false - var settled = 0 - let deadline = Date().addingTimeInterval(60) - while Date() < deadline { - Thread.sleep(forTimeInterval: 0.2) - let now = scrubber.value.map { String(describing: $0) } - if now == last { - settled += 1 - } - else { - settled = 0 - moved = true - } - last = now - if settled >= 4, moved || Date().timeIntervalSince(pressed) > 3 { - linger(1.1) - return - } - } - throw TeaserError.stalled - } - - // MARK: - The log - - private func caption(_ text: String) { - record(["type": "caption", "text": text]) - } - - /// Asks the driver to keep this moment on screen for `seconds` even though - /// nothing moves: the driver cuts every still stretch short otherwise, and - /// a finished drawing is the one thing worth looking at for a while. - private func linger(_ seconds: TimeInterval) { - record(["type": "linger", "seconds": seconds]) - } - - private func pause(_ seconds: TimeInterval) { - Thread.sleep(forTimeInterval: seconds) - } - - private func record(_ event: [String: Any]) { - var event = event - event["t"] = Date().timeIntervalSince1970 - guard let data = try? JSONSerialization.data(withJSONObject: event) else { return } - events?.write(data + Data("\n".utf8)) - } - - private func environment(_ name: String) throws -> String { - guard let value = ProcessInfo.processInfo.environment[name], !value.isEmpty else { - throw XCTSkip("\(name) is not set — run this through Tools/teaser/teaser.rb") - } - return value - } - - private enum TeaserError: Error, CustomStringConvertible { - case missing(String) - case stalled - - var description: String { - switch self { - case .missing(let what): "nothing on screen matches \(what)" - case .stalled: "the drawing never finished" - } - } + camera(shot.rect(on: app.frame)) } }