diff --git a/README.md b/README.md index 4690c7e..0b20160 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@

A small, quiet video editor you can talk to.
- Rust core · GPU-rendered · titles, looks, music and motion made in place · a story board · screen recording · an assistant that edits beside you + Rust core · GPU-rendered · titles, looks, music and motion made in place · a story board · recording screen, camera and mic · an assistant that edits beside you

@@ -285,19 +285,35 @@ while you cut it. - **Shot list** copies the plan as text. - Keys: B adds a beat, C marks a cue. -**Record** (⇧⌘R, or the record button) captures the screen with macOS's own -recorder, so there is no OBS and no extra install. - -- You can record the entire screen or any one window, with the cursor and - optional clicks and microphone. -- Recording the whole screen hides Splice. Click it in the Dock to stop. -- The take comes into the media panel. In Story view it goes into the chosen - beat instead. -- The first recording asks for Screen Recording permission. It is granted to - whatever launched Splice: the app, or your terminal. Without it, Start says - so at once. A rebuilt app is new to macOS (it is signed only locally), so - after `make install-app` turn the permission off and on again in System - Settings → Privacy & Security → Screen Recording. +**Record** is a tab beside Edit and Story, and it replaces OBS for recording +yourself and your screen. Nothing extra to install. + +- **Sources:** a display or one window (with the pointer), a camera + (Continuity Camera too), a microphone, and the sound the Mac plays. Each is + live in the tab, sounds with a level meter. Splice keeps its own windows + out of the shot. +- **Scene:** Screen, Bubble (the camera in a round or square corner, small, + medium or large), Side by side, or Camera. +- **A file per source**, like OBS's Source Record: every take writes + `screen.mp4`, `camera.mp4`, `microphone.m4a` and `system.m4a` into its own + folder under `~/Movies/Splice/Recordings//`. All start at the same + moment and run as long. +- **The take comes onto the timeline** after what is there, as layers in the + scene's places. The scene is an edit, not baked in, so you can still move + the bubble, change the layout or drop a source afterwards. In Story view, + ⇧⌘R records into the chosen beat. +- **Files:** H.264 or HEVC on the Mac's hardware encoder at constant quality + (Good, High or Best), 30 or 60 fps, and the screen at full size, 4K, 1440p + or 1080p. Files are written in one-second pieces, so a crash keeps what + was recorded. +- **Pause** and resume within a take. Clicks are noted, to zoom in on later. +- **Each project keeps its own setup.** "Make this my default" sets the one + new projects start with. +- **Permissions:** Screen Recording, Camera and Microphone, asked for the + first time each is used. They are granted to whatever launched Splice: the + app, or your terminal. A rebuilt app is new to macOS (it is signed only + locally), so after `make install-app` turn its permissions off and on again + in System Settings → Privacy & Security. ## Projects @@ -349,7 +365,7 @@ with `make install-skill`) teaches Claude Code to use it. | `splice ctl save`, `versions`, `export FILE [--wait]` | versions and export | | `splice ctl ui`, `click`, `hover`, `drag`, `scroll`, `key`, `type` | operate the window as a person would, for checking the interface | | `splice ctl story read / add / set / move / remove / fill / list / shot`, `cue add / set / remove`, `view edit|story` | plan the movie: beats and cues | -| `splice ctl record sources / start / stop / status` | record a screen, a window or a region, into a beat or the timeline | +| `splice ctl record sources / setup / start / pause / stop / status` | record screen, camera, mic and the Mac's sound, each to its own file, laid out as a scene | | `splice ctl wait --name Claude` | connects an assistant and waits for your message | | `splice ctl say --text T` | shows a line in the status bar | diff --git a/crates/splice-app/build.rs b/crates/splice-app/build.rs new file mode 100644 index 0000000..4d4dab4 --- /dev/null +++ b/crates/splice-app/build.rs @@ -0,0 +1,31 @@ +//! Builds splice-capture, the Swift helper that records the screen, camera +//! and microphone (Apple's capture frameworks are made for Swift), to be +//! carried inside the splice binary. + +use std::path::PathBuf; +use std::process::Command; + +fn main() { + println!("cargo:rerun-if-changed=capture/main.swift"); + let out = PathBuf::from(std::env::var("OUT_DIR").expect("OUT_DIR")).join("splice-capture"); + let target = std::env::var("TARGET").unwrap_or_default(); + if !target.contains("apple-darwin") { + // Elsewhere there is nothing to record with. + std::fs::write(&out, b"").expect("writing an empty helper"); + return; + } + let arch = if target.starts_with("aarch64") { + "arm64" + } else { + "x86_64" + }; + let status = Command::new("xcrun") + .args(["--sdk", "macosx", "swiftc", "-O", "-swift-version", "5", "-target"]) + .arg(format!("{arch}-apple-macos13.0")) + .arg("capture/main.swift") + .arg("-o") + .arg(&out) + .status() + .expect("running swiftc (install Xcode or its command line tools: xcode-select --install)"); + assert!(status.success(), "swiftc could not build capture/main.swift"); +} diff --git a/crates/splice-app/capture/main.swift b/crates/splice-app/capture/main.swift new file mode 100644 index 0000000..114ab60 --- /dev/null +++ b/crates/splice-app/capture/main.swift @@ -0,0 +1,845 @@ +// splice-capture: the sources Splice records from (a screen or window and +// the sound the Mac plays, a camera, a microphone), kept live and shown +// small to Splice, and written on request each to its own file, all on one +// clock, so that the files start and end together. +// +// Splice runs it and talks to it: +// +// splice-capture devices cameras and microphones, as JSON +// splice-capture run the sources in the JSON line on stdin +// +// While running, it reads commands as JSON lines on stdin (record, pause, +// resume, stop, quit) and writes messages on stdout, each a kind byte and a +// little-endian u32 length before its body: "J" an event as JSON, "F" a +// preview frame (u8 source, u32 width, u32 height, then BGRA pixels). +// Closing stdin quits, finishing any file being written. + +import AVFoundation +import CoreImage +import CoreMedia +import Foundation +import ScreenCaptureKit + +// MARK: - Talking to Splice + +let out = FileHandle.standardOutput +let outLock = NSLock() + +func send(_ kind: UInt8, _ body: Data) { + var message = Data([kind]) + var length = UInt32(body.count).littleEndian + message.append(Data(bytes: &length, count: 4)) + message.append(body) + outLock.lock() + out.write(message) + outLock.unlock() +} + +func event(_ name: String, _ fields: [String: Any] = [:]) { + var all = fields + all["event"] = name + if let data = try? JSONSerialization.data(withJSONObject: all, options: [.sortedKeys]) { + send(UInt8(ascii: "J"), data) + } +} + +func fail(_ message: String) -> Never { + event("error", ["message": message]) + exit(1) +} + +// MARK: - Devices + +func cameras() -> [AVCaptureDevice] { + var types: [AVCaptureDevice.DeviceType] = [.builtInWideAngleCamera, .deskViewCamera] + if #available(macOS 14.0, *) { + types += [.external, .continuityCamera] + } else { + types.append(.externalUnknown) + } + return AVCaptureDevice.DiscoverySession(deviceTypes: types, mediaType: .video, position: .unspecified).devices +} + +func microphones() -> [AVCaptureDevice] { + var types: [AVCaptureDevice.DeviceType] = [] + if #available(macOS 14.0, *) { + types += [.microphone, .external] + } else { + types += [.builtInMicrophone, .externalUnknown] + } + return AVCaptureDevice.DiscoverySession(deviceTypes: types, mediaType: .audio, position: .unspecified).devices +} + +func listDevices() { + func describe(_ d: AVCaptureDevice, _ fallback: AVCaptureDevice?) -> [String: Any] { + ["id": d.uniqueID, "name": d.localizedName, "default": d.uniqueID == fallback?.uniqueID] + } + let camera = AVCaptureDevice.default(for: .video) + let microphone = AVCaptureDevice.default(for: .audio) + let body: [String: Any] = [ + "cameras": cameras().map { describe($0, camera) }, + "microphones": microphones().map { describe($0, microphone) }, + "camera_access": access(.video), + "microphone_access": access(.audio), + "screen_access": CGPreflightScreenCaptureAccess(), + ] + let data = try! JSONSerialization.data(withJSONObject: body, options: [.sortedKeys]) + FileHandle.standardOutput.write(data) + FileHandle.standardOutput.write("\n".data(using: .utf8)!) +} + +func access(_ media: AVMediaType) -> String { + switch AVCaptureDevice.authorizationStatus(for: media) { + case .authorized: return "granted" + case .denied: return "denied" + case .restricted: return "restricted" + case .notDetermined: return "unasked" + @unknown default: return "unknown" + } +} + +/// Asks for camera or microphone access if it has not been asked, and waits. +func allow(_ media: AVMediaType) -> Bool { + switch AVCaptureDevice.authorizationStatus(for: media) { + case .authorized: return true + case .notDetermined: + event("asking", ["source": media == .video ? "camera" : "microphone"]) + let done = DispatchSemaphore(value: 0) + var granted = false + AVCaptureDevice.requestAccess(for: media) { ok in + granted = ok + done.signal() + } + done.wait() + return granted + default: return false + } +} + +// MARK: - What to capture + +struct Screen: Decodable { + /// A display, by its CoreGraphics id. + var display: UInt32? + var window: UInt32? + /// Part of the display, in points from its top left: x, y, width, height. + var region: [Double]? +} + +struct Config: Decodable { + var screen: Screen? + var systemAudio: Bool = false + var camera: String? + var microphone: String? + var codec: String = "h264" + /// 0 to 1: the encoder's constant quality. + var quality: Double = 0.65 + var fps: Int = 30 + /// The screen file is made no taller than this (keeping its shape). + var maxHeight: Int? + var cursor: Bool = true + /// A process whose windows stay out of the shot (Splice). + var exclude: Int32? + /// The longest side of preview frames; 0 sends none. + var preview: Int = 480 + var previewFps: Int = 12 + + enum CodingKeys: String, CodingKey { + case screen, camera, microphone, codec, quality, fps, cursor, exclude, preview + case systemAudio = "system_audio" + case maxHeight = "max_height" + case previewFps = "preview_fps" + } + + init(from decoder: Decoder) throws { + let c = try decoder.container(keyedBy: CodingKeys.self) + screen = try c.decodeIfPresent(Screen.self, forKey: .screen) + systemAudio = try c.decodeIfPresent(Bool.self, forKey: .systemAudio) ?? false + camera = try c.decodeIfPresent(String.self, forKey: .camera) + microphone = try c.decodeIfPresent(String.self, forKey: .microphone) + codec = try c.decodeIfPresent(String.self, forKey: .codec) ?? "h264" + quality = try c.decodeIfPresent(Double.self, forKey: .quality) ?? 0.65 + fps = try c.decodeIfPresent(Int.self, forKey: .fps) ?? 30 + maxHeight = try c.decodeIfPresent(Int.self, forKey: .maxHeight) + cursor = try c.decodeIfPresent(Bool.self, forKey: .cursor) ?? true + exclude = try c.decodeIfPresent(Int32.self, forKey: .exclude) + preview = try c.decodeIfPresent(Int.self, forKey: .preview) ?? 480 + previewFps = try c.decodeIfPresent(Int.self, forKey: .previewFps) ?? 12 + } +} + +/// The sources, in the order their files and preview frames are numbered. +enum Feed: Int, CaseIterable { + case screen = 0, camera = 1, microphone = 2, system = 3 + + var name: String { + switch self { + case .screen: return "screen" + case .camera: return "camera" + case .microphone: return "microphone" + case .system: return "system" + } + } + + var isVideo: Bool { self == .screen || self == .camera } + var file: String { name + (isVideo ? ".mp4" : ".m4a") } +} + +// MARK: - Time + +let hostClock = CMClockGetHostTimeClock() + +func hostNow() -> CMTime { CMClockGetTime(hostClock) } + +/// The same buffer, every timing shifted by `delta`. +func shifted(_ buffer: CMSampleBuffer, by delta: CMTime) -> CMSampleBuffer? { + var count: CMItemCount = 0 + CMSampleBufferGetSampleTimingInfoArray(buffer, entryCount: 0, arrayToFill: nil, entriesNeededOut: &count) + var timing = [CMSampleTimingInfo](repeating: CMSampleTimingInfo(), count: max(count, 1)) + CMSampleBufferGetSampleTimingInfoArray(buffer, entryCount: timing.count, arrayToFill: &timing, entriesNeededOut: &count) + for i in 0.. CMSampleBuffer? { + shifted(buffer, by: CMTimeSubtract(time, CMSampleBufferGetPresentationTimeStamp(buffer))) +} + +// MARK: - Files + +/// One source's file. +final class Sink { + let feed: Feed + let writer: AVAssetWriter + let input: AVAssetWriterInput + var appended = 0 + var dropped = 0 + + init(feed: Feed, url: URL, sample: CMSampleBuffer, config: Config) throws { + self.feed = feed + try? FileManager.default.removeItem(at: url) + writer = try AVAssetWriter(outputURL: url, fileType: feed.isVideo ? .mp4 : .m4a) + // Written in fragments, so a crash leaves a file that plays. + writer.movieFragmentInterval = CMTime(seconds: 1, preferredTimescale: 600) + let format = CMSampleBufferGetFormatDescription(sample) + if feed.isVideo { + let size = format.map { CMVideoFormatDescriptionGetDimensions($0) } ?? CMVideoDimensions(width: 1920, height: 1080) + let hevc = config.codec == "hevc" + let quality = max(0.05, min(1.0, config.quality)) + var compression: [String: Any] = [ + AVVideoExpectedSourceFrameRateKey: config.fps, + AVVideoMaxKeyFrameIntervalDurationKey: 2.0, + AVVideoAllowFrameReorderingKey: false, + ] + if hevc { + compression[AVVideoQualityKey] = quality + } else { + // H.264 here takes a bitrate, not a quality: bits for each + // pixel of each frame, more as the quality rises. + let perPixel = 0.04 + quality * 0.16 + compression[AVVideoAverageBitRateKey] = Int(Double(size.width) * Double(size.height) * Double(max(1, config.fps)) * perPixel) + } + input = AVAssetWriterInput(mediaType: .video, outputSettings: [ + AVVideoCodecKey: hevc ? AVVideoCodecType.hevc : AVVideoCodecType.h264, + AVVideoWidthKey: Int(size.width), + AVVideoHeightKey: Int(size.height), + AVVideoCompressionPropertiesKey: compression, + ]) + } else { + let asbd = format.flatMap { CMAudioFormatDescriptionGetStreamBasicDescription($0)?.pointee } + let channels = min(Int(asbd?.mChannelsPerFrame ?? 2), 2) + input = AVAssetWriterInput(mediaType: .audio, outputSettings: [ + AVFormatIDKey: kAudioFormatMPEG4AAC, + AVSampleRateKey: asbd?.mSampleRate ?? 48000, + AVNumberOfChannelsKey: channels, + AVEncoderBitRateKey: channels == 1 ? 128_000 : 256_000, + ]) + } + input.expectsMediaDataInRealTime = true + guard writer.canAdd(input) else { throw NSError(domain: "splice-capture", code: 1) } + writer.add(input) + guard writer.startWriting() else { + throw writer.error ?? NSError(domain: "splice-capture", code: 2) + } + writer.startSession(atSourceTime: .zero) + } + + func append(_ buffer: CMSampleBuffer) { + guard writer.status == .writing else { return } + if input.isReadyForMoreMediaData, input.append(buffer) { + appended += 1 + } else { + dropped += 1 + } + } + + func finish(at end: CMTime, _ done: @escaping () -> Void) { + guard writer.status == .writing else { return done() } + input.markAsFinished() + writer.endSession(atSourceTime: end) + writer.finishWriting(completionHandler: done) + } +} + +// MARK: - The capture + +final class Capture: NSObject, SCStreamOutput, SCStreamDelegate, + AVCaptureVideoDataOutputSampleBufferDelegate, AVCaptureAudioDataOutputSampleBufferDelegate +{ + let config: Config + /// Everything that touches a buffer happens here, one at a time. + let queue = DispatchQueue(label: "splice-capture", qos: .userInteractive) + + var streams: [SCStream] = [] + var cameraSession: AVCaptureSession? + var micSession: AVCaptureSession? + var cameraOutput: AVCaptureVideoDataOutput? + var micOutput: AVCaptureAudioDataOutput? + + /// The latest frame from each video source. + var latest: [Feed: CMSampleBuffer] = [:] + var live: Set = [] + var expected: Set = [] + + // While recording: the host time the files start at, how long has been + // paused, and since when. + var dir: URL? + var zero: CMTime? + var pausedFor = CMTime.zero + var pausedAt: CMTime? + var sinks: [Feed: Sink] = [:] + + var lastPreview: [Feed: CMTime] = [:] + let context = CIContext(options: [.cacheIntermediates: false]) + var peaks: [Feed: Float] = [:] + var lastLevels = CMTime.zero + + init(config: Config) { + self.config = config + } + + // MARK: Starting + + func start() async throws { + if config.screen != nil || config.systemAudio { + if !CGPreflightScreenCaptureAccess() { + CGRequestScreenCaptureAccess() + throw Problem("Splice needs Screen Recording permission: turn it on in System Settings → Privacy & Security → Screen Recording, then reopen Splice") + } + try await startScreen() + } + if let id = config.camera { + guard allow(.video) else { + throw Problem("Splice needs Camera permission: turn it on in System Settings → Privacy & Security → Camera") + } + try startCamera(id) + } + if let id = config.microphone { + guard allow(.audio) else { + throw Problem("Splice needs Microphone permission: turn it on in System Settings → Privacy & Security → Microphone") + } + try startMicrophone(id) + } + } + + func startScreen() async throws { + let content = try await SCShareableContent.excludingDesktopWindows(false, onScreenWindowsOnly: true) + let away = content.applications.filter { $0.processID == config.exclude } + guard let shown = config.screen else { + if config.systemAudio { try await startSound(content, away) } + return + } + let filter: SCContentFilter + let settings = SCStreamConfiguration() + var size: CGSize + var scale: CGFloat = 2 + // A window's filter hears only its own app: the Mac's sound then + // comes from the whole display, apart. + var sound = config.systemAudio + if let id = shown.window { + guard let window = content.windows.first(where: { $0.windowID == id }) else { + throw Problem("that window is not on screen any more") + } + filter = SCContentFilter(desktopIndependentWindow: window) + size = window.frame.size + scale = backing(for: window.frame) + sound = false + } else { + let wanted = shown.display ?? CGMainDisplayID() + guard let display = content.displays.first(where: { $0.displayID == wanted }) ?? content.displays.first else { + throw Problem("no display to record") + } + filter = SCContentFilter(display: display, excludingApplications: away, exceptingWindows: []) + size = CGSize(width: display.width, height: display.height) + scale = backing(for: CGDisplayBounds(display.displayID)) + if let r = shown.region, r.count == 4 { + settings.sourceRect = CGRect(x: r[0], y: r[1], width: r[2], height: r[3]) + size = CGSize(width: r[2], height: r[3]) + } + } + if #available(macOS 14.0, *) { + scale = CGFloat(filter.pointPixelScale) + } + var pixels = CGSize(width: size.width * scale, height: size.height * scale) + if let most = config.maxHeight, most > 0, pixels.height > CGFloat(most) { + pixels = CGSize(width: pixels.width * CGFloat(most) / pixels.height, height: CGFloat(most)) + } + settings.width = max(2, Int(pixels.width) / 2 * 2) + settings.height = max(2, Int(pixels.height) / 2 * 2) + settings.minimumFrameInterval = CMTime(value: 1, timescale: CMTimeScale(max(1, config.fps))) + settings.pixelFormat = kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange + settings.colorMatrix = CGDisplayStream.yCbCrMatrix_ITU_R_709_2 + settings.showsCursor = config.cursor + settings.queueDepth = 8 + if sound { hear(settings) } + let stream = SCStream(filter: filter, configuration: settings, delegate: self) + try stream.addStreamOutput(self, type: .screen, sampleHandlerQueue: queue) + expected.insert(.screen) + if sound { + try stream.addStreamOutput(self, type: .audio, sampleHandlerQueue: queue) + expected.insert(.system) + } + try await stream.startCapture() + streams.append(stream) + if config.systemAudio, !sound { + try await startSound(content, away) + } + } + + /// The Mac's sound, from the main display, its picture kept tiny and unread. + func startSound(_ content: SCShareableContent, _ away: [SCRunningApplication]) async throws { + guard let display = content.displays.first(where: { $0.displayID == CGMainDisplayID() }) ?? content.displays.first else { + throw Problem("no display to hear the Mac's sound from") + } + let filter = SCContentFilter(display: display, excludingApplications: away, exceptingWindows: []) + let settings = SCStreamConfiguration() + settings.width = 64 + settings.height = 64 + settings.minimumFrameInterval = CMTime(value: 1, timescale: 1) + hear(settings) + let stream = SCStream(filter: filter, configuration: settings, delegate: self) + try stream.addStreamOutput(self, type: .audio, sampleHandlerQueue: queue) + expected.insert(.system) + try await stream.startCapture() + streams.append(stream) + } + + func hear(_ settings: SCStreamConfiguration) { + settings.capturesAudio = true + settings.excludesCurrentProcessAudio = true + settings.sampleRate = 48000 + settings.channelCount = 2 + } + + func backing(for rect: CGRect) -> CGFloat { + NSScreenScale(containing: rect) + } + + func startCamera(_ id: String) throws { + guard let device = AVCaptureDevice(uniqueID: id) ?? cameras().first(where: { $0.localizedName == id }) else { + throw Problem("no camera \(id)") + } + let session = AVCaptureSession() + session.beginConfiguration() + let input = try AVCaptureDeviceInput(device: device) + guard session.canAddInput(input) else { throw Problem("the camera is busy") } + session.addInput(input) + // The largest picture no bigger than 1080p at the frame rate wanted. + let fps = Double(config.fps) + let fitting = device.formats.filter { f in + let d = CMVideoFormatDescriptionGetDimensions(f.formatDescription) + return d.height <= 1080 && d.width <= 1920 + && f.videoSupportedFrameRateRanges.contains { $0.maxFrameRate >= fps - 0.5 } + } + let best = fitting.max { a, b in + let (x, y) = (CMVideoFormatDescriptionGetDimensions(a.formatDescription), CMVideoFormatDescriptionGetDimensions(b.formatDescription)) + return Int(x.width) * Int(x.height) < Int(y.width) * Int(y.height) + } + if let best, (try? device.lockForConfiguration()) != nil { + device.activeFormat = best + if let range = best.videoSupportedFrameRateRanges.first(where: { $0.maxFrameRate >= fps - 0.5 }) { + let frame = CMTime(value: 1000, timescale: CMTimeScale(min(fps, range.maxFrameRate) * 1000)) + device.activeVideoMinFrameDuration = frame + device.activeVideoMaxFrameDuration = frame + } + device.unlockForConfiguration() + } + let output = AVCaptureVideoDataOutput() + output.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange] + output.alwaysDiscardsLateVideoFrames = true + output.setSampleBufferDelegate(self, queue: queue) + guard session.canAddOutput(output) else { throw Problem("the camera cannot be read") } + session.addOutput(output) + session.commitConfiguration() + session.startRunning() + cameraSession = session + cameraOutput = output + expected.insert(.camera) + } + + func startMicrophone(_ id: String) throws { + guard let device = AVCaptureDevice(uniqueID: id) ?? microphones().first(where: { $0.localizedName == id }) else { + throw Problem("no microphone \(id)") + } + let session = AVCaptureSession() + session.beginConfiguration() + let input = try AVCaptureDeviceInput(device: device) + guard session.canAddInput(input) else { throw Problem("the microphone is busy") } + session.addInput(input) + let output = AVCaptureAudioDataOutput() + // One channel of voice, at 48 kHz. + output.audioSettings = [ + AVFormatIDKey: kAudioFormatLinearPCM, + AVSampleRateKey: 48000, + AVNumberOfChannelsKey: 1, + AVLinearPCMBitDepthKey: 32, + AVLinearPCMIsFloatKey: true, + AVLinearPCMIsNonInterleaved: false, + ] + output.setSampleBufferDelegate(self, queue: queue) + guard session.canAddOutput(output) else { throw Problem("the microphone cannot be read") } + session.addOutput(output) + session.commitConfiguration() + session.startRunning() + micSession = session + micOutput = output + expected.insert(.microphone) + } + + // MARK: Buffers + + func stream(_ stream: SCStream, didOutputSampleBuffer buffer: CMSampleBuffer, of type: SCStreamOutputType) { + guard buffer.isValid else { return } + switch type { + case .screen: + // Frames that are not complete say only that nothing changed. + guard let attachments = CMSampleBufferGetSampleAttachmentsArray(buffer, createIfNecessary: false) as? [[SCStreamFrameInfo: Any]], + let raw = attachments.first?[.status] as? Int, SCFrameStatus(rawValue: raw) == .complete + else { return } + take(buffer, .screen, CMSampleBufferGetPresentationTimeStamp(buffer)) + case .audio: + take(buffer, .system, CMSampleBufferGetPresentationTimeStamp(buffer)) + default: + break + } + } + + func stream(_ stream: SCStream, didStopWithError error: Error) { + queue.async { + if self.zero != nil { self.finish(reason: "the screen stopped being recorded: \(error.localizedDescription)") } + else { fail("the screen stopped being recorded: \(error.localizedDescription)") } + } + } + + func captureOutput(_ output: AVCaptureOutput, didOutput buffer: CMSampleBuffer, from connection: AVCaptureConnection) { + let feed: Feed = output === cameraOutput ? .camera : .microphone + let session = feed == .camera ? cameraSession : micSession + // On the host clock, as the screen is. + let time = CMSampleBufferGetPresentationTimeStamp(buffer) + let clock = session?.synchronizationClock ?? hostClock + take(buffer, feed, CMSyncConvertTime(time, from: clock, to: hostClock), clock: clock) + } + + /// A buffer from `feed`, presented at `time` on the host clock. + func take(_ buffer: CMSampleBuffer, _ feed: Feed, _ time: CMTime, clock: CMClock? = nil) { + var buffer = buffer + if clock != nil, clock !== hostClock { + let delta = CMTimeSubtract(time, CMSampleBufferGetPresentationTimeStamp(buffer)) + guard let moved = shifted(buffer, by: delta) else { return } + buffer = moved + } + if !live.contains(feed) { + live.insert(feed) + if live == expected { event("live", ["sources": expected.map(\.name).sorted()]) } + } + if feed.isVideo { + latest[feed] = buffer + preview(buffer, feed, time) + } else { + level(buffer, feed, time) + } + guard let zero, pausedAt == nil, let dir else { return } + let local = CMTimeSubtract(CMTimeSubtract(time, zero), pausedFor) + if local < .zero { return } + guard let moved = shifted(buffer, by: CMTimeSubtract(local, time)) else { return } + sink(feed, dir, moved)?.append(moved) + } + + func sink(_ feed: Feed, _ dir: URL, _ sample: CMSampleBuffer) -> Sink? { + if let s = sinks[feed] { return s } + do { + let s = try Sink(feed: feed, url: dir.appendingPathComponent(feed.file), sample: sample, config: config) + sinks[feed] = s + return s + } catch { + event("warning", ["message": "could not write the \(feed.name): \(error.localizedDescription)"]) + return nil + } + } + + // MARK: Recording + + func record(into dir: URL) { + guard zero == nil else { return event("warning", ["message": "already recording"]) } + do { + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + } catch { + return event("warning", ["message": "could not make \(dir.path): \(error.localizedDescription)"]) + } + let now = hostNow() + self.dir = dir + zero = now + pausedFor = .zero + pausedAt = nil + // A picture that is not changing sends no frames: the files begin + // with what each source shows now. + for (feed, frame) in latest { + if let first = at(frame, .zero) { sink(feed, dir, first)?.append(first) } + } + event("recording", ["dir": dir.path, "wall": wall(at: now)]) + } + + func pause() { + guard zero != nil, pausedAt == nil else { return } + let now = hostNow() + pausedAt = now + event("paused", ["seconds": seconds(), "wall": wall(at: now)]) + } + + func resume() { + guard let since = pausedAt, let zero, let dir else { return } + let now = hostNow() + pausedFor = CMTimeAdd(pausedFor, CMTimeSubtract(now, since)) + pausedAt = nil + let local = CMTimeSubtract(CMTimeSubtract(now, zero), pausedFor) + for (feed, frame) in latest { + if let again = at(frame, local) { sink(feed, dir, again)?.append(again) } + } + event("resumed", ["seconds": seconds(), "wall": wall(at: now)]) + } + + /// How long has been recorded, not counting pauses. + func seconds() -> Double { + guard let zero else { return 0 } + let end = pausedAt ?? hostNow() + return CMTimeSubtract(CMTimeSubtract(end, zero), pausedFor).seconds + } + + /// Finishes the files. Each ends at the same moment. + func finish(reason: String? = nil, then: (() -> Void)? = nil) { + guard zero != nil, let dir else { + then?() + return + } + let length = CMTime(seconds: seconds(), preferredTimescale: 600) + let group = DispatchGroup() + var files: [String: Any] = [:] + var dropped: [String: Int] = [:] + for (feed, s) in sinks { + // The picture holds to the end, however long since it changed. + if feed.isVideo, let frame = latest[feed], let last = at(frame, length) { s.append(last) } + group.enter() + files[feed.name] = dir.appendingPathComponent(feed.file).path + if s.dropped > 0 { dropped[feed.name] = s.dropped } + s.finish(at: length) { group.leave() } + } + group.wait() + let failed = sinks.filter { $0.value.writer.status != .completed } + .map { "\($0.key.name): \($0.value.writer.error?.localizedDescription ?? "not finished")" } + sinks = [:] + zero = nil + self.dir = nil + pausedAt = nil + var fields: [String: Any] = ["dir": dir.path, "files": files, "seconds": length.seconds] + if !dropped.isEmpty { fields["dropped"] = dropped } + if !failed.isEmpty { fields["failed"] = failed } + if let reason { fields["reason"] = reason } + event("stopped", fields) + then?() + } + + func shutdown() { + let done = DispatchSemaphore(value: 0) + queue.async { + self.finish { done.signal() } + } + done.wait() + cameraSession?.stopRunning() + micSession?.stopRunning() + for stream in streams { + let stopped = DispatchSemaphore(value: 0) + stream.stopCapture { _ in stopped.signal() } + _ = stopped.wait(timeout: .now() + 2) + } + } + + // MARK: Preview and levels + + func preview(_ buffer: CMSampleBuffer, _ feed: Feed, _ time: CMTime) { + guard config.preview > 0, let pixels = CMSampleBufferGetImageBuffer(buffer) else { return } + if let last = lastPreview[feed], CMTimeSubtract(time, last).seconds < 1.0 / Double(max(1, config.previewFps)) { + return + } + lastPreview[feed] = time + let image = CIImage(cvPixelBuffer: pixels) + let (w, h) = (image.extent.width, image.extent.height) + let k = min(1, CGFloat(config.preview) / max(w, h)) + let (pw, ph) = (max(2, Int(w * k)), max(2, Int(h * k))) + let small = image.transformed(by: CGAffineTransform(scaleX: CGFloat(pw) / w, y: CGFloat(ph) / h)) + var body = Data(count: 9 + pw * ph * 4) + body.withUnsafeMutableBytes { raw in + let p = raw.baseAddress! + p.storeBytes(of: UInt8(feed.rawValue), as: UInt8.self) + p.advanced(by: 1).storeBytes(of: UInt32(pw).littleEndian, toByteOffset: 0, as: UInt32.self) + p.advanced(by: 5).storeBytes(of: UInt32(ph).littleEndian, toByteOffset: 0, as: UInt32.self) + context.render( + small, toBitmap: p.advanced(by: 9), rowBytes: pw * 4, + bounds: CGRect(x: 0, y: 0, width: pw, height: ph), format: .BGRA8, + colorSpace: CGColorSpace(name: CGColorSpace.sRGB)) + } + send(UInt8(ascii: "F"), body) + } + + /// The loudest sample of each sound, sent ten times a second. + func level(_ buffer: CMSampleBuffer, _ feed: Feed, _ time: CMTime) { + var peak: Float = peaks[feed] ?? 0 + // Float samples, all channels one after another or interleaved: + // the loudest of them is the same either way. + if let format = CMSampleBufferGetFormatDescription(buffer), + let asbd = CMAudioFormatDescriptionGetStreamBasicDescription(format)?.pointee, + asbd.mFormatFlags & kAudioFormatFlagIsFloat != 0, asbd.mBitsPerChannel == 32, + let block = CMSampleBufferGetDataBuffer(buffer) + { + var length = 0 + var data: UnsafeMutablePointer? + if CMBlockBufferGetDataPointer(block, atOffset: 0, lengthAtOffsetOut: nil, totalLengthOut: &length, dataPointerOut: &data) == kCMBlockBufferNoErr, + let data, CMBlockBufferIsRangeContiguous(block, atOffset: 0, length: length) + { + let n = length / 4 + data.withMemoryRebound(to: Float.self, capacity: n) { samples in + for i in 0..= 0.1 { + lastLevels = time + var fields: [String: Any] = [:] + for (f, p) in peaks { fields[f.name] = Double(p) } + if zero != nil { fields["seconds"] = seconds() } + event("levels", fields) + peaks = [:] + } + } +} + +struct Problem: LocalizedError { + let message: String + init(_ message: String) { self.message = message } + var errorDescription: String? { message } +} + +/// The backing scale of the screen holding most of `rect` (CoreGraphics +/// coordinates, from the main display's top left). +func NSScreenScale(containing rect: CGRect) -> CGFloat { + var ids = [CGDirectDisplayID](repeating: 0, count: 16) + var count: UInt32 = 0 + CGGetDisplaysWithRect(rect, 16, &ids, &count) + guard count > 0, let mode = CGDisplayCopyDisplayMode(ids[0]) else { return 2 } + return CGFloat(mode.pixelWidth) / CGFloat(max(1, mode.width)) +} + +/// Seconds since 1970 at host time `t`. +func wall(at t: CMTime) -> Double { + Date().timeIntervalSince1970 - CMTimeSubtract(hostNow(), t).seconds +} + +// MARK: - Main + +let arguments = CommandLine.arguments.dropFirst() +switch arguments.first { +case "devices": + listDevices() + exit(0) +case "run": + break +default: + FileHandle.standardError.write("usage: splice-capture devices | run (the sources as a JSON line on stdin)\n".data(using: .utf8)!) + exit(2) +} + +guard let line = readLine(), let data = line.data(using: .utf8) else { fail("no sources given") } +let config: Config +do { + config = try JSONDecoder().decode(Config.self, from: data) +} catch { + fail("could not read the sources: \(error)") +} + +let capture = Capture(config: config) +signal(SIGPIPE, SIG_IGN) + +Task { + do { + try await capture.start() + } catch { + fail(error.localizedDescription) + } + event("started", ["sources": capture.expected.map(\.name).sorted()]) + // A source that never sends anything is worth knowing about. + capture.queue.asyncAfter(deadline: .now() + 4) { + let silent = capture.expected.subtracting(capture.live) + if !silent.isEmpty { event("warning", ["message": "nothing yet from: \(silent.map(\.name).sorted().joined(separator: ", "))"]) } + } +} + +for sig in [SIGINT, SIGTERM] { + signal(sig, SIG_IGN) + let source = DispatchSource.makeSignalSource(signal: sig, queue: .global()) + source.setEventHandler { + capture.shutdown() + exit(0) + } + source.resume() + _ = Unmanaged.passRetained(source) +} + +Thread.detachNewThread { + while let line = readLine() { + guard let data = line.data(using: .utf8), + let command = (try? JSONSerialization.jsonObject(with: data)) as? [String: Any], + let name = command["cmd"] as? String + else { continue } + switch name { + case "record": + guard let path = command["dir"] as? String else { continue } + capture.queue.async { capture.record(into: URL(fileURLWithPath: path)) } + case "pause": + capture.queue.async { capture.pause() } + case "resume": + capture.queue.async { capture.resume() } + case "stop": + capture.queue.async { capture.finish() } + case "quit": + capture.shutdown() + exit(0) + default: + event("warning", ["message": "unknown command \(name)"]) + } + } + // Splice went away: finish what is being written. + capture.shutdown() + exit(0) +} + +dispatchMain() diff --git a/crates/splice-app/src/capture.rs b/crates/splice-app/src/capture.rs new file mode 100644 index 0000000..3a36c70 --- /dev/null +++ b/crates/splice-app/src/capture.rs @@ -0,0 +1,532 @@ +//! Recording through splice-capture, the helper built from `capture/` and +//! carried inside Splice: it keeps the chosen sources live (their pictures +//! shown small, their sound as levels), and on request writes each source +//! to its own file, all starting at the same moment. + +use crate::ffmpeg; +use crate::recorder::{self, Click, Source}; +use crate::session::dirs; +use anyhow::{Context as _, Result, bail}; +use gpui::RenderImage; +use serde::Deserialize; +use serde_json::{Value, json}; +use splice_core::studio::{Codec, Screen, Studio, Take}; +use std::collections::HashMap; +use std::io::{Read, Write}; +use std::path::{Path, PathBuf}; +use std::process::{Child, ChildStdin, Command, Stdio}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::mpsc::{Receiver, channel}; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; + +static HELPER: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/splice-capture")); + +/// The helper, written out beside Splice's other caches the first time +/// this build of it is needed. +pub fn helper() -> Result { + if HELPER.is_empty() { + bail!("recording needs macOS"); + } + let hash = blake3::hash(HELPER).to_hex(); + let dir = dirs().cache.join("bin"); + let path = dir.join(format!("splice-capture-{}", &hash[..12])); + if !path.exists() { + std::fs::create_dir_all(&dir)?; + let partial = dir.join(format!("splice-capture.{}.part", std::process::id())); + std::fs::write(&partial, HELPER)?; + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(&partial, std::fs::Permissions::from_mode(0o755))?; + std::fs::rename(&partial, &path)?; + } + Ok(path) +} + +#[derive(Clone, Debug, Default, Deserialize, serde::Serialize)] +pub struct Device { + pub id: String, + pub name: String, + #[serde(default)] + pub default: bool, +} + +/// The cameras and microphones there are, and whether Splice may use them. +#[derive(Clone, Debug, Default, Deserialize, serde::Serialize)] +pub struct Devices { + pub cameras: Vec, + pub microphones: Vec, + pub camera_access: String, + pub microphone_access: String, + pub screen_access: bool, +} + +/// Blocking: a fraction of a second. +pub fn devices() -> Result { + let output = Command::new(helper()?) + .arg("devices") + .output() + .context("running splice-capture")?; + if !output.status.success() { + bail!("{}", String::from_utf8_lossy(&output.stderr).trim()); + } + Ok(serde_json::from_slice(&output.stdout)?) +} + +/// A device chosen before, found again: by its id, else by its name (ids of +/// phones and USB devices can change). +pub fn find<'a>(chosen: &splice_core::studio::Device, there: &'a [Device]) -> Option<&'a Device> { + there + .iter() + .find(|d| d.id == chosen.id) + .or_else(|| there.iter().find(|d| d.name == chosen.name)) +} + +/// What of the screen `studio` records, as the click watcher sees it. +pub fn source(screen: &Screen) -> Result { + Ok(match screen { + Screen::Display { number } => Source::Display { number: *number }, + Screen::Window { app, title } => { + let windows = recorder::windows(); + let found = windows + .iter() + .find(|w| &w.app == app && &w.title == title) + .or_else(|| windows.iter().find(|w| &w.app == app)); + match found { + Some(w) => Source::Window { id: w.id }, + None => bail!("{app} has no window open to record"), + } + } + Screen::Region { + display, + x, + y, + width, + height, + } => { + let origin = recorder::displays() + .into_iter() + .find(|d| d.number == *display) + .map_or((0.0, 0.0), |d| (d.x, d.y)); + Source::Region { + x: (origin.0 + x) as i32, + y: (origin.1 + y) as i32, + width: *width as u32, + height: *height as u32, + } + } + }) +} + +/// The helper's instructions for `studio`, with the devices there are. +pub fn config(studio: &Studio, devices: &Devices, preview: u32) -> Result { + let screen = match &studio.screen { + None => Value::Null, + Some(screen) => match source(screen)? { + Source::Window { id } => json!({ "window": id }), + Source::Display { number } => json!({ "display": display_id(number)? }), + Source::Region { .. } => { + let Screen::Region { + display, + x, + y, + width, + height, + } = screen + else { + unreachable!() + }; + json!({ "display": display_id(*display)?, "region": [x, y, width, height] }) + } + }, + }; + let device = |chosen: &Option, there: &[Device], what: &str| -> Result { + match chosen { + None => Ok(Value::Null), + Some(c) => match find(c, there) { + Some(d) => Ok(json!(d.id)), + None => bail!("the {what} “{}” is not connected", c.name), + }, + } + }; + let out = studio.output; + Ok(json!({ + "screen": screen, + "system_audio": studio.system_audio, + "camera": device(&studio.camera, &devices.cameras, "camera")?, + "microphone": device(&studio.microphone, &devices.microphones, "microphone")?, + "codec": match out.codec { Codec::H264 => "h264", Codec::Hevc => "hevc" }, + "quality": out.quality.value(), + "fps": out.fps.clamp(1, 120), + "max_height": out.height, + "cursor": out.cursor, + "exclude": std::process::id(), + "preview": preview, + })) +} + +fn display_id(number: u32) -> Result { + recorder::displays() + .into_iter() + .find(|d| d.number == number) + .map(|d| d.id) + .with_context(|| format!("there is no display {number}")) +} + +/// The sources, as the helper numbers them. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub enum Feed { + Screen, + Camera, +} + +enum Message { + Event(Value), + Frame(Feed, Arc), + Gone, +} + +/// A recording under way. +pub struct Rec { + pub dir: PathBuf, + /// The files' first moment, in seconds since 1970. + pub zero: Option, + pub paused: Option, + /// Pauses so far, by the clock of `zero`: from, to. + pub pauses: Vec<(f64, Option)>, + /// How long has been recorded, as the helper last said. + pub seconds: f64, + clicks: Option<(Arc, std::thread::JoinHandle>)>, +} + +/// The helper, running: the sources live. +pub struct Live { + child: Option, + stdin: ChildStdin, + messages: Receiver, + /// The instructions it was started with. + pub config: Value, + pub frames: HashMap>, + /// The loudest each sound has been lately, 0 to 1. + pub levels: HashMap, + /// Sources that have sent something. + pub live: bool, + pub recording: Option, + pub problem: Option, + /// What macOS is asking the person to allow, while it waits. + pub asking: Option, + /// What to watch for clicks while recording. + area: Option, +} + +/// Something the editor should act on. +pub enum Happened { + Recording, + Paused, + Resumed, + /// A take is finished; its clicks are noted beside the screen file. + Stopped(Result), + Warning(String), + Failed(String), +} + +impl Live { + pub fn start(config: Value, area: Option) -> Result { + let mut child = Command::new(helper()?) + .arg("run") + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .spawn() + .context("starting splice-capture")?; + let mut stdin = child.stdin.take().expect("piped"); + writeln!(stdin, "{config}")?; + let mut stdout = child.stdout.take().expect("piped"); + let (tx, messages) = channel(); + std::thread::Builder::new() + .name("splice-capture".into()) + .spawn(move || { + let mut head = [0u8; 5]; + while stdout.read_exact(&mut head).is_ok() { + let len = u32::from_le_bytes([head[1], head[2], head[3], head[4]]) as usize; + let mut body = vec![0u8; len]; + if stdout.read_exact(&mut body).is_err() { + break; + } + let message = match head[0] { + b'J' => match serde_json::from_slice(&body) { + Ok(v) => Message::Event(v), + Err(_) => continue, + }, + b'F' if len >= 9 => { + let feed = if body[0] == 0 { Feed::Screen } else { Feed::Camera }; + let w = u32::from_le_bytes([body[1], body[2], body[3], body[4]]); + let h = u32::from_le_bytes([body[5], body[6], body[7], body[8]]); + if body.len() != 9 + (w * h * 4) as usize { + continue; + } + body.drain(..9); + match ffmpeg::render_image(w, h, body) { + Some(image) => Message::Frame(feed, image), + None => continue, + } + } + _ => continue, + }; + if tx.send(message).is_err() { + return; + } + } + let _ = tx.send(Message::Gone); + })?; + Ok(Live { + child: Some(child), + stdin, + messages, + config, + frames: HashMap::new(), + levels: HashMap::new(), + live: false, + recording: None, + problem: None, + asking: None, + area, + }) + } + + fn command(&mut self, command: Value) { + let _ = writeln!(self.stdin, "{command}"); + let _ = self.stdin.flush(); + } + + /// Starts writing the files into `dir`. + pub fn record(&mut self, dir: PathBuf) -> Result<()> { + if self.recording.is_some() { + bail!("already recording"); + } + std::fs::create_dir_all(&dir)?; + self.command(json!({ "cmd": "record", "dir": dir })); + let clicks = self.area.as_ref().and_then(recorder::watch_clicks); + self.recording = Some(Rec { + dir, + zero: None, + paused: None, + pauses: Vec::new(), + seconds: 0.0, + clicks, + }); + Ok(()) + } + + pub fn pause(&mut self) { + self.command(json!({ "cmd": "pause" })); + } + + pub fn resume(&mut self) { + self.command(json!({ "cmd": "resume" })); + } + + pub fn stop(&mut self) { + self.command(json!({ "cmd": "stop" })); + } + + /// How long has been recorded, not counting pauses. + pub fn seconds(&self) -> f64 { + let Some(rec) = &self.recording else { return 0.0 }; + match (rec.zero, rec.paused) { + (Some(zero), None) => { + let paused: f64 = rec.pauses.iter().map(|(a, b)| b.unwrap_or(*a) - a).sum(); + (now() - zero - paused).max(rec.seconds) + } + _ => rec.seconds, + } + } + + /// Takes in what the helper has said, and whether it said anything. + /// `retire` gets frames no longer shown. + pub fn poll(&mut self, retire: &mut Vec>) -> (Vec, bool) { + let mut out = Vec::new(); + let mut heard = false; + while let Ok(message) = self.messages.try_recv() { + heard = true; + match message { + Message::Frame(feed, image) => { + self.live = true; + if let Some(old) = self.frames.insert(feed, image) { + retire.push(old); + } + } + Message::Gone => { + if self.problem.is_none() { + self.problem = Some("the recorder stopped".into()); + } + out.push(Happened::Failed(self.problem.clone().unwrap_or_default())); + let seconds = self.seconds(); + if let Some(rec) = self.recording.take() { + // Its files are written in pieces as it goes: what + // was written before it stopped still plays. + let file = + |name: &str| Some(rec.dir.join(name)).filter(|p| p.metadata().is_ok_and(|m| m.len() > 0)); + let take = Take { + screen: file("screen.mp4"), + camera: file("camera.mp4"), + microphone: file("microphone.m4a"), + system: file("system.m4a"), + seconds, + }; + out.push(Happened::Stopped(if take.files().is_empty() { + Err("the recorder stopped before anything was written".into()) + } else { + Ok(take) + })); + } + } + Message::Event(e) => self.event(e, &mut out), + } + } + (out, heard) + } + + fn event(&mut self, e: Value, out: &mut Vec) { + let wall = e["wall"].as_f64(); + match e["event"].as_str().unwrap_or("") { + "levels" => { + self.live = true; + for feed in ["microphone", "system"] { + match e[feed].as_f64() { + Some(v) => self.levels.insert(feed.into(), v as f32), + None => self.levels.remove(feed), + }; + } + if let (Some(rec), Some(s)) = (&mut self.recording, e["seconds"].as_f64()) { + rec.seconds = s; + } + } + "live" => self.live = true, + // Every source has started: nothing is waiting on permission. + "started" => self.asking = None, + "asking" => self.asking = e["source"].as_str().map(str::to_string), + "recording" => { + if let Some(rec) = &mut self.recording { + rec.zero = wall; + } + out.push(Happened::Recording); + } + "paused" => { + if let Some(rec) = &mut self.recording { + rec.paused = Some(Instant::now()); + rec.seconds = e["seconds"].as_f64().unwrap_or(rec.seconds); + rec.pauses.push((wall.unwrap_or_else(now), None)); + } + out.push(Happened::Paused); + } + "resumed" => { + if let Some(rec) = &mut self.recording { + rec.paused = None; + if let Some(last) = rec.pauses.last_mut() { + last.1 = Some(wall.unwrap_or_else(now)); + } + } + out.push(Happened::Resumed); + } + "stopped" => { + let Some(mut rec) = self.recording.take() else { return }; + let clicks = rec.clicks.take().and_then(|(stop, watcher)| { + stop.store(true, Ordering::Relaxed); + watcher.join().ok() + }); + out.push(Happened::Stopped(finished(&e, &rec, clicks))); + } + "warning" => out.push(Happened::Warning(e["message"].as_str().unwrap_or("").to_string())), + "error" => { + let message = plain(e["message"].as_str().unwrap_or("the recorder failed")); + self.problem = Some(message.clone()); + out.push(Happened::Failed(message)); + } + _ => {} + } + } + + pub fn running(&mut self) -> bool { + self.child.as_mut().is_some_and(|c| matches!(c.try_wait(), Ok(None))) + } +} + +impl Drop for Live { + fn drop(&mut self) { + // Closing its input finishes any file and ends it. + let _ = writeln!(self.stdin, "{}", json!({ "cmd": "quit" })); + let _ = self.stdin.flush(); + if let Some(mut child) = self.child.take() { + std::thread::spawn(move || { + let _ = child.wait(); + }); + } + } +} + +/// The helper's errors, said so a person knows what to do. +fn plain(message: &str) -> String { + if message.contains("declined TCC") { + return "macOS stopped Splice recording the screen. Answer its prompt, or turn Splice (or the terminal it \ + was started from) on in System Settings → Privacy & Security → Screen Recording, then try again." + .into(); + } + message.to_string() +} + +fn now() -> f64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0.0, |d| d.as_secs_f64()) +} + +/// The take from the helper's "stopped", its clicks written beside the +/// screen file in the take's own time. +fn finished(e: &Value, rec: &Rec, clicks: Option>) -> Result { + if let Some(failed) = e["failed"].as_array().filter(|f| !f.is_empty()) { + let said: Vec<&str> = failed.iter().filter_map(Value::as_str).collect(); + return Err(format!("some files could not be finished: {}", said.join("; "))); + } + let files = &e["files"]; + let file = |name: &str| files[name].as_str().map(PathBuf::from).filter(|p| p.exists()); + let take = Take { + screen: file("screen"), + camera: file("camera"), + microphone: file("microphone"), + system: file("system"), + seconds: e["seconds"].as_f64().unwrap_or(0.0), + }; + if take.files().is_empty() { + return Err("nothing was recorded; check that the sources are live".into()); + } + if let (Some(screen), Some(zero), Some(clicks)) = (&take.screen, rec.zero, clicks) { + let pauses: Vec<(f64, f64)> = rec.pauses.iter().map(|(a, b)| (*a, b.unwrap_or(f64::MAX))).collect(); + let clicks = recorder::in_take(clicks, zero, &pauses, take.seconds); + if !clicks.is_empty() { + let _ = std::fs::write( + recorder::clicks_file(screen), + serde_json::to_string(&clicks).unwrap_or_default(), + ); + } + } + let _ = std::fs::write( + rec.dir.join("take.json"), + serde_json::to_string_pretty(&take).unwrap_or_default(), + ); + Ok(take) +} + +/// A new folder for a take of `project`: `Recordings//Take `. +pub fn take_dir(project: &Path) -> PathBuf { + let name = project + .file_stem() + .map(|s| s.to_string_lossy().into_owned()) + .unwrap_or_else(|| "Untitled".into()); + let mut dir = recorder::takes().join(name).join(format!("Take {}", recorder::stamp())); + let mut n = 2; + while dir.exists() { + dir.set_file_name(format!("Take {} {n}", recorder::stamp())); + n += 1; + } + dir +} diff --git a/crates/splice-app/src/ctl.rs b/crates/splice-app/src/ctl.rs index fde8701..1c3f5ca 100644 --- a/crates/splice-app/src/ctl.rs +++ b/crates/splice-app/src/ctl.rs @@ -184,7 +184,7 @@ Story (planning: beats end to end along the timeline, and cues) story take fail --take T a generation that failed: its pending take goes story take choose --take T | story take remove --take T cue add [--at T] --label TEXT | cue set --cue C [--at T] [--label TEXT] | cue remove --cue C - view edit|story switch the window between Edit view and the Story board + view edit|story|record switch the window between Edit, the Story board and Record The bible (who and what the movie keeps coming back to, and its look) bible read entries: kind, name, words, pictures, the beats each is in @@ -193,13 +193,23 @@ The bible (who and what the movie keeps coming back to, and its look) bible picture --entry E --image FILE | --remove I (the first picture is the face briefs lead with) bible remove --entry E E is an entry's id or name -Recording the screen - record sources the displays and windows that can be recorded - record start [--display N | --window ID|APP | --region X,Y,W,H] [--mic] [--clicks] [--for S] - [--beat B | --at T [--track N]] [--hide] - start; the take goes into beat B, onto the timeline at T, or the media +Recording (each source to its own file; the take is laid out as the scene says) + record sources displays, windows, cameras, microphones, and permissions + record setup [SETUP] [--default] + show or change what this project records; --default makes it the + setup new projects start with. SETUP is any of: + --screen display:N|window:APP|none --region X,Y,W,H (points, on that display) + --camera NAME|none --microphone NAME|default|none --system | --no-system (the Mac's sound) + --layout screen|bubble|side|camera --corner tl|tr|bl|br --size 0.3 (bubble height) + --shape circle|square --split 0.32 (camera's share, side by side) + --codec h264|hevc --quality good|high|best --fps 30|60 --height full|4k|1440|1080 + --cursor | --no-cursor --clicks | --no-clicks (note clicks, to zoom in on later) + record start [SETUP] [--for S] [--beat B | --at T [--track N] | --into media] + start (answers once recording); the take goes after everything on + the timeline, into beat B, at T, or only into the media + record pause | resume record stop [--wait] stop; with --wait, answer once the take is in the project - record status whether it is recording, and for how long + record status recording or not, for how long, sound levels, the last take's folder Operating the window (as a person would: for checking the interface) ui window size and scale, mouse, stage, timeline and picture bounds @@ -515,9 +525,22 @@ fn build(args: &[String]) -> Result { let v: f64 = next(arg)?.parse().map_err(|_| "--for expects seconds".to_string())?; request.insert("for".into(), json!(v)); } - "--mic" | "--clicks" => { + "--mic" | "--clicks" | "--system" | "--cursor" | "--default" => { request.insert(arg[2..].into(), json!(true)); } + "--no-clicks" | "--no-system" | "--no-cursor" => { + request.insert(arg[5..].into(), json!(false)); + } + "--screen" | "--camera" | "--microphone" | "--layout" | "--corner" | "--shape" | "--codec" + | "--quality" | "--height" => { + request.insert(arg[2..].into(), json!(next(arg)?)); + } + "--split" => { + let v: f64 = next(arg)? + .parse() + .map_err(|_| "--split expects a fraction".to_string())?; + request.insert("split".into(), json!(v)); + } "--every" => { let v: f64 = next(arg)?.parse().map_err(|_| "--every expects seconds".to_string())?; request.insert("every".into(), json!(v)); @@ -681,7 +704,7 @@ fn build(args: &[String]) -> Result { command.as_str() } "view" => { - let which = words.first().ok_or("view expects edit or story")?; + let which = words.first().ok_or("view expects edit, story or record")?; request.insert("view".into(), json!(which)); "view" } @@ -822,6 +845,15 @@ mod tests { ); assert!(build(&args("move --clip x")).is_err()); assert!(build(&args("explode")).is_err()); + let r = build(&args( + "record setup --screen window:Safari --camera C920 --no-system --layout bubble --size 0.25 --fps 60 --height 1440 --default", + )) + .unwrap(); + assert_eq!( + r, + json!({"cmd": "record", "sub": "setup", "screen": "window:Safari", "camera": "C920", "system": false, + "layout": "bubble", "size": 0.25, "fps": 60, "height": "1440", "default": true}) + ); } #[test] diff --git a/crates/splice-app/src/editor.rs b/crates/splice-app/src/editor.rs index 9368e69..77084e5 100644 --- a/crates/splice-app/src/editor.rs +++ b/crates/splice-app/src/editor.rs @@ -137,13 +137,6 @@ enum Overlay { index: usize, list: Vec, }, - /// What to record: a screen or a window. - Record { - choices: Vec, - index: usize, - microphone: bool, - clicks: bool, - }, /// Who and what from the bible is in a beat. Cast { beat: splice_core::story::BeatId, @@ -271,8 +264,8 @@ pub struct Editor { /// Words being typed into a beat or a cue. typing: Option, story_focus: FocusHandle, - /// A screen recording under way. - recording: Option, + /// Recording: the live sources, and a take under way. + rec: record::Recorder, /// The reference shot each beat's card shows, when one was clicked. shown_shots: HashMap, /// Takes to lay into a beat once they are imported. @@ -418,7 +411,7 @@ impl Editor { beat: None, typing: None, story_focus: cx.focus_handle(), - recording: None, + rec: record::Recorder::new(), fill_after_import: HashMap::new(), take_after_import: HashMap::new(), page: plan::Page::default(), @@ -429,11 +422,6 @@ impl Editor { _watch: watch, }; editor.prepare_media(cx); - // Coming back to Splice while it hides for a recording stops it. - cx.observe_window_activation(window, |this, window, cx| { - this.window_activated(window.is_window_active(), cx); - }) - .detach(); editor } @@ -1803,13 +1791,6 @@ impl Editor { } _ => {} }, - Overlay::Record { choices, index, .. } => match key { - "escape" => return self.close_overlay(window, cx), - "up" => *index = index.saturating_sub(1), - "down" => *index = (*index + 1).min(choices.len().saturating_sub(1)), - "enter" => return self.start_from_overlay(window, cx), - _ => {} - }, Overlay::Instruments { part, query, index, .. } => match key { "escape" => return self.close_overlay(window, cx), "up" => *index = index.saturating_sub(1), diff --git a/crates/splice-app/src/editor/record.rs b/crates/splice-app/src/editor/record.rs index 4235606..c65bf31 100644 --- a/crates/splice-app/src/editor/record.rs +++ b/crates/splice-app/src/editor/record.rs @@ -1,14 +1,19 @@ -//! Recording the screen from inside Splice: choose a display, a window or a -//! region, record (Splice steps aside when it would be in the shot), stop from -//! the title bar, by clicking Splice in the Dock, or through `splice ctl`, -//! and the take comes in as media, laid into the chosen beat if there is one. +//! The Record tab: what the project records (a screen or a window, a +//! camera, a microphone, the sound the Mac plays), seen live and laid out as +//! a scene, and how the files are made. A take writes each source to its own +//! file and comes onto the timeline as layers in the scene's places, where +//! they can still be moved. Splice keeps itself out of the shot. Stop from +//! the title bar, ⇧⌘R, or `splice ctl record stop`. -use super::render::icon; +use super::render::{GAP, caps, icon, panel, tip}; use super::*; -use crate::recorder::{self, Options, Recording, Source}; +use crate::capture::{self, Devices, Feed, Happened, Live}; +use crate::recorder; use gpui::FontWeight; use serde_json::{Value, json}; +use splice_core::look::Look; use splice_core::story::BeatId; +use splice_core::studio::{self, Bubble, Codec, Corner, Layout, Quality, Screen, Studio, Take}; use std::sync::mpsc::Sender; /// Where a take goes when it is done. @@ -18,448 +23,1251 @@ pub enum Into { Media, /// Laid into a beat. Beat(BeatId), - /// On the timeline at a time, on a layer. + /// On the timeline at a time, from a layer up. At(Time, usize), + /// After everything on the timeline. + End, } -pub struct Active { - pub recording: Recording, - pub into: Into, - /// Splice hid itself to stay out of the shot. - pub hid: bool, - /// `splice ctl record stop --wait` callers. - pub waiters: Vec>, +/// A list opened in the sources panel. +#[derive(Clone, Copy, Debug, PartialEq)] +pub enum Pick { + Screen, + Camera, + Microphone, } -/// A choice in the record panel. -#[derive(Clone, Debug)] -pub struct Choice { - pub label: String, - pub detail: String, - pub source: Source, +/// Recording: the helper while the sources are live, and what waits on it. +#[derive(Default)] +pub struct Recorder { + /// The setup the user made their default. + default: Option, + pub live: Option, + devices: Option, + listing: bool, + /// Where the take being recorded goes. + into: Option, + /// Asked to record before the sources were live: since when. + pending: Option, + /// `record start` callers waiting for the take to begin. + starters: Vec>, + /// `record stop --wait` callers. + waiters: Vec>, + pick: Option, + /// The last take's folder. + pub last: Option, + polling: bool, + problem: Option, + /// Where the stage is drawn, measured as it is painted. + bounds: Rc>>, } -pub fn choices() -> Vec { - let mut out: Vec = recorder::displays() - .into_iter() - .map(|d| Choice { - label: if d.number == 1 { - "Entire screen".to_string() - } else { - format!("Screen {}", d.number) - }, - detail: format!("{} · {:.0}×{:.0}", d.name, d.width, d.height), - source: Source::Display { number: d.number }, - }) - .collect(); - out.extend(recorder::windows().into_iter().map(|w| Choice { - label: w.app.clone(), - detail: if w.title.is_empty() { - format!("window · {:.0}×{:.0}", w.width, w.height) - } else { - format!("{} · {:.0}×{:.0}", w.title, w.width, w.height) - }, - source: Source::Window { id: w.id }, - })); - out +impl Recorder { + pub fn new() -> Recorder { + Recorder { + default: saved_default(), + ..Recorder::default() + } + } + + pub fn recording(&self) -> bool { + self.pending.is_some() || self.live.as_ref().is_some_and(|l| l.recording.is_some()) + } + + fn paused(&self) -> bool { + self.live + .as_ref() + .and_then(|l| l.recording.as_ref()) + .is_some_and(|r| r.paused.is_some()) + } + + fn seconds(&self) -> f64 { + self.live.as_ref().map_or(0.0, Live::seconds) + } +} + +fn defaults_file() -> PathBuf { + dirs().data.join("studio.json") +} + +/// The setup the user has made their default, for projects that have none. +fn saved_default() -> Option { + let text = std::fs::read_to_string(defaults_file()).ok()?; + serde_json::from_str(&text).ok() +} + +fn save_default(studio: &Studio) -> std::io::Result<()> { + std::fs::create_dir_all(&dirs().data)?; + std::fs::write( + defaults_file(), + serde_json::to_string_pretty(studio).unwrap_or_default(), + ) +} + +fn screen_name(screen: &Option) -> String { + match screen { + None => "None".into(), + Some(Screen::Display { number: 1 }) => "Main display".into(), + Some(Screen::Display { number }) => format!("Display {number}"), + Some(Screen::Window { app, title }) if title.is_empty() => format!("{app} window"), + Some(Screen::Window { app, title }) => format!("{app} — {title}"), + Some(Screen::Region { width, height, .. }) => format!("Part of the screen, {width:.0}×{height:.0}"), + } } impl Editor { - pub(super) fn open_record(&mut self, window: &mut Window, cx: &mut Context) { - if self.recording.is_some() { - return self.stop_recording(cx); - } - self.overlay = Overlay::Record { - choices: choices(), - index: 0, - microphone: false, - clicks: true, - }; - window.focus(&self.palette_focus); - cx.notify(); + // The setup. + + /// What this project records: its own setup, else the user's default, + /// else the main display and the Mac's microphone. + pub(super) fn studio(&self) -> Studio { + if let Some(s) = &self.doc.project.studio { + return s.clone(); + } + self.rec.default.clone().unwrap_or_else(|| Studio { + screen: Some(Screen::Display { number: 1 }), + microphone: self.rec.devices.as_ref().and_then(|d| { + d.microphones.iter().find(|m| m.default).map(|m| studio::Device { + id: m.id.clone(), + name: m.name.clone(), + }) + }), + ..Studio::default() + }) } - /// Where the next take goes: the chosen beat in Story view, else the media panel. - fn take_target(&self) -> Into { - match (self.mode, self.beat) { - (story::Mode::Story, Some(beat)) => Into::Beat(beat), - _ => Into::Media, + /// Changes the project's setup (one undo step), and the live sources to match. + pub(super) fn set_studio(&mut self, cx: &mut Context, f: impl FnOnce(&mut Studio)) { + let mut studio = self.studio(); + f(&mut studio); + self.edit(cx, |p| p.studio = Some(studio)); + self.relive(cx); + } + + fn list_devices(&mut self, cx: &mut Context) { + if self.rec.listing { + return; } + self.rec.listing = true; + cx.spawn(async move |this, cx| { + let found = cx.background_spawn(async move { capture::devices() }).await; + this.update(cx, |editor, cx| { + editor.rec.listing = false; + match found { + Ok(devices) => { + editor.rec.devices = Some(devices); + editor.relive(cx); + } + Err(e) => editor.rec.problem = Some(format!("Could not list cameras and microphones: {e}")), + } + cx.notify(); + }) + .ok(); + }) + .detach(); } - pub(super) fn start_from_overlay(&mut self, window: &mut Window, cx: &mut Context) { - let Overlay::Record { - choices, - index, - microphone, - clicks, - } = &self.overlay - else { + /// Keeps the sources live while the Record tab is open or a take is + /// under way, matching the setup; lets them go otherwise. + pub(super) fn relive(&mut self, cx: &mut Context) { + let wanted = self.mode == story::Mode::Record || self.rec.recording(); + if self.rec.live.as_ref().is_some_and(|l| l.recording.is_some()) { + return; + } + if !wanted { + self.let_go(); return; + } + let Some(devices) = &self.rec.devices else { + return self.list_devices(cx); }; - let Some(choice) = choices.get(*index).cloned() else { + let studio = self.studio(); + if !studio.has_sources() { + self.let_go(); + self.rec.problem = None; return; + } + let config = match capture::config(&studio, devices, 480) { + Ok(config) => config, + Err(e) => { + self.let_go(); + self.rec.problem = Some(e.to_string()); + return; + } }; - let options = Options { - microphone: *microphone, - clicks: *clicks, - seconds: None, + if let Some(live) = &mut self.rec.live + && live.config == config + && live.running() + { + return; + } + self.let_go(); + let area = studio + .screen + .as_ref() + .filter(|_| studio.output.clicks) + .and_then(|s| capture::source(s).ok()); + match Live::start(config, area) { + Ok(live) => { + self.rec.live = Some(live); + self.rec.problem = None; + self.keep_polling(cx); + } + Err(e) => self.rec.problem = Some(e.to_string()), + } + cx.notify(); + } + + fn let_go(&mut self) { + if let Some(live) = self.rec.live.take() { + self.player.retire(live.frames.values().cloned().collect()); + } + } + + fn keep_polling(&mut self, cx: &mut Context) { + if self.rec.polling { + return; + } + self.rec.polling = true; + cx.spawn(async move |this, cx| { + loop { + cx.background_executor().timer(Duration::from_millis(40)).await; + let going = this.update(cx, |editor, cx| editor.poll_recorder(cx)).unwrap_or(false); + if !going { + break; + } + } + }) + .detach(); + } + + /// Takes in what the helper has said. False when there is no helper to hear. + fn poll_recorder(&mut self, cx: &mut Context) -> bool { + let Some(live) = &mut self.rec.live else { + self.rec.polling = false; + return false; }; - // A whole screen would show Splice: step aside. - let hide = matches!(choice.source, Source::Display { .. }); - let into = self.take_target(); - self.close_overlay(window, cx); - if let Err(e) = self.start_recording(choice.source, options, into, hide, cx) { + let mut retired = Vec::new(); + let (happened, heard) = live.poll(&mut retired); + let ready = live.live; + self.player.retire(retired); + if let Some(asked) = self.rec.pending { + if ready { + self.rec.pending = None; + self.begin_take(cx); + } else if asked.elapsed() > Duration::from_secs(10) { + self.rec.pending = None; + self.take_failed("the sources did not start; check the Record tab".into()); + } + } + for h in happened { + match h { + Happened::Recording => self.say("Recording"), + Happened::Paused => self.say("Paused"), + Happened::Resumed => self.say("Recording"), + Happened::Warning(w) => self.say(w), + Happened::Stopped(Ok(take)) => { + let into = self.rec.into.take().unwrap_or(Into::End); + let reply = std::mem::take(&mut self.rec.waiters).into_iter().next(); + self.import_take(take, into, reply, cx); + self.relive(cx); + } + Happened::Stopped(Err(e)) => { + self.rec.into = None; + self.take_failed(format!("Recording failed: {e}")); + } + Happened::Failed(e) => { + self.rec.problem = Some(e.clone()); + self.let_go(); + self.take_failed(e); + } + } + } + // A new picture or level to show, or the clock to tick. + if heard && (self.mode == story::Mode::Record || self.rec.recording()) { + cx.notify(); + } + true + } + + fn take_failed(&mut self, e: String) { + self.say_error(e.clone()); + for w in std::mem::take(&mut self.rec.starters) + .into_iter() + .chain(std::mem::take(&mut self.rec.waiters)) + { + let _ = w.send(crate::ctl::fail(e.clone()).to_string()); + } + } + + // Takes. + + /// ⇧⌘R, the title bar's button, Story's Record: stop a take; record + /// into the chosen beat in Story; record in the Record tab; else open it. + pub(super) fn open_record(&mut self, _window: &mut Window, cx: &mut Context) { + if self.rec.recording() { + return self.stop_take(cx); + } + let into = match (self.mode, self.beat) { + (story::Mode::Story, Some(beat)) => Into::Beat(beat), + (story::Mode::Record, _) => Into::End, + _ => return self.set_mode(story::Mode::Record, cx), + }; + if !self.studio().has_sources() { + return self.set_mode(story::Mode::Record, cx); + } + if let Err(e) = self.start_take(into, None, cx) { self.say_error(e); } } - pub(super) fn start_recording( + /// Records a take with the project's setup. `reply` hears once it has begun. + pub(super) fn start_take( &mut self, - source: Source, - options: Options, into: Into, - hide: bool, + reply: Option>, cx: &mut Context, ) -> Result<(), String> { - if self.recording.is_some() { + if self.rec.recording() { return Err("already recording; stop it first".into()); } - if hide { - cx.hide(); + if !self.studio().has_sources() { + return Err("nothing to record: choose a screen, camera or microphone in the Record tab".into()); } - let recording = recorder::start(source, &options).map_err(|e| e.to_string())?; - self.recording = Some(Active { - recording, - into, - hid: hide, - waiters: Vec::new(), - }); - self.say(if hide { - "Recording. Click Splice in the Dock to stop." - } else { - "Recording" - }); - // Tick the clock, notice a time limit, and stop when Splice comes back. - cx.spawn(async move |this, cx| { - loop { - cx.background_executor().timer(Duration::from_millis(250)).await; - let going = this - .update(cx, |editor, cx| { - let Some(active) = &mut editor.recording else { - return false; - }; - if active.recording.finished() { - editor.stop_recording(cx); - return false; - } - cx.notify(); - true - }) - .unwrap_or(false); - if !going { - break; + self.rec.into = Some(into); + self.rec.pending = Some(Instant::now()); + self.rec.starters.extend(reply); + self.say("Starting the sources…"); + self.relive(cx); + if self.rec.live.is_none() && self.rec.devices.is_some() { + self.rec.pending = None; + let e = self + .rec + .problem + .clone() + .unwrap_or_else(|| "the sources could not start".into()); + self.take_failed(e.clone()); + return Err(e); + } + self.keep_polling(cx); + cx.notify(); + Ok(()) + } + + fn begin_take(&mut self, cx: &mut Context) { + let dir = capture::take_dir(self.doc.path()); + let Some(live) = &mut self.rec.live else { return }; + match live.record(dir.clone()) { + Ok(()) => { + for s in std::mem::take(&mut self.rec.starters) { + let _ = s.send(json!({ "ok": true, "recording": true, "dir": dir }).to_string()); } } + Err(e) => self.take_failed(e.to_string()), + } + cx.notify(); + } + + /// Stops the take; it comes in when its files are finished. + pub(super) fn stop_take(&mut self, cx: &mut Context) { + if self.rec.pending.take().is_some() { + self.rec.into = None; + self.take_failed("stopped before the sources were live".into()); + return; + } + if let Some(live) = &mut self.rec.live + && live.recording.is_some() + { + live.stop(); + let seconds = live.seconds(); + self.say(format!("Finishing the take ({seconds:.0}s)…")); + } + cx.notify(); + } + + fn pause_take(&mut self, cx: &mut Context) { + let paused = self.rec.paused(); + if let Some(live) = &mut self.rec.live + && live.recording.is_some() + { + if paused { live.resume() } else { live.pause() } + } + cx.notify(); + } + + fn import_take(&mut self, take: Take, into: Into, reply: Option>, cx: &mut Context) { + self.working += 1; + self.rec.last = take + .files() + .first() + .and_then(|(_, p)| p.parent()) + .map(Path::to_path_buf); + let files: Vec<(&'static str, PathBuf)> = take.files().into_iter().map(|(n, p)| (n, p.to_path_buf())).collect(); + cx.spawn(async move |this, cx| { + let probed = cx + .background_spawn(async move { + files + .into_iter() + .map(|(name, path)| { + let probe = ffmpeg::probe(&path)?; + let hash = hash_file(&path)?; + Ok((name, path, probe, hash)) + }) + .collect::>>() + }) + .await; + this.update(cx, |editor, cx| editor.add_take(probed, take, into, reply, cx)) + .ok(); }) .detach(); cx.notify(); - Ok(()) } - /// Stops recording; the take comes in when the file is finished. - pub(super) fn stop_recording(&mut self, cx: &mut Context) { - let Some(active) = self.recording.take() else { return }; - let Active { - recording, - into, - hid, - waiters, - } = active; - if hid { - cx.activate(true); - } - let seconds = recording.seconds(); - self.say(format!("Finishing the recording ({seconds:.0}s)…")); - cx.spawn(async move |this, cx| { - let result = cx.background_spawn(async move { recording.stop() }).await; - this.update(cx, |editor, cx| { + #[allow(clippy::type_complexity)] + fn add_take( + &mut self, + probed: Vec>, + take: Take, + into: Into, + reply: Option>, + cx: &mut Context, + ) { + self.working -= 1; + let studio = self.studio(); + let mut errors = Vec::new(); + let mut added: Vec<(Media, PathBuf)> = Vec::new(); + let mut placed: Vec = Vec::new(); + let first = self.doc.project.is_empty(); + self.doc.edit(|project| { + let mut ids: HashMap<&str, MediaId> = HashMap::new(); + for result in probed { match result { - Ok(path) => { - let place = match into { - Into::At(t, track) => Some((t, track)), - _ => None, - }; - if let Into::Beat(beat) = into { - editor.fill_after_import.insert(path.clone(), beat); + Ok((name, path, probe, hash)) => { + let id = project.add_media(Media { + id: MediaId(0), + name: format!("{} {}", name_of(name), take_name(&path)), + hash, + ext: path + .extension() + .map(|e| e.to_string_lossy().to_lowercase()) + .unwrap_or_default(), + kind: probe.kind, + duration: probe.duration, + width: probe.width, + height: probe.height, + has_audio: probe.has_audio, + source: path.clone(), + recipe: None, + content: None, + clicks: recorder::read_clicks(&path), + }); + ids.insert(name, id); + added.push((project.media(id).cloned().expect("just added"), path)); + } + Err(e) => errors.push(e.to_string()), + } + } + let size = |p: &Project, name: &str| { + ids.get(name) + .and_then(|id| p.media(*id)) + .filter(|m| m.width > 0) + .map(|m| (m.width, m.height)) + }; + match into { + Into::Media => {} + Into::Beat(beat) => { + let main = if studio.scene.layout == Layout::Camera { + ["camera", "screen"] + } else { + ["screen", "camera"] + }; + if let Some(id) = main.iter().find_map(|n| ids.get(n)) + && let Some(clip) = project.fill_beat(beat, *id) + { + placed.push(clip); + } + } + Into::At(..) | Into::End => { + let (t, track) = match into { + Into::At(t, track) => (t, track), + _ => (project.duration(), 0), + }; + let canvas = (project.settings.width, project.settings.height); + let shown = studio.looks(canvas, size(project, "screen"), size(project, "camera")); + let mut clips: HashMap<&str, ClipId> = HashMap::new(); + for (name, wanted) in [ + ("screen", shown.screen.is_some()), + ("camera", shown.camera.is_some()), + ("microphone", true), + ("system", true), + ] { + if wanted + && let Some(id) = ids.get(name) + && let Some(clip) = project.insert_clip(*id, track, t) + { + clips.insert(name, clip); + placed.push(clip); } - // Answer `--wait` callers when the take is in. - let reply = waiters.into_iter().next(); - editor.import_paths(vec![path], place, reply, cx); } - Err(e) => { - editor.say_error(format!("Recording failed: {e}")); - for w in waiters { - let _ = w.send(crate::ctl::fail(e.to_string()).to_string()); + // The first pictures in a project set its shape: place them by it. + let canvas = (project.settings.width, project.settings.height); + let shown = studio.looks(canvas, size(project, "screen"), size(project, "camera")); + for (name, look) in [("screen", shown.screen), ("camera", shown.camera)] { + if let (Some(clip), Some(look)) = (clips.get(name), look) + && look != Look::default() + { + project.set_look(*clip, |l| *l = look); } } } - cx.notify(); - }) - .ok(); - }) - .detach(); + } + }); + for (media, path) in added.iter().cloned() { + if self.library.get(&media.hash).is_some_and(|a| a.path.is_some()) { + continue; + } + self.library.entry(&media.hash).path = Some(path.clone()); + self.embed(&media, path.clone(), cx); + self.build_assets(media, path, cx); + } + if let Some(reply) = reply { + let media: Vec<_> = added + .iter() + .map(|(m, _)| json!({ "id": m.id.0, "name": m.name })) + .collect(); + let _ = reply.send( + json!({ + "ok": errors.is_empty(), + "take": self.rec.last, + "seconds": take.seconds, + "media": media, + "clips": placed.iter().map(|c| c.0).collect::>(), + "errors": errors, + "revision": self.doc.revision, + }) + .to_string(), + ); + } + if !placed.is_empty() { + self.selection = placed; + self.after_edit(); + } + match errors.first() { + Some(e) => self.say_error(e.clone()), + None => self.say(format!( + "Recorded {:.0}s: {}", + take.seconds, + take.files().iter().map(|(n, _)| *n).collect::>().join(", ") + )), + } + if first && !self.doc.project.is_empty() { + self.zoom_fit(); + } cx.notify(); } - /// Splice came back to the front while it was hiding for a recording: - /// that is the way to stop. - pub(super) fn window_activated(&mut self, active: bool, cx: &mut Context) { - if active - && let Some(rec) = &self.recording - && rec.hid - && rec.recording.seconds() > 1.0 - { - self.stop_recording(cx); - } - } + // Drawing. /// The title bar's record control: a button, or the running clock. pub(super) fn record_button(&mut self, cx: &mut Context) -> AnyElement { - match &self.recording { - Some(active) => { - let s = active.recording.seconds() as u64; - div() - .id("recording") - .h(px(26.0)) - .px(px(10.0)) - .rounded(px(7.0)) - .flex() - .items_center() - .gap(px(7.0)) - .cursor_pointer() - .bg(gpui::rgba(0xef6a5533)) - .border_1() - .border_color(Theme::accent()) - .child(div().size(px(8.0)).rounded_full().bg(Theme::accent())) - .child( - div() - .font_family(MONO_FONT) - .text_size(px(11.5)) - .text_color(Theme::text()) - .child(format!("{}:{:02}", s / 60, s % 60)), - ) - .child(div().text_size(px(11.5)).text_color(Theme::text()).child("Stop")) - .tooltip(render::tip("Stop recording", Some("⇧⌘R"))) - .on_click(cx.listener(|this, _, _, cx| this.stop_recording(cx))) - .into_any_element() - } - None => render::icon_button("record", "record", true) - .tooltip(render::tip("Record the screen", Some("⇧⌘R"))) + if !self.rec.recording() { + return render::icon_button("record", "record", true) + .tooltip(tip("Record", Some("⇧⌘R"))) .on_click(cx.listener(|this, _, window, cx| this.open_record(window, cx))) - .into_any_element(), + .into_any_element(); + } + let s = self.rec.seconds() as u64; + let paused = self.rec.paused(); + div() + .id("recording") + .h(px(26.0)) + .px(px(10.0)) + .rounded(px(7.0)) + .flex() + .items_center() + .gap(px(7.0)) + .cursor_pointer() + .bg(gpui::rgba(0xef6a5533)) + .border_1() + .border_color(Theme::accent()) + .child( + div() + .size(px(8.0)) + .rounded_full() + .bg(if paused { Theme::overlay() } else { Theme::accent() }), + ) + .child( + div() + .font_family(MONO_FONT) + .text_size(px(11.5)) + .text_color(Theme::text()) + .child(format!("{}:{:02}", s / 60, s % 60)), + ) + .child(div().text_size(px(11.5)).text_color(Theme::text()).child("Stop")) + .tooltip(tip("Stop recording", Some("⇧⌘R"))) + .on_click(cx.listener(|this, _, _, cx| this.stop_take(cx))) + .into_any_element() + } + + /// The Record tab: the scene, live, and beside it the sources and how + /// the files are made. + pub(super) fn render_studio(&mut self, cx: &mut Context) -> AnyElement { + if self.rec.devices.is_none() { + self.list_devices(cx); } + let studio = self.studio(); + div() + .flex_1() + .min_w_0() + .flex() + .gap(px(GAP)) + .child( + panel() + .flex_1() + .min_w_0() + .flex() + .flex_col() + .child(self.render_scene(&studio, cx)) + .child(self.scene_bar(&studio, cx)), + ) + .child( + panel().w(px(292.0)).flex_none().flex().flex_col().child( + div() + .id("studio-side") + .flex_1() + .overflow_y_scroll() + .p(px(14.0)) + .flex() + .flex_col() + .gap(px(14.0)) + .child(self.sources_panel(&studio, cx)) + .child(self.output_panel(&studio, cx)), + ), + ) + .into_any_element() } - pub(super) fn record_overlay(&self, cx: &mut Context) -> AnyElement { - let Overlay::Record { - choices, - index, - microphone, - clicks, - } = &self.overlay - else { - return div().into_any_element(); + /// The canvas, with each live picture where the scene puts it. + fn render_scene(&mut self, studio: &Studio, cx: &mut Context) -> AnyElement { + let settings = self.doc.project.settings; + let cell = self.rec.bounds.clone(); + let measure = canvas(move |bounds, _, _| cell.set(bounds), |_, _, _, _| {}) + .absolute() + .size_full(); + let area = self.rec.bounds.get(); + let (bw, bh) = (f32::from(area.size.width) - 40.0, f32::from(area.size.height) - 32.0); + let aspect = settings.width as f32 / settings.height.max(1) as f32; + let (w, h) = if bw / bh.max(1.0) > aspect { + (bh * aspect, bh) + } else { + (bw, bw / aspect) }; - let (index, microphone, clicks) = (*index, *microphone, *clicks); - let into = match self.take_target() { - Into::Beat(beat) => { - let title = self - .doc - .project - .story - .beats - .iter() - .find(|b| b.id == beat) - .map(|b| b.title.clone()) - .unwrap_or_default(); - format!("The take goes into the beat “{title}”.") - } - _ => "The take goes into the media panel.".to_string(), + let (w, h) = (w.max(1.0), h.max(1.0)); + let (left, top) = ( + (f32::from(area.size.width) - w) / 2.0, + (f32::from(area.size.height) - h) / 2.0, + ); + let frames = self.rec.live.as_ref().map(|l| l.frames.clone()).unwrap_or_default(); + let dims = |f: Feed| { + frames.get(&f).map(|i| { + let s = i.size(0); + (s.width.0 as u32, s.height.0 as u32) + }) }; - let first = index.saturating_sub(3); - let rows: Vec = choices - .iter() - .enumerate() - .skip(first) - .take(7) - .map(|(i, c)| { - let active = i == index; - let glyph = if matches!(c.source, Source::Display { .. }) { - "film" - } else { - "sidebar-off" - }; + let canvas_px = (settings.width, settings.height); + let shown = studio.looks(canvas_px, dims(Feed::Screen), dims(Feed::Camera)); + let mut layers: Vec = Vec::new(); + for (feed, look) in [(Feed::Screen, shown.screen), (Feed::Camera, shown.camera)] { + let (Some(look), Some(image)) = (look, frames.get(&feed)) else { + continue; + }; + let Some(natural) = dims(feed) else { continue }; + layers.push(picture(image.clone(), natural, &look, (w, h))); + } + let failed = self.rec.problem.is_some(); + let status = if let Some(problem) = &self.rec.problem { + Some(problem.clone()) + } else if !studio.has_sources() { + Some("Choose what to record on the right.".into()) + } else if let Some(what) = self.rec.live.as_ref().and_then(|l| l.asking.clone()) { + Some(format!("Waiting for you to allow the {what}: macOS is asking.")) + } else if self.rec.live.as_ref().is_none_or(|l| !l.live) { + Some("Starting the sources…".into()) + } else { + None + }; + div() + .flex_1() + .min_h_0() + .relative() + .child(measure) + .child( div() - .id(("record-choice", i)) - .h(px(40.0)) - .px(px(12.0)) - .rounded(px(7.0)) - .flex() - .items_center() - .gap(px(10.0)) - .cursor_pointer() - .when(active, |d| d.bg(Theme::surface0())) - .hover(|d| d.bg(Theme::surface0())) - .child(icon(glyph, if active { Theme::text() } else { Theme::overlay() }).size(px(14.0))) - .child( - div() - .flex_1() - .min_w_0() - .flex() - .flex_col() - .child( - div() - .text_size(px(12.5)) - .text_color(if active { Theme::text() } else { Theme::subtext() }) - .truncate() - .child(c.label.clone()), - ) - .child( - div() - .text_size(px(10.5)) - .text_color(Theme::overlay()) - .truncate() - .child(c.detail.clone()), - ), - ) - .on_click(cx.listener(move |this, e: &ClickEvent, window, cx| { - if let Overlay::Record { index, .. } = &mut this.overlay { - *index = i; - } - if e.click_count() >= 2 { - this.start_from_overlay(window, cx); - } - cx.notify(); - })) - .into_any_element() - }) - .collect(); - let toggle = |id: &'static str, label: &'static str, on: bool| { + .absolute() + .left(px(left)) + .top(px(top)) + .w(px(w)) + .h(px(h)) + .overflow_hidden() + .bg(gpui::black()) + .children(layers) + .when_some(status, |d, s| { + d.child( + div() + .absolute() + .inset_0() + .bg(gpui::rgba(0x000000b3)) + .p(px(20.0)) + .flex() + .flex_col() + .items_center() + .justify_center() + .gap(px(10.0)) + .child( + div() + .w_full() + .text_center() + .text_size(px(12.5)) + .line_height(px(18.0)) + .text_color(Theme::subtext()) + .child(s), + ) + .when(failed, |d| { + d.child(chip("studio-retry", "Try again", false).on_click(cx.listener( + |this, _, _, cx| { + this.rec.problem = None; + this.let_go(); + this.relive(cx); + }, + ))) + }), + ) + }), + ) + .into_any_element() + } + + /// Layouts, and the record and pause buttons. + fn scene_bar(&mut self, studio: &Studio, cx: &mut Context) -> impl IntoElement { + let scene = studio.scene; + let recording = self.rec.recording(); + let layouts = Layout::ALL.map(|layout| { + chip(("layout", layout as usize), layout.name(), scene.layout == layout) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.scene.layout = layout))) + }); + let corners = Corner::ALL.map(|corner| { + let on = scene.corner == corner; + let (right, bottom) = ( + matches!(corner, Corner::TopRight | Corner::BottomRight), + matches!(corner, Corner::BottomLeft | Corner::BottomRight), + ); + // A little screen, its camera in that corner. div() - .id(id) - .h(px(26.0)) - .px(px(9.0)) - .rounded(px(6.0)) - .flex() - .items_center() - .gap(px(6.0)) - .cursor_pointer() + .id(("corner", corner as usize)) + .w(px(28.0)) + .h(px(18.0)) + .rounded(px(4.0)) .border_1() - .border_color(if on { Theme::surface2() } else { Theme::edge() }) + .border_color(if on { Theme::accent() } else { Theme::surface1() }) .when(on, |d| d.bg(Theme::surface0())) + .hover(|d| d.border_color(Theme::surface2())) + .cursor_pointer() + .relative() .child( div() - .size(px(7.0)) + .absolute() + .size(px(6.0)) .rounded_full() - .bg(if on { Theme::accent() } else { Theme::surface1() }), + .bg(if on { Theme::accent() } else { Theme::overlay() }) + .map(|d| if right { d.right(px(2.0)) } else { d.left(px(2.0)) }) + .map(|d| if bottom { d.bottom(px(2.0)) } else { d.top(px(2.0)) }), ) - .child(div().text_size(px(11.5)).text_color(Theme::text()).child(label)) - }; + .tooltip(tip("Put the camera in this corner", None)) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.scene.corner = corner))) + }); + let sizes = [("S", 0.22), ("M", 0.3), ("L", 0.4)].map(|(name, size)| { + chip( + ("bubble-size", (size * 100.0) as usize), + name, + (scene.size - size).abs() < 0.01, + ) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.scene.size = size))) + }); + let shapes = [(Bubble::Circle, "Round"), (Bubble::Square, "Square")].map(|(shape, name)| { + chip(("bubble-shape", shape as usize), name, scene.shape == shape) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.scene.shape = shape))) + }); + let record = div() + .id("studio-record") + .h(px(34.0)) + .px(px(16.0)) + .rounded(px(9.0)) + .flex() + .items_center() + .gap(px(8.0)) + .cursor_pointer() + .when(!recording, |d| d.bg(Theme::accent()).hover(|s| s.opacity(0.9))) + .when(recording, |d| { + d.border_1().border_color(Theme::accent()).bg(gpui::rgba(0xef6a5533)) + }) + .child( + div() + .size(px(10.0)) + .when(recording, |d| d.rounded(px(2.0)).bg(Theme::accent())) + .when(!recording, |d| d.rounded_full().bg(Theme::ground())), + ) + .child( + div() + .text_size(px(12.5)) + .font_weight(FontWeight::SEMIBOLD) + .text_color(if recording { Theme::text() } else { Theme::ground() }) + .child(if recording { + let s = self.rec.seconds() as u64; + format!("Stop {}:{:02}", s / 60, s % 60) + } else { + "Record".to_string() + }), + ) + .tooltip(tip( + if recording { + "Stop, and lay the take on the timeline" + } else { + "Record a take" + }, + Some("⇧⌘R"), + )) + .on_click(cx.listener(move |this, _, _, cx| { + if this.rec.recording() { + this.stop_take(cx); + } else if let Err(e) = this.start_take(Into::End, None, cx) { + this.say_error(e); + } + })); + let paused = self.rec.paused(); + let label = |text: &'static str| div().text_size(px(11.0)).text_color(Theme::overlay()).child(text); div() + .flex_none() + .px(px(14.0)) + .py(px(10.0)) + .border_t_1() + .border_color(Theme::rule()) .flex() .flex_col() + .gap(px(8.0)) .child( div() - .h(px(48.0)) - .px(px(16.0)) .flex() + .flex_wrap() .items_center() - .gap(px(10.0)) - .border_b_1() - .border_color(Theme::rule()) - .child(icon("record", Theme::accent())) - .child( - div() - .text_size(px(14.0)) - .font_weight(FontWeight::SEMIBOLD) - .text_color(Theme::text()) - .child("Record"), - ) - .child( - div() - .ml_auto() - .text_size(px(11.0)) - .text_color(Theme::overlay()) - .child("what to capture"), - ), + .gap(px(6.0)) + .children(layouts) + .child(div().flex_1()) + .when(recording && self.rec.pending.is_none(), |d| { + d.child( + chip("studio-pause", if paused { "Resume" } else { "Pause" }, paused) + .on_click(cx.listener(|this, _, _, cx| this.pause_take(cx))), + ) + }) + .child(record), ) - .child(div().id("record-rows").p(px(6.0)).flex().flex_col().children(rows)) - .child( + .when(scene.layout == Layout::Bubble, |d| { + d.child( + div() + .flex() + .flex_wrap() + .items_center() + .gap(px(6.0)) + .child(label("Corner")) + .children(corners) + .child(div().w(px(10.0))) + .child(label("Size")) + .children(sizes) + .child(div().w(px(10.0))) + .children(shapes), + ) + }) + } + + fn sources_panel(&mut self, studio: &Studio, cx: &mut Context) -> impl IntoElement { + let levels = self.rec.live.as_ref().map(|l| l.levels.clone()).unwrap_or_default(); + let devices = self.rec.devices.clone().unwrap_or_default(); + let open = self.rec.pick; + let row = |id: &'static str, + glyph: &'static str, + label: &'static str, + value: String, + pick: Pick, + cx: &mut Context| { + let on = open == Some(pick); + div() + .id(id) + .h(px(34.0)) + .px(px(8.0)) + .rounded(px(7.0)) + .flex() + .items_center() + .gap(px(9.0)) + .cursor_pointer() + .when(on, |d| d.bg(Theme::surface0())) + .hover(|d| d.bg(Theme::surface0())) + .child(icon(glyph, Theme::subtext()).size(px(14.0))) + .child( + div() + .w(px(64.0)) + .text_size(px(11.5)) + .text_color(Theme::overlay()) + .child(label), + ) + .child( + div() + .flex_1() + .min_w_0() + .truncate() + .text_size(px(12.0)) + .text_color(Theme::text()) + .child(value), + ) + .child(icon("chevron", Theme::overlay()).size(px(11.0))) + .on_click(cx.listener(move |this, _, _, cx| { + this.rec.pick = if this.rec.pick == Some(pick) { None } else { Some(pick) }; + // Something may have been plugged in since. + if this.rec.pick.is_some_and(|p| p != Pick::Screen) { + this.list_devices(cx); + } + cx.notify(); + })) + }; + let mut list: Vec = Vec::new(); + let screen_row = row( + "pick-screen", + "film", + "Screen", + screen_name(&studio.screen), + Pick::Screen, + cx, + ); + let camera_row = row( + "pick-camera", + "camera", + "Camera", + studio.camera.as_ref().map_or("None".into(), |d| d.name.clone()), + Pick::Camera, + cx, + ); + let mic_row = row( + "pick-mic", + "mic", + "Mic", + studio.microphone.as_ref().map_or("None".into(), |d| d.name.clone()), + Pick::Microphone, + cx, + ); + let options = |pick: Pick, cx: &mut Context| -> Vec { + let option = |id: (&'static str, usize), label: String, chosen: bool| { div() - .px(px(16.0)) - .py(px(10.0)) + .id(id) + .h(px(28.0)) + .pl(px(32.0)) + .pr(px(8.0)) + .rounded(px(6.0)) .flex() .items_center() - .gap(px(6.0)) - .border_t_1() - .border_color(Theme::rule()) - .child( - toggle("record-mic", "Microphone", microphone).on_click(cx.listener(|this, _, _, cx| { - if let Overlay::Record { microphone, .. } = &mut this.overlay { - *microphone = !*microphone; - } - cx.notify(); - })), - ) - .child( - toggle("record-clicks", "Show clicks", clicks).on_click(cx.listener(|this, _, _, cx| { - if let Overlay::Record { clicks, .. } = &mut this.overlay { - *clicks = !*clicks; - } - cx.notify(); - })), - ) - .child(div().flex_1()) + .gap(px(8.0)) + .cursor_pointer() + .hover(|d| d.bg(Theme::surface0())) .child( div() - .id("record-start") - .h(px(28.0)) - .px(px(12.0)) - .rounded(px(7.0)) - .flex() - .items_center() - .gap(px(6.0)) - .cursor_pointer() - .bg(Theme::accent()) - .hover(|s| s.opacity(0.9)) - .child(div().size(px(8.0)).rounded_full().bg(Theme::ground())) - .child( - div() - .text_size(px(12.0)) - .font_weight(FontWeight::SEMIBOLD) - .text_color(Theme::ground()) - .child("Start"), - ) - .on_click(cx.listener(|this, _, window, cx| this.start_from_overlay(window, cx))), - ), - ) + .flex_1() + .min_w_0() + .truncate() + .text_size(px(11.5)) + .text_color(if chosen { Theme::text() } else { Theme::subtext() }) + .child(label), + ) + .when(chosen, |d| d.child(icon("check", Theme::accent()).size(px(12.0)))) + }; + let mut out: Vec = Vec::new(); + match pick { + Pick::Screen => { + let mut choices: Vec<(String, Option)> = vec![("None".into(), None)]; + for d in recorder::displays() { + let name = if d.number == 1 { + "Main display".into() + } else { + format!("Display {}", d.number) + }; + choices.push(( + format!("{name} · {:.0}×{:.0}", d.width, d.height), + Some(Screen::Display { number: d.number }), + )); + } + for w in recorder::windows().into_iter().take(14) { + let label = if w.title.is_empty() { + w.app.clone() + } else { + format!("{} — {}", w.app, w.title) + }; + choices.push(( + label, + Some(Screen::Window { + app: w.app, + title: w.title, + }), + )); + } + for (i, (label, screen)) in choices.into_iter().enumerate() { + let chosen = studio.screen == screen; + out.push( + option(("screen-choice", i), label, chosen) + .on_click(cx.listener(move |this, _, _, cx| { + let screen = screen.clone(); + this.rec.pick = None; + this.set_studio(cx, |s| s.screen = screen); + })) + .into_any_element(), + ); + } + } + Pick::Camera | Pick::Microphone => { + let (there, current) = if pick == Pick::Camera { + (&devices.cameras, &studio.camera) + } else { + (&devices.microphones, &studio.microphone) + }; + let mut choices: Vec<(String, Option)> = vec![("None".into(), None)]; + for d in there { + choices.push(( + d.name.clone(), + Some(studio::Device { + id: d.id.clone(), + name: d.name.clone(), + }), + )); + } + for (i, (label, device)) in choices.into_iter().enumerate() { + let chosen = current.as_ref().map(|c| &c.id) == device.as_ref().map(|d| &d.id); + out.push( + option(("device-choice", i), label, chosen) + .on_click(cx.listener(move |this, _, _, cx| { + let device = device.clone(); + this.rec.pick = None; + this.set_studio(cx, |s| { + if pick == Pick::Camera { + s.camera = device; + } else { + s.microphone = device; + } + }); + })) + .into_any_element(), + ); + } + if there.is_empty() { + out.push( + div() + .pl(px(32.0)) + .text_size(px(11.0)) + .text_color(Theme::overlay()) + .child("None connected") + .into_any_element(), + ); + } + } + } + out + }; + list.push(screen_row.into_any_element()); + if open == Some(Pick::Screen) { + list.extend(options(Pick::Screen, cx)); + } + list.push(camera_row.into_any_element()); + if open == Some(Pick::Camera) { + list.extend(options(Pick::Camera, cx)); + } + list.push( + div() + .flex() + .flex_col() + .child(mic_row) + .when(studio.microphone.is_some(), |d| { + d.child(meter(levels.get("microphone").copied())) + }) + .into_any_element(), + ); + if open == Some(Pick::Microphone) { + list.extend(options(Pick::Microphone, cx)); + } + let system = studio.system_audio; + list.push( + div() + .flex() + .flex_col() + .child( + div() + .id("system-audio") + .h(px(34.0)) + .px(px(8.0)) + .rounded(px(7.0)) + .flex() + .items_center() + .gap(px(9.0)) + .cursor_pointer() + .hover(|d| d.bg(Theme::surface0())) + .child(icon("volume", Theme::subtext()).size(px(14.0))) + .child( + div() + .flex_1() + .text_size(px(12.0)) + .text_color(Theme::text()) + .child("Sound the Mac plays"), + ) + .child(switch(system)) + .on_click( + cx.listener(|this, _, _, cx| this.set_studio(cx, |s| s.system_audio = !s.system_audio)), + ), + ) + .when(system, |d| d.child(meter(levels.get("system").copied()))) + .into_any_element(), + ); + let access = self.rec.devices.as_ref().map(|d| { + let mut missing = Vec::new(); + if studio.screen.is_some() && !d.screen_access { + missing.push("Screen Recording"); + } + if studio.camera.is_some() && d.camera_access == "denied" { + missing.push("Camera"); + } + if studio.microphone.is_some() && d.microphone_access == "denied" { + missing.push("Microphone"); + } + missing + }); + div() + .flex() + .flex_col() + .gap(px(2.0)) + .child(caps("Sources").mb(px(6.0))) + .children(list) + .when_some(access.filter(|m| !m.is_empty()), |d, missing| { + d.child( + div() + .mt(px(8.0)) + .text_size(px(11.0)) + .line_height(px(15.0)) + .text_color(Theme::accent()) + .child(format!( + "Turn on {} for Splice in System Settings → Privacy & Security.", + missing.join(" and ") + )), + ) + }) + } + + fn output_panel(&mut self, studio: &Studio, cx: &mut Context) -> impl IntoElement { + let out = studio.output; + let line = |label: &'static str, chips: Vec| { + div() + .flex() + .flex_col() + .gap(px(5.0)) + .child(div().text_size(px(11.0)).text_color(Theme::overlay()).child(label)) + .child(div().flex().flex_wrap().gap(px(4.0)).children(chips)) + }; + let codecs = [(Codec::H264, "H.264"), (Codec::Hevc, "HEVC")] + .map(|(codec, name)| { + chip(("codec", codec as usize), name, out.codec == codec) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.output.codec = codec))) + .into_any_element() + }) + .into_iter() + .collect(); + let qualities = Quality::ALL + .map(|q| { + chip(("quality", q as usize), q.name(), out.quality == q) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.output.quality = q))) + .into_any_element() + }) + .into_iter() + .collect(); + let rates = [30u32, 60] + .map(|fps| { + chip(("fps", fps as usize), &format!("{fps} fps"), out.fps == fps) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.output.fps = fps))) + .into_any_element() + }) + .into_iter() + .collect(); + let heights = studio::HEIGHTS + .iter() + .enumerate() + .map(|(i, h)| { + let h = *h; + chip(("height", i), &studio::height_name(h), out.height == h) + .on_click(cx.listener(move |this, _, _, cx| this.set_studio(cx, |s| s.output.height = h))) + .into_any_element() + }) + .collect(); + let toggles = vec![ + chip("cursor", "Pointer", out.cursor) + .on_click(cx.listener(|this, _, _, cx| this.set_studio(cx, |s| s.output.cursor = !s.output.cursor))) + .into_any_element(), + chip("clicks", "Note clicks", out.clicks) + .tooltip(tip("Note where you click, to zoom in on later", None)) + .on_click(cx.listener(|this, _, _, cx| this.set_studio(cx, |s| s.output.clicks = !s.output.clicks))) + .into_any_element(), + ]; + let is_default = self.rec.default.as_ref() == Some(studio); + let last = self.rec.last.clone(); + div() + .flex() + .flex_col() + .gap(px(10.0)) + .child(caps("Files")) + .child(line("Codec", codecs)) + .child(line("Quality", qualities)) + .child(line("Frame rate", rates)) + .child(line("Screen size", heights)) + .child(line("Screen", toggles)) .child( div() - .px(px(16.0)) - .pb(px(12.0)) .text_size(px(11.0)) - .line_height(px(16.0)) + .line_height(px(15.0)) .text_color(Theme::overlay()) - .child(format!( - "{into} Recording the whole screen hides Splice; click it in the Dock to stop. \ - ↑↓ choose · ⏎ start · esc close" - )), + .child("Each source is its own file: screen.mp4, camera.mp4, microphone.m4a, system.m4a."), ) - .into_any_element() + .child( + chip( + "make-default", + if is_default { + "This is your default" + } else { + "Make this my default" + }, + is_default, + ) + .tooltip(tip("New projects record with this setup", None)) + .on_click(cx.listener(|this, _, _, cx| { + let studio = this.studio(); + match save_default(&studio) { + Ok(()) => { + this.rec.default = Some(studio); + this.say("New projects will record with this setup"); + } + Err(e) => this.say_error(format!("Could not save the default: {e}")), + } + cx.notify(); + })), + ) + .when_some(last, |d, last| { + d.child( + chip("show-take", "Show the last take's files", false) + .on_click(cx.listener(move |_, _, _, cx| cx.reveal_path(&last))), + ) + }) } - /// `splice ctl record …`. + // `splice ctl record …`. + pub(super) fn ctl_record( &mut self, request: &Value, @@ -469,92 +1277,106 @@ impl Editor { use crate::ctl::fail; let sub = request.get("sub").and_then(Value::as_str).unwrap_or("status"); match sub { - "sources" => Some(json!({ - "ok": true, - "displays": recorder::displays(), - "windows": recorder::windows(), - "note": "Record one with `record start --display N`, `--window ID`, or `--region X,Y,W,H` (screen points).", - })), + "sources" => { + let devices = match capture::devices() { + Ok(d) => d, + Err(e) => return Some(fail(e.to_string())), + }; + Some(json!({ + "ok": true, + "displays": recorder::displays(), + "windows": recorder::windows(), + "cameras": devices.cameras, + "microphones": devices.microphones, + "access": { + "screen": devices.screen_access, + "camera": devices.camera_access, + "microphone": devices.microphone_access, + }, + })) + } + "setup" => { + if let Err(e) = self.apply_setup(request, cx) { + return Some(fail(e)); + } + if request.get("default").and_then(Value::as_bool) == Some(true) { + let studio = self.studio(); + if let Err(e) = save_default(&studio) { + return Some(fail(format!("could not save the default: {e}"))); + } + self.rec.default = Some(studio); + } + Some(json!({ + "ok": true, + "setup": self.studio(), + "default": self.rec.default.as_ref() == Some(&self.studio()), + "own": self.doc.project.studio.is_some(), + })) + } "status" => Some(json!({ "ok": true, - "recording": self.recording.as_ref().map(|a| json!({ - "seconds": a.recording.seconds(), - "source": a.recording.source, - "file": a.recording.path, - })), + "recording": self.rec.recording(), + "paused": self.rec.paused(), + "seconds": self.rec.seconds(), + "live": self.rec.live.as_ref().is_some_and(|l| l.live), + "levels": self.rec.live.as_ref().map(|l| l.levels.clone()), + "problem": self.rec.problem, + "asking": self.rec.live.as_ref().and_then(|l| l.asking.clone()), + "last": self.rec.last, })), "start" => { - let source = if let Some(spec) = request.get("window") { - let id = match spec { - Value::Number(n) => n.as_u64().map(|n| n as u32), - Value::String(name) => { - let lower = name.to_lowercase(); - recorder::windows() - .into_iter() - .find(|w| { - w.app.to_lowercase().contains(&lower) || w.title.to_lowercase().contains(&lower) - }) - .map(|w| w.id) - } - _ => None, - }; - match id { - Some(id) => Source::Window { id }, - None => return Some(fail("no such window; `splice ctl record sources` lists them")), - } - } else if let Some(region) = request.get("region").and_then(Value::as_str) { - let n: Vec = region.split(',').filter_map(|v| v.trim().parse().ok()).collect(); - match n[..] { - [x, y, w, h] if w > 0 && h > 0 => Source::Region { - x: x as i32, - y: y as i32, - width: w as u32, - height: h as u32, - }, - _ => return Some(fail("--region is X,Y,WIDTH,HEIGHT in screen points")), - } - } else { - Source::Display { - number: request.get("display").and_then(Value::as_u64).unwrap_or(1) as u32, - } - }; - let options = Options { - microphone: request.get("mic").and_then(Value::as_bool) == Some(true), - clicks: request.get("clicks").and_then(Value::as_bool) == Some(true), - seconds: request.get("for").and_then(Value::as_f64).map(|s| s.ceil() as u32), - }; + if let Err(e) = self.apply_setup(request, cx) { + return Some(fail(e)); + } let into = if let Some(beat) = request.get("beat").and_then(Value::as_u64) { let id = BeatId(beat); if self.doc.project.story.index(id).is_none() { return Some(fail(format!("no beat {beat}; `splice ctl story read` lists them"))); } Into::Beat(id) + } else if request.get("into").and_then(Value::as_str) == Some("media") { + Into::Media } else { match self.time(request, "at") { Ok(Some(t)) => Into::At(t, self.track(request).unwrap_or(0)), - Ok(None) => Into::Media, + Ok(None) => Into::End, Err(e) => return Some(fail(e)), } }; - // `--hide` (shared with `track --hide`): Splice steps out of the shot. - let hide = request.get("hidden").and_then(Value::as_bool) == Some(true); - match self.start_recording(source, options, into, hide, cx) { + match self.start_take(into, Some(reply), cx) { Ok(()) => { - let file = self.recording.as_ref().map(|a| a.recording.path.clone()); - Some(json!({ "ok": true, "recording": true, "file": file })) + if let Some(s) = request.get("for").and_then(Value::as_f64) { + cx.spawn(async move |this, cx| { + cx.background_executor() + .timer(Duration::from_secs_f64(s.max(0.5) + 0.3)) + .await; + this.update(cx, |editor, cx| editor.stop_take(cx)).ok(); + }) + .detach(); + } + None } Err(e) => Some(fail(e)), } } + "pause" | "resume" => { + if !self.rec.recording() { + return Some(fail("not recording")); + } + if (sub == "pause") != self.rec.paused() { + self.pause_take(cx); + } + Some(json!({ "ok": true })) + } "stop" => { - let Some(active) = &mut self.recording else { + if !self.rec.recording() { return Some(fail("not recording")); - }; + } let wait = request.get("wait").and_then(Value::as_bool) == Some(true); if wait { - active.waiters.push(reply); + self.rec.waiters.push(reply); } - self.stop_recording(cx); + self.stop_take(cx); if wait { None } else { @@ -562,8 +1384,288 @@ impl Editor { } } other => Some(fail(format!( - "unknown `record {other}`: use sources, start, stop or status" + "unknown `record {other}`: use sources, setup, start, pause, resume, stop or status" ))), } } + + /// Changes the setup from `record setup` or `record start` flags. + fn apply_setup(&mut self, request: &Value, cx: &mut Context) -> Result<(), String> { + let text = |k: &str| request.get(k).and_then(Value::as_str).map(str::to_string); + let flag = |k: &str| request.get(k).and_then(Value::as_bool); + let mut studio = self.studio(); + let before = studio.clone(); + if let Some(screen) = text("screen") { + studio.screen = parse_screen(&screen)?; + } else if let Some(n) = request.get("display").and_then(Value::as_u64) { + studio.screen = Some(Screen::Display { number: n as u32 }); + } else if let Some(w) = request.get("window") { + let windows = recorder::windows(); + let found = match w { + Value::Number(n) => windows.iter().find(|x| Some(x.id as u64) == n.as_u64()), + Value::String(name) => { + let lower = name.to_lowercase(); + windows + .iter() + .find(|x| x.app.to_lowercase().contains(&lower) || x.title.to_lowercase().contains(&lower)) + } + _ => None, + }; + let w = found.ok_or("no such window; `splice ctl record sources` lists them")?; + studio.screen = Some(Screen::Window { + app: w.app.clone(), + title: w.title.clone(), + }); + } + if let Some(region) = text("region") { + let n: Vec = region.split(',').filter_map(|v| v.trim().parse().ok()).collect(); + let [x, y, width, height] = n[..] else { + return Err("--region is X,Y,WIDTH,HEIGHT in points from the display's top left".into()); + }; + let display = match &studio.screen { + Some(Screen::Display { number }) => *number, + _ => 1, + }; + studio.screen = Some(Screen::Region { + display, + x, + y, + width, + height, + }); + } + let need_devices = text("camera").is_some() || text("microphone").is_some() || flag("mic") == Some(true); + let devices = if need_devices { + Some(capture::devices().map_err(|e| e.to_string())?) + } else { + None + }; + let choose = |name: &str, there: &[capture::Device], what: &str| -> Result, String> { + if matches!(name, "none" | "off" | "") { + return Ok(None); + } + let lower = name.to_lowercase(); + let found = there + .iter() + .find(|d| d.id == name) + .or_else(|| there.iter().find(|d| d.name.to_lowercase().contains(&lower))) + .or_else(|| there.iter().find(|d| d.default).filter(|_| lower == "default")); + let d = found.ok_or(format!( + "no {what} like “{name}”; `splice ctl record sources` lists them" + ))?; + Ok(Some(studio::Device { + id: d.id.clone(), + name: d.name.clone(), + })) + }; + if let (Some(name), Some(d)) = (text("camera"), &devices) { + studio.camera = choose(&name, &d.cameras, "camera")?; + } + if let Some(d) = &devices { + if let Some(name) = text("microphone") { + studio.microphone = choose(&name, &d.microphones, "microphone")?; + } else if flag("mic") == Some(true) && studio.microphone.is_none() { + studio.microphone = choose("default", &d.microphones, "microphone")?; + } + } + if let Some(on) = flag("system") { + studio.system_audio = on; + } + if let Some(layout) = text("layout") { + studio.scene.layout = Layout::parse(&layout).ok_or("--layout is screen, bubble, side or camera")?; + } + if let Some(corner) = text("corner") { + studio.scene.corner = Corner::parse(&corner).ok_or("--corner is tl, tr, bl or br")?; + } + if let Some(size) = request.get("size").and_then(Value::as_f64) { + studio.scene.size = (size as f32).clamp(0.08, 0.9); + } + if let Some(split) = request.get("split").and_then(Value::as_f64) { + studio.scene.split = (split as f32).clamp(0.15, 0.6); + } + if let Some(shape) = text("shape") { + studio.scene.shape = match shape.as_str() { + "circle" | "round" => Bubble::Circle, + "square" => Bubble::Square, + _ => return Err("--shape is circle or square".into()), + }; + } + if let Some(codec) = text("codec") { + studio.output.codec = match codec.to_lowercase().as_str() { + "h264" | "h.264" | "avc" => Codec::H264, + "hevc" | "h265" | "h.265" => Codec::Hevc, + _ => return Err("--codec is h264 or hevc".into()), + }; + } + if let Some(q) = text("quality") { + studio.output.quality = Quality::ALL + .into_iter() + .find(|x| x.name().eq_ignore_ascii_case(&q)) + .ok_or("--quality is good, high or best")?; + } + if let Some(fps) = request.get("fps").and_then(Value::as_u64) { + studio.output.fps = (fps as u32).clamp(1, 120); + } + if let Some(h) = text("height") { + studio.output.height = match h.to_lowercase().trim_end_matches('p') { + "full" | "native" | "none" => None, + "4k" => Some(2160), + n => Some(n.parse().map_err(|_| "--height is full, 4k, 1440, 1080 or a number")?), + }; + } + if let Some(on) = flag("cursor") { + studio.output.cursor = on; + } + if let Some(on) = flag("clicks") { + studio.output.clicks = on; + } + if studio != before || self.doc.project.studio.is_none() && request.get("default").is_some() { + self.edit(cx, |p| p.studio = Some(studio)); + self.relive(cx); + } + Ok(()) + } +} + +/// `--screen`: display:N (or N), window:APP, none. +fn parse_screen(s: &str) -> Result, String> { + let (kind, rest) = s.split_once(':').unwrap_or((s, "")); + match kind.to_lowercase().as_str() { + "none" | "off" => Ok(None), + "main" => Ok(Some(Screen::Display { number: 1 })), + "display" => Ok(Some(Screen::Display { + number: rest.parse().unwrap_or(1).max(1), + })), + "window" => { + let lower = rest.to_lowercase(); + let w = recorder::windows() + .into_iter() + .find(|w| w.app.to_lowercase().contains(&lower) || w.title.to_lowercase().contains(&lower)) + .ok_or(format!( + "no window like “{rest}”; `splice ctl record sources` lists them" + ))?; + Ok(Some(Screen::Window { + app: w.app, + title: w.title, + })) + } + n => n + .parse::() + .map(|number| Some(Screen::Display { number: number.max(1) })) + .map_err(|_| "--screen is display:N, window:APP or none".to_string()), + } +} + +fn name_of(feed: &str) -> &'static str { + match feed { + "screen" => "Screen", + "camera" => "Camera", + "microphone" => "Mic", + _ => "Mac sound", + } +} + +/// "Take 2026-10-28 14.03.12" from a file in its folder. +fn take_name(path: &Path) -> String { + path.parent() + .and_then(|d| d.file_name()) + .map(|n| n.to_string_lossy().trim_start_matches("Take ").to_string()) + .unwrap_or_default() +} + +/// A live picture where `look` puts it on a stage of `stage` points. +fn picture(image: Arc, natural: (u32, u32), look: &Look, stage: (f32, f32)) -> AnyElement { + let (sw, sh) = stage; + let (nw, nh) = (natural.0.max(1) as f32, natural.1.max(1) as f32); + let fit = (sw / nw).min(sh / nh) * look.scale; + let (pw, ph) = (nw * fit, nh * fit); + let (cx, cy) = (sw / 2.0 + look.x * sw, sh / 2.0 + look.y * sh); + let (pl, pt) = (cx - pw / 2.0, cy - ph / 2.0); + // What shows of it: all, a mask's shape, or what a crop leaves. + let (mut l, mut t, mut r, mut b) = (0.0, 0.0, 1.0, 1.0); + let mut round = false; + if let Some(m) = look.mask { + (l, t, r, b) = ( + m.x - m.width / 2.0, + m.y - m.height / 2.0, + m.x + m.width / 2.0, + m.y + m.height / 2.0, + ); + round = m.shape == splice_core::look::Shape::Ellipse; + } else if let Some(c) = look.crop { + (l, t, r, b) = (c.left, c.top, 1.0 - c.right, 1.0 - c.bottom); + } + let (vl, vt, vw, vh) = (pl + l * pw, pt + t * ph, (r - l) * pw, (b - t) * ph); + div() + .absolute() + .left(px(vl)) + .top(px(vt)) + .w(px(vw)) + .h(px(vh)) + .overflow_hidden() + .when(round, |d| d.rounded_full()) + .child( + img(image) + .absolute() + .left(px(pl - vl)) + .top(px(pt - vt)) + .w(px(pw)) + .h(px(ph)) + .object_fit(ObjectFit::Fill), + ) + .into_any_element() +} + +/// A small choice that is on or off. +fn chip(id: impl std::convert::Into, label: &str, on: bool) -> gpui::Stateful { + div() + .id(id) + .h(px(24.0)) + .px(px(9.0)) + .rounded(px(6.0)) + .flex() + .items_center() + .cursor_pointer() + .border_1() + .border_color(if on { Theme::surface2() } else { Theme::edge() }) + .when(on, |d| d.bg(Theme::surface0())) + .hover(|d| d.bg(Theme::surface0())) + .text_size(px(11.5)) + .text_color(if on { Theme::text() } else { Theme::subtext() }) + .child(label.to_string()) +} + +fn switch(on: bool) -> impl IntoElement { + div() + .w(px(26.0)) + .h(px(15.0)) + .rounded_full() + .p(px(2.0)) + .flex() + .when(on, |d| d.justify_end().bg(Theme::accent())) + .when(!on, |d| d.bg(Theme::surface1())) + .child(div().size(px(11.0)).rounded_full().bg(Theme::ground())) +} + +/// How loud a sound is: a bar, green to the accent near the top. +fn meter(level: Option) -> impl IntoElement { + let level = level.unwrap_or(0.0).clamp(0.0, 1.0); + // Shown in decibels, so speech sits around the middle. + let shown = if level <= 0.0 { + 0.0 + } else { + ((20.0 * level.log10() + 60.0) / 60.0).clamp(0.0, 1.0) + }; + div() + .mx(px(8.0)) + .mb(px(4.0)) + .h(px(4.0)) + .rounded_full() + .bg(Theme::surface0()) + .overflow_hidden() + .child(div().h_full().w(relative(shown)).rounded_full().bg(if level > 0.89 { + Theme::accent() + } else { + gpui::rgb(0x7fb069) + })) } diff --git a/crates/splice-app/src/editor/render.rs b/crates/splice-app/src/editor/render.rs index 129c253..4f4abfe 100644 --- a/crates/splice-app/src/editor/render.rs +++ b/crates/splice-app/src/editor/render.rs @@ -365,6 +365,8 @@ impl Render for Editor { d.child(self.render_workbench(window, cx)) } else if self.mode == story::Mode::Story { d.child(self.render_board(cx)) + } else if self.mode == story::Mode::Record { + d.child(self.render_studio(cx)) } else { d.child(self.render_viewer(cx)).when(inspector_open, |d| { d.child(div().w(px(GAP)).flex_none()) @@ -543,7 +545,7 @@ impl Editor { } } - /// Edit or Story: the two ways to look at the movie. + /// Edit, Story or Record: the ways to work on the movie. fn mode_switch(&mut self, cx: &mut Context) -> impl IntoElement + use<> { let tab = |this: &Self, mode: story::Mode, label: &'static str, cx: &mut Context| { let on = this.mode == mode; @@ -574,6 +576,10 @@ impl Editor { .gap(px(2.0)) .child(tab(self, story::Mode::Edit, "Edit", cx).tooltip(tip("Edit the movie", Some("⌘3")))) .child(tab(self, story::Mode::Story, "Story", cx).tooltip(tip("Plan it: beats, cues, shots", Some("⌘3")))) + .child( + tab(self, story::Mode::Record, "Record", cx) + .tooltip(tip("Record: screen, camera, microphone", Some("⇧⌘R"))), + ) } fn render_titlebar(&mut self, cx: &mut Context) -> impl IntoElement { @@ -1662,7 +1668,6 @@ impl Editor { let content = match &self.overlay { Overlay::None => return None, - Overlay::Record { .. } => panel().child(self.record_overlay(cx)), Overlay::Cast { beat, index } => { let (beat, index) = (*beat, *index); panel().w(px(380.0)).child(self.cast_overlay(beat, index, cx)) diff --git a/crates/splice-app/src/editor/story.rs b/crates/splice-app/src/editor/story.rs index 00531da..4547b86 100644 --- a/crates/splice-app/src/editor/story.rs +++ b/crates/splice-app/src/editor/story.rs @@ -16,6 +16,8 @@ pub enum Mode { #[default] Edit, Story, + /// What the project records, live. + Record, } /// Which words are being typed. @@ -79,10 +81,12 @@ fn seconds(t: Time) -> String { impl Editor { pub(super) fn set_mode(&mut self, mode: Mode, cx: &mut Context) { self.mode = mode; - if mode == Mode::Story { + if mode != Mode::Edit { self.workbench = None; } self.typing = None; + // The sources are live while the Record tab is open. + self.relive(cx); cx.notify(); } @@ -1085,7 +1089,7 @@ impl Editor { })).collect::>(), "length": secs(story.end()), "selected": self.beat.map(|b| b.0), - "view": match self.mode { Mode::Edit => "edit", Mode::Story => "story" }, + "view": match self.mode { Mode::Edit => "edit", Mode::Story => "story", Mode::Record => "record" }, }) } @@ -1132,7 +1136,8 @@ impl Editor { let mode = match text("view").as_deref() { Some("story") => Mode::Story, Some("edit") => Mode::Edit, - _ => return fail("view is edit or story"), + Some("record") => Mode::Record, + _ => return fail("view is edit, story or record"), }; self.set_mode(mode, cx); json!({ "ok": true, "view": text("view") }) diff --git a/crates/splice-app/src/icons.rs b/crates/splice-app/src/icons.rs index cd25d54..448fa74 100644 --- a/crates/splice-app/src/icons.rs +++ b/crates/splice-app/src/icons.rs @@ -96,6 +96,10 @@ fn icon(name: &str) -> Option<&'static str> { r#""# } "flag" => r#""#, + "camera" => r#""#, + "mic" => { + r#""# + } "record" => r#""#, "check" => r#""#, "close" => r#""#, diff --git a/crates/splice-app/src/main.rs b/crates/splice-app/src/main.rs index a641ba5..b78fac2 100644 --- a/crates/splice-app/src/main.rs +++ b/crates/splice-app/src/main.rs @@ -2,6 +2,7 @@ mod actions; mod audio; +mod capture; mod cli; mod ctl; mod editor; diff --git a/crates/splice-app/src/recorder.rs b/crates/splice-app/src/recorder.rs index b2650cd..8943619 100644 --- a/crates/splice-app/src/recorder.rs +++ b/crates/splice-app/src/recorder.rs @@ -1,10 +1,9 @@ -//! Recording the screen: a whole display, one window, or a rectangle, with -//! the cursor, optional clicks and microphone. macOS's own `screencapture` -//! does the capture (hardware H.264, 60 fps); Splice starts it, stops it with -//! an interrupt so the file is finished properly, and brings the take in. +//! What can be recorded (displays and windows), and the clicks noticed +//! while recording, kept beside a take to zoom in on later. The recording +//! itself is `capture`'s. use crate::session::dirs; -use anyhow::{Context, Result}; + use core_foundation::array::CFArray; use core_foundation::base::{CFType, TCFType}; use core_foundation::dictionary::CFDictionary; @@ -14,10 +13,9 @@ use objc::runtime::Object; use objc::{class, msg_send, sel, sel_impl}; use serde::Serialize; use std::path::PathBuf; -use std::process::{Child, Command, Stdio}; +use std::process::Command; use std::sync::Arc; use std::sync::atomic::{AtomicBool, Ordering}; -use std::time::Instant; /// What to record. #[derive(Clone, Debug, PartialEq, Serialize)] @@ -39,18 +37,15 @@ pub enum Source { }, } -#[derive(Clone, Debug, Default, Serialize)] -pub struct Options { - pub microphone: bool, - pub clicks: bool, - /// Stops by itself after this many seconds. - pub seconds: Option, -} - #[derive(Clone, Debug, Serialize)] pub struct Display { pub number: u32, + /// CoreGraphics' number for it. + pub id: u32, pub name: String, + /// Its top left, in points from the main display's top left. + pub x: f64, + pub y: f64, pub width: f64, pub height: f64, } @@ -78,9 +73,21 @@ pub fn displays() -> Vec { unsafe { let screens: *mut Object = msg_send![class!(NSScreen), screens]; let count: usize = msg_send![screens, count]; + let key = CFString::new("NSScreenNumber"); + let mut main_height = 0.0; for i in 0..count { let screen: *mut Object = msg_send![screens, objectAtIndex: i]; let frame: Rect = msg_send![screen, frame]; + if i == 0 { + main_height = frame.h; + } + let description: *mut Object = msg_send![screen, deviceDescription]; + let number: *mut Object = msg_send![description, objectForKey: key.as_concrete_TypeRef() as *mut Object]; + let id: u32 = if number.is_null() { + 0 + } else { + msg_send![number, unsignedIntValue] + }; let name: *mut Object = msg_send![screen, localizedName]; let text: *const std::os::raw::c_char = msg_send![name, UTF8String]; let name = if text.is_null() { @@ -90,7 +97,11 @@ pub fn displays() -> Vec { }; out.push(Display { number: i as u32 + 1, + id, name, + // Cocoa counts up from the bottom; CoreGraphics, down from the top. + x: frame.x, + y: main_height - (frame.y + frame.h), width: frame.w, height: frame.h, }); @@ -150,18 +161,6 @@ pub fn windows() -> Vec { out } -/// A recording under way. -pub struct Recording { - child: Child, - pub path: PathBuf, - pub started: Instant, - /// The same moment by the clock files are stamped with. - begun: std::time::SystemTime, - pub source: Source, - /// Watching the mouse for clicks, until told to stop. - clicks: Option<(Arc, std::thread::JoinHandle>)>, -} - /// A click while recording: seconds in, and where, as fractions of what is /// recorded from its top left. #[derive(Clone, Copy, Debug, PartialEq, Serialize, serde::Deserialize)] @@ -204,8 +203,6 @@ unsafe extern "C" { fn CGEventSourceCounterForEventType(state: u32, event: u32) -> u32; fn CGEventCreate(source: *const std::ffi::c_void) -> *mut std::ffi::c_void; fn CGEventGetLocation(event: *mut std::ffi::c_void) -> Point; - fn CGPreflightScreenCaptureAccess() -> bool; - fn CGRequestScreenCaptureAccess() -> bool; } #[link(name = "CoreFoundation", kind = "framework")] @@ -300,36 +297,39 @@ fn window_bounds(id: u32) -> Option<(f64, f64, f64, f64)> { None } -/// When the file says nothing, how long after it is started screencapture -/// begins. -const LATE: f64 = 0.5; - -/// How long after `started` the take's first frame was: screencapture -/// takes a while to begin (a quarter second to more than one, measured), and -/// its file is born then. -fn lateness(path: &std::path::Path, started: std::time::SystemTime) -> f64 { - std::fs::metadata(path) - .and_then(|m| m.created()) - .ok() - .and_then(|born| born.duration_since(started).ok()) - .map(|d| d.as_secs_f64()) - .filter(|&d| d < 5.0) - .unwrap_or(LATE) -} - -/// Clicks in the recording's own time, without the click that stopped it -/// (the Stop button, or Splice in the Dock) in its last half second. -/// `late` is how long after the clicks' clock the take begins. -fn settle(clicks: Vec, ended: f64, late: f64) -> Vec { +/// Clicks noted by the wall clock (seconds since 1970) in a take's own +/// time: the take began at `zero`, was paused for `pauses` (wall clock, +/// from and to) and lasted `length`. Clicks outside it are left out, as is +/// the click that stopped it, in its last half second. +pub fn in_take(clicks: Vec, zero: f64, pauses: &[(f64, f64)], length: f64) -> Vec { clicks .into_iter() - .filter(|c| c.t < ended - 0.5 && c.t >= late) - .map(|c| Click { t: c.t - late, ..c }) + .filter_map(|c| { + if pauses.iter().any(|(a, b)| c.t >= *a && c.t < *b) { + return None; + } + let paused: f64 = pauses.iter().filter(|(_, b)| *b <= c.t).map(|(a, b)| b - a).sum(); + let t = c.t - zero - paused; + (t >= 0.0 && t < length - 0.5).then_some(Click { t, ..c }) + }) .collect() } -/// Watches for clicks inside `area` until `stop`, from `started`. -fn watch(area: (f64, f64, f64, f64), started: Instant, stop: Arc) -> Vec { +/// Starts noting clicks inside what `source` records, by the wall clock, +/// until the flag is set; the thread's result is the clicks. +pub fn watch_clicks(source: &Source) -> Option<(Arc, std::thread::JoinHandle>)> { + let area = area(source)?; + let stop = Arc::new(AtomicBool::new(false)); + let flag = stop.clone(); + std::thread::Builder::new() + .name("splice-clicks".into()) + .spawn(move || watch(area, flag)) + .ok() + .map(|watcher| (stop, watcher)) +} + +/// Watches for clicks inside `area` until `stop`. +fn watch(area: (f64, f64, f64, f64), stop: Arc) -> Vec { let (ax, ay, aw, ah) = area; let mut clicks = Vec::new(); // Counting presses rather than looking at the button: a tap on a @@ -342,11 +342,10 @@ fn watch(area: (f64, f64, f64, f64), started: Instant, stop: Arc) -> if new && let Some((x, y)) = mouse() { let (u, v) = ((x - ax) / aw.max(1.0), (y - ay) / ah.max(1.0)); if (0.0..=1.0).contains(&u) && (0.0..=1.0).contains(&v) { - clicks.push(Click { - t: started.elapsed().as_secs_f64(), - x: u, - y: v, - }); + let t = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0.0, |d| d.as_secs_f64()); + clicks.push(Click { t, x: u, y: v }); } } std::thread::sleep(std::time::Duration::from_millis(8)); @@ -354,141 +353,24 @@ fn watch(area: (f64, f64, f64, f64), started: Instant, stop: Arc) -> clicks } -impl Recording { - pub fn seconds(&self) -> f64 { - self.started.elapsed().as_secs_f64() - } - - /// Whether it has stopped by itself (a time limit, an error). - pub fn finished(&mut self) -> bool { - matches!(self.child.try_wait(), Ok(Some(_))) - } - - /// Stops it and waits for the file to be finished, with its clicks - /// beside it. Blocking (a second or two). - pub fn stop(mut self) -> Result { - if !self.finished() { - unsafe { - libc_kill(self.child.id() as i32, 2); - } - } - let ended = self.started.elapsed().as_secs_f64(); - let clicks = self.clicks.take().and_then(|(stop, watcher)| { - stop.store(true, Ordering::Relaxed); - watcher.join().ok() - }); - let output = self.child.wait_with_output()?; - let ok = std::fs::metadata(&self.path).is_ok_and(|m| m.len() > 0); - if !ok { - let why = String::from_utf8_lossy(&output.stderr).trim().to_string(); - anyhow::bail!(if why.is_empty() { - "the recording came out empty; check that Splice's app (or your terminal) has Screen Recording \ - permission in System Settings → Privacy & Security" - .to_string() - } else { - why - }); - } - if let Some(clicks) = clicks - && let clicks = settle(clicks, ended, lateness(&self.path, self.begun)) - && !clicks.is_empty() - { - let _ = std::fs::write( - clicks_file(&self.path), - serde_json::to_string(&clicks).unwrap_or_default(), - ); - } - Ok(self.path) - } -} - -unsafe extern "C" { - #[link_name = "kill"] - fn libc_kill(pid: i32, signal: i32) -> i32; -} - /// Where takes are kept before they are brought into a project. pub fn takes() -> PathBuf { dirs().projects.join("Recordings") } -/// Starts recording `source`. -pub fn start(source: Source, options: &Options) -> Result { - // Without permission, screencapture writes nothing, and that would only - // show when the take ends. Asking puts Splice in the list (and, the - // first time, brings up the system's prompt). - if !unsafe { CGPreflightScreenCaptureAccess() } { - unsafe { CGRequestScreenCaptureAccess() }; - anyhow::bail!( - "Splice needs Screen Recording permission: turn it on in System Settings → Privacy & Security → \ - Screen Recording, then reopen Splice" - ); - } - let dir = takes(); - std::fs::create_dir_all(&dir)?; - let stamp = { - let secs = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .map_or(0, |d| d.as_secs()); - let output = Command::new("date") - .args(["-r", &secs.to_string(), "+%Y-%m-%d %H.%M.%S"]) - .output() - .ok(); - output - .map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()) - .filter(|s| !s.is_empty()) - .unwrap_or_else(|| secs.to_string()) - }; - let path = dir.join(format!("Recording {stamp}.mov")); - let mut command = Command::new("/usr/sbin/screencapture"); - command.args(["-x", "-v", "-C"]); - match &source { - Source::Display { number } => { - command.arg(format!("-D{}", number.max(&1))); - } - Source::Window { id } => { - command.arg(format!("-l{id}")).arg("-o"); - } - Source::Region { x, y, width, height } => { - command.arg(format!("-R{x},{y},{width},{height}")); - } - } - if options.microphone { - command.arg("-g"); - } - if options.clicks { - command.arg("-k"); - } - if let Some(s) = options.seconds { - command.arg(format!("-V{}", s.max(1))); - } - let child = command - .arg(&path) - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::piped()) - .spawn() - .context("starting screencapture")?; - let started = Instant::now(); - let begun = std::time::SystemTime::now(); - // Clicks are noted for zooming in on later. - let clicks = area(&source).and_then(|area| { - let stop = Arc::new(AtomicBool::new(false)); - let flag = stop.clone(); - std::thread::Builder::new() - .name("splice-clicks".into()) - .spawn(move || watch(area, started, flag)) - .ok() - .map(|watcher| (stop, watcher)) - }); - Ok(Recording { - child, - path, - started, - begun, - source, - clicks, - }) +/// Now, as a take's name: 2026-10-28 14.03.12. +pub fn stamp() -> String { + let secs = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| d.as_secs()); + let output = Command::new("date") + .args(["-r", &secs.to_string(), "+%Y-%m-%d %H.%M.%S"]) + .output() + .ok(); + output + .map(|o| String::from_utf8_lossy(&o.stdout).trim().to_string()) + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| secs.to_string()) } #[cfg(test)] @@ -496,20 +378,15 @@ mod tests { use super::*; #[test] - fn the_click_that_stops_a_recording_is_not_kept() { - let clicks = vec![ - Click { t: 1.0, x: 0.5, y: 0.5 }, - Click { - t: 9.8, - x: 0.9, - y: 0.99, - }, - ]; - let kept = settle(clicks, 10.0, 0.4); - assert_eq!(kept.len(), 1); - assert!((kept[0].t - 0.6).abs() < 1e-9, "in the recording's own time"); - let early = settle(vec![Click { t: 0.1, x: 0.5, y: 0.5 }], 10.0, 0.4); - assert!(early.is_empty(), "a click before the first frame is not in the take"); + fn clicks_are_timed_in_the_take_without_its_pauses() { + let at = |t: f64| Click { t, x: 0.5, y: 0.5 }; + let zero = 1000.0; + let clicks = vec![at(999.0), at(1001.0), at(1003.0), at(1006.0), at(1009.8)]; + // Paused from 2 s in until 5 s in (wall clock 1002 to 1005). + let kept = in_take(clicks, zero, &[(1002.0, 1005.0)], 7.0); + let times: Vec = kept.iter().map(|c| c.t).collect(); + // Before the take, while paused, and the stopping click are gone. + assert_eq!(times, vec![1.0, 3.0]); } #[test] diff --git a/crates/splice-core/src/lib.rs b/crates/splice-core/src/lib.rs index 491e1bc..21917d5 100644 --- a/crates/splice-core/src/lib.rs +++ b/crates/splice-core/src/lib.rs @@ -15,6 +15,7 @@ pub mod motion; pub mod score; pub mod store; pub mod story; +pub mod studio; pub mod symbol; pub mod textmotion; pub mod time; diff --git a/crates/splice-core/src/model.rs b/crates/splice-core/src/model.rs index 5d6bf47..77d204e 100644 --- a/crates/splice-core/src/model.rs +++ b/crates/splice-core/src/model.rs @@ -467,6 +467,9 @@ pub struct Project { /// shape: 2.39 for a cinema look. Drawn over everything. #[serde(default, skip_serializing_if = "Option::is_none")] pub letterbox: Option, + /// How this project records; None uses the default the user has set. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub studio: Option, next_id: u64, } @@ -486,6 +489,7 @@ impl Default for Project { bible: crate::bible::Bible::default(), hold: None, letterbox: None, + studio: None, next_id: 2, } } diff --git a/crates/splice-core/src/studio.rs b/crates/splice-core/src/studio.rs new file mode 100644 index 0000000..a5f12d8 --- /dev/null +++ b/crates/splice-core/src/studio.rs @@ -0,0 +1,478 @@ +//! The recording setup a project keeps: what is recorded (a screen or a +//! window, a camera, a microphone, the sound the Mac plays), how the +//! pictures sit together, and how the files are made. Every source is +//! written to its own file; the scene is how a take is laid on the timeline, +//! so it can still be changed after recording. + +use crate::look::{Crop, Look, Mask, Shape}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(default)] +pub struct Studio { + #[serde(skip_serializing_if = "Option::is_none")] + pub screen: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub camera: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub microphone: Option, + /// The sound the Mac plays (not Splice's own). + pub system_audio: bool, + pub scene: Scene, + pub output: Output, +} + +/// What of the screen is recorded. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(tag = "kind", rename_all = "lowercase")] +pub enum Screen { + /// A display, counting from 1 (the main one). + Display { number: u32 }, + /// A window, found again by its app and title (window numbers do not + /// last from one launch to the next). + Window { app: String, title: String }, + /// Part of a display, in points from its top left. + Region { + display: u32, + x: f64, + y: f64, + width: f64, + height: f64, + }, +} + +/// A camera or microphone: the system's id for it, and its name, to show +/// and to find it by when the id has changed. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct Device { + pub id: String, + pub name: String, +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum Layout { + /// The screen alone. + Screen, + /// The screen, with the camera small in a corner. + #[default] + Bubble, + /// The screen beside the camera. + Side, + /// The camera alone. + Camera, +} + +impl Layout { + pub const ALL: [Layout; 4] = [Layout::Screen, Layout::Bubble, Layout::Side, Layout::Camera]; + + pub fn name(self) -> &'static str { + match self { + Layout::Screen => "Screen", + Layout::Bubble => "Bubble", + Layout::Side => "Side by side", + Layout::Camera => "Camera", + } + } + + pub fn parse(s: &str) -> Option { + match s.to_lowercase().as_str() { + "screen" => Some(Layout::Screen), + "bubble" | "corner" => Some(Layout::Bubble), + "side" | "side-by-side" | "sidebyside" => Some(Layout::Side), + "camera" | "cam" => Some(Layout::Camera), + _ => None, + } + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum Corner { + TopLeft, + TopRight, + BottomLeft, + #[default] + BottomRight, +} + +impl Corner { + pub const ALL: [Corner; 4] = [ + Corner::TopLeft, + Corner::TopRight, + Corner::BottomLeft, + Corner::BottomRight, + ]; + + pub fn parse(s: &str) -> Option { + match s.to_lowercase().as_str() { + "tl" | "top-left" => Some(Corner::TopLeft), + "tr" | "top-right" => Some(Corner::TopRight), + "bl" | "bottom-left" => Some(Corner::BottomLeft), + "br" | "bottom-right" => Some(Corner::BottomRight), + _ => None, + } + } + + fn right(self) -> bool { + matches!(self, Corner::TopRight | Corner::BottomRight) + } + + fn bottom(self) -> bool { + matches!(self, Corner::BottomLeft | Corner::BottomRight) + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum Bubble { + #[default] + Circle, + Square, +} + +/// How the pictures sit together. +#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)] +#[serde(default)] +pub struct Scene { + pub layout: Layout, + pub corner: Corner, + /// The bubble's height, as a fraction of the canvas's. + pub size: f32, + pub shape: Bubble, + /// Side by side: the camera's share of the width. + pub split: f32, +} + +impl Default for Scene { + fn default() -> Scene { + Scene { + layout: Layout::Bubble, + corner: Corner::BottomRight, + size: 0.3, + shape: Bubble::Circle, + split: 0.32, + } + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum Codec { + /// Plays everywhere. + #[default] + H264, + /// Smaller files, the same picture. + Hevc, +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] +pub enum Quality { + Good, + #[default] + High, + Best, +} + +impl Quality { + pub const ALL: [Quality; 3] = [Quality::Good, Quality::High, Quality::Best]; + + /// The encoder's constant quality, 0 to 1. + pub fn value(self) -> f64 { + match self { + Quality::Good => 0.5, + Quality::High => 0.65, + Quality::Best => 0.8, + } + } + + pub fn name(self) -> &'static str { + match self { + Quality::Good => "Good", + Quality::High => "High", + Quality::Best => "Best", + } + } +} + +/// How the files are made. +#[derive(Clone, Copy, Debug, PartialEq, Serialize, Deserialize)] +#[serde(default)] +pub struct Output { + pub codec: Codec, + pub quality: Quality, + pub fps: u32, + /// The screen file is no taller than this; None keeps every pixel. + pub height: Option, + /// The pointer drawn in the screen file. + pub cursor: bool, + /// Clicks noted, to zoom in on later. + pub clicks: bool, +} + +impl Default for Output { + fn default() -> Output { + Output { + codec: Codec::H264, + quality: Quality::High, + fps: 30, + height: Some(2160), + cursor: true, + clicks: true, + } + } +} + +/// Heights the screen file can be held to. +pub const HEIGHTS: [Option; 4] = [None, Some(2160), Some(1440), Some(1080)]; + +pub fn height_name(h: Option) -> String { + match h { + None => "Full".into(), + Some(2160) => "4K".into(), + Some(h) => format!("{h}p"), + } +} + +impl Studio { + pub fn is_default(&self) -> bool { + *self == Studio::default() + } + + /// Whether anything is chosen to record. + pub fn has_sources(&self) -> bool { + self.screen.is_some() || self.camera.is_some() || self.microphone.is_some() || self.system_audio + } + + /// What the take's pictures look like on a canvas of `canvas` pixels: + /// the screen's and the camera's, given their sizes. None leaves that + /// picture off the timeline (it is still recorded, and in the media). + pub fn looks(&self, canvas: (u32, u32), screen: Option<(u32, u32)>, camera: Option<(u32, u32)>) -> Placed { + let scene = self.scene; + let (w, h) = (canvas.0.max(1) as f32, canvas.1.max(1) as f32); + // Scale 1 is the picture fitted inside the canvas. + let fit = |s: (u32, u32)| (w / s.0.max(1) as f32).min(h / s.1.max(1) as f32); + let full = |s: (u32, u32)| Look { + // Filling the canvas, any spare cut away. + scale: (w / s.0.max(1) as f32).max(h / s.1.max(1) as f32) / fit(s), + ..Look::default() + }; + let layout = match (scene.layout, screen, camera) { + // A layout with a picture missing falls back to the one there is. + (Layout::Camera, Some(_), None) | (_, Some(_), None) => Layout::Screen, + (_, None, Some(_)) => Layout::Camera, + (layout, _, _) => layout, + }; + match (layout, screen, camera) { + (Layout::Screen, Some(_), _) => Placed { + screen: Some(Look::default()), + camera: None, + }, + (Layout::Camera, _, Some(c)) => Placed { + screen: None, + camera: Some(full(c)), + }, + (Layout::Bubble, Some(_), Some(c)) => { + let (cw, ch) = (c.0.max(1) as f32, c.1.max(1) as f32); + // The bubble shows most of the camera's shorter side. + let side = cw.min(ch) * 0.94; + let diameter = scene.size.clamp(0.08, 0.9) * h; + let margin = 0.035 * h; + let scale = diameter / (side * fit(c)); + let cx = if scene.corner.right() { + w - margin - diameter / 2.0 + } else { + margin + diameter / 2.0 + }; + let cy = if scene.corner.bottom() { + h - margin - diameter / 2.0 + } else { + margin + diameter / 2.0 + }; + let mask = Mask { + shape: match scene.shape { + Bubble::Circle => Shape::Ellipse, + Bubble::Square => Shape::Rect, + }, + x: 0.5, + y: 0.5, + width: side / cw, + height: side / ch, + feather: 0.015, + invert: false, + }; + Placed { + screen: Some(Look::default()), + camera: Some(Look { + x: (cx - w / 2.0) / w, + y: (cy - h / 2.0) / h, + scale, + mask: Some(mask), + ..Look::default() + }), + } + } + (Layout::Side, Some(s), Some(c)) => { + let share = scene.split.clamp(0.15, 0.6); + let (left, right) = (w * (1.0 - share), w * share); + // The screen fitted into its part, the camera filling its own. + let (sw, sh) = (s.0.max(1) as f32, s.1.max(1) as f32); + let screen_fit = (left / sw).min(h / sh); + let (cw, ch) = (c.0.max(1) as f32, c.1.max(1) as f32); + let cover = (right / cw).max(h / ch); + let (shown_w, shown_h) = (cw * cover, ch * cover); + let cut_x = ((shown_w - right) / shown_w / 2.0).max(0.0); + let cut_y = ((shown_h - h) / shown_h / 2.0).max(0.0); + Placed { + screen: Some(Look { + x: (left / 2.0 - w / 2.0) / w, + scale: screen_fit / fit(s), + ..Look::default() + }), + camera: Some(Look { + x: (w - right / 2.0 - w / 2.0) / w, + scale: cover / fit(c), + crop: Some(Crop { + left: cut_x, + right: cut_x, + top: cut_y, + bottom: cut_y, + }), + ..Look::default() + }), + } + } + _ => Placed::default(), + } + } +} + +/// Where a take's pictures go. +#[derive(Clone, Debug, Default, PartialEq)] +pub struct Placed { + pub screen: Option, + pub camera: Option, +} + +/// One recording: its folder, and the file each source was written to, +/// each starting at the same moment and lasting as long. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(default)] +pub struct Take { + pub screen: Option, + pub camera: Option, + pub microphone: Option, + pub system: Option, + pub seconds: f64, +} + +impl Take { + /// The files, pictures first (the screen lowest). + pub fn files(&self) -> Vec<(&'static str, &std::path::Path)> { + [ + ("screen", &self.screen), + ("camera", &self.camera), + ("microphone", &self.microphone), + ("system", &self.system), + ] + .into_iter() + .filter_map(|(name, p)| p.as_deref().map(|p| (name, p))) + .collect() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn studio(layout: Layout) -> Studio { + Studio { + scene: Scene { + layout, + ..Scene::default() + }, + ..Studio::default() + } + } + + const CANVAS: (u32, u32) = (1920, 1080); + const SCREEN: Option<(u32, u32)> = Some((3840, 2160)); + const CAMERA: Option<(u32, u32)> = Some((1920, 1080)); + + #[test] + fn the_bubble_is_round_and_sits_in_its_corner() { + let placed = studio(Layout::Bubble).looks(CANVAS, SCREEN, CAMERA); + assert_eq!(placed.screen, Some(Look::default())); + let cam = placed.camera.unwrap(); + let mask = cam.mask.unwrap(); + // Round in pixels: as wide as tall on the camera's own picture. + assert!((mask.width * 1920.0 - mask.height * 1080.0).abs() < 0.5); + // Its height on the canvas is the size asked for. + let shown = mask.height * 1080.0 * cam.scale; + assert!((shown - 0.3 * 1080.0).abs() < 0.5, "{shown}"); + // Bottom right, clear of the edges. + let (cx, cy) = (960.0 + cam.x * 1920.0, 540.0 + cam.y * 1080.0); + assert!(cx + shown / 2.0 < 1920.0 && cx > 1440.0); + assert!(cy + shown / 2.0 < 1080.0 && cy > 700.0); + let left = Studio { + scene: Scene { + corner: Corner::TopLeft, + ..Scene::default() + }, + ..Studio::default() + } + .looks(CANVAS, SCREEN, CAMERA) + .camera + .unwrap(); + assert!(left.x < 0.0 && left.y < 0.0); + } + + #[test] + fn side_by_side_shares_the_width_without_overlap() { + let placed = studio(Layout::Side).looks(CANVAS, SCREEN, CAMERA); + let (screen, cam) = (placed.screen.unwrap(), placed.camera.unwrap()); + // The screen's right edge and the camera's left edge meet. + let screen_w = 1920.0 * screen.scale; + let screen_right = 960.0 + screen.x * 1920.0 + screen_w / 2.0; + let crop = cam.crop.unwrap(); + let cam_w = 1920.0 * cam.scale * (1.0 - crop.left - crop.right); + let cam_left = 960.0 + cam.x * 1920.0 - cam_w / 2.0; + assert!((cam_w - 0.32 * 1920.0).abs() < 1.0, "{cam_w}"); + assert!(screen_right <= cam_left + 1.0, "{screen_right} {cam_left}"); + // The camera fills the height. + assert!((1080.0 * cam.scale * (1.0 - crop.top - crop.bottom) - 1080.0).abs() < 1.0); + } + + #[test] + fn a_missing_picture_leaves_the_other_alone() { + let placed = studio(Layout::Bubble).looks(CANVAS, SCREEN, None); + assert_eq!(placed.screen, Some(Look::default())); + assert!(placed.camera.is_none()); + let placed = studio(Layout::Side).looks(CANVAS, None, Some((1080, 1920))); + assert!(placed.screen.is_none()); + // A tall camera fills a wide canvas. + assert!((placed.camera.unwrap().scale - (1920.0 / 1080.0) / (1080.0 / 1920.0)).abs() < 1e-3); + let placed = studio(Layout::Screen).looks(CANVAS, SCREEN, CAMERA); + assert!(placed.camera.is_none(), "recorded, in the media, not laid out"); + } + + #[test] + fn a_studio_reads_back_what_it_wrote() { + let s = Studio { + screen: Some(Screen::Window { + app: "Safari".into(), + title: "Docs".into(), + }), + camera: Some(Device { + id: "0x1".into(), + name: "C920".into(), + }), + system_audio: true, + ..Studio::default() + }; + let text = serde_json::to_string(&s).unwrap(); + assert_eq!(serde_json::from_str::(&text).unwrap(), s); + assert_eq!(serde_json::from_str::("{}").unwrap(), Studio::default()); + } +} diff --git a/scripts/bundle-macos.sh b/scripts/bundle-macos.sh index 4c096ef..f5de483 100755 --- a/scripts/bundle-macos.sh +++ b/scripts/bundle-macos.sh @@ -66,7 +66,9 @@ cat > "$app/Contents/Info.plist" <NSHumanReadableCopyright © ArchAstro. MIT license. NSMicrophoneUsageDescription - Splice records your microphone with a screen recording when you turn Microphone on. + Splice records the microphone you choose in the Record tab. + NSCameraUsageDescription + Splice records the camera you choose in the Record tab. CFBundleDocumentTypes diff --git a/skill/splice/SKILL.md b/skill/splice/SKILL.md index ddfaeb7..ab47e49 100644 --- a/skill/splice/SKILL.md +++ b/skill/splice/SKILL.md @@ -1,6 +1,6 @@ --- name: splice -description: Work with the user inside their Splice video editor - see the project they have open, place, move, trim, split and delete clips on the timeline, import media, plan the story as beats and cues, record the screen, compose music as a score, make animated scenes and diagrams, make and fill reusable symbols (intros, lower thirds with placeholders), look at any frame, export, and take requests they send from the editor. Also covers installing Splice and this skill. Use when the user asks for help editing a video they have open in Splice, says "my edit/timeline/movie/story/score", asks you to listen for requests from Splice, or asks to install or set up Splice. +description: Work with the user inside their Splice video editor - see the project they have open, place, move, trim, split and delete clips on the timeline, import media, plan the story as beats and cues, record the screen, camera and microphone, compose music as a score, make animated scenes and diagrams, make and fill reusable symbols (intros, lower thirds with placeholders), look at any frame, export, and take requests they send from the editor. Also covers installing Splice and this skill. Use when the user asks for help editing a video they have open in Splice, says "my edit/timeline/movie/story/score", asks you to listen for requests from Splice, or asks to install or set up Splice. --- # Working in the user's Splice editor @@ -472,24 +472,42 @@ and Splice keeps the plan, the references and the takes. For each beat: Check each landed take with `splice ctl frame` before calling it done. -### Recording the screen +### Recording (screen, camera, microphone, the Mac's sound) + +Each project has a recording setup (the Record tab shows it live). A take +writes every source to its own file in one folder, all starting together, +and lands on the timeline as layers placed by the scene. ```sh -splice ctl record sources # displays and windows (ids, apps, titles) -splice ctl record start --window Safari --beat 27 # or --display 1, or --region X,Y,W,H -splice ctl record start --display 1 --for 20 --mic --clicks --at 30 -splice ctl record status -splice ctl record stop --wait # answers once the take is in the project +splice ctl record sources # displays, windows, cameras, microphones, permissions +splice ctl record setup # this project's setup +splice ctl record setup --screen window:Safari --camera C920 --microphone default --system +splice ctl record setup --layout bubble --corner br --size 0.3 --shape circle +splice ctl record setup --codec hevc --quality high --fps 60 --height 1440 --default +splice ctl record start [--for 20] [--beat 27 | --at 30 | --into media] +splice ctl record pause | resume +splice ctl record status # recording, seconds, levels, the last take's folder +splice ctl record stop --wait # answers once the take is in the project ``` -- The take goes into the beat, onto the timeline at `--at`, or (by default) - just into the media panel. +- `--screen` is `display:N`, `window:APP` or `none`; `--region X,Y,W,H` takes + part of the display. `--camera`/`--microphone` take a name (or part of + one), `default` or `none`. `--system`/`--no-system` is the sound the Mac + plays. Layouts: `screen`, `bubble`, `side`, `camera`. `--default` also + makes the setup the one new projects start with. +- `record start` answers once recording has begun (the sources take a moment + to start). The take goes after everything on the timeline unless `--beat`, + `--at` or `--into media` says otherwise. The reply to `stop --wait` lists + its media and clips. The folder (`take` in the reply) holds `screen.mp4`, + `camera.mp4`, `microphone.m4a` and `system.m4a`. +- The layout is only where the take's layers are put: move or resize them + afterwards like any clip (`look`). - Ask before recording, and record only what the person wants: a window is - usually better than the whole screen. `--hide` makes Splice step out of the - shot when recording a whole display. -- The first recording needs Screen Recording permission. If a take comes out - empty, say so and ask the person to grant it in System Settings → Privacy & - Security. + usually better than the whole screen. Splice keeps itself out of the shot. +- Screen Recording, Camera and Microphone permission are each asked for the + first time they are used. If `record status` shows a `problem`, or `record + sources` shows access `denied`, say so and ask the person to turn it on in + System Settings → Privacy & Security. - Splice notes where and when the mouse is clicked while it records (no extra permission). `splice ctl zoom --clip C --to 2` then zooms the take in towards each burst of clicks and back out between them, as keyframes on