From 6a048ba4b2bd745b89c7febe7bb90c41b4713a53 Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Fri, 11 Sep 2026 10:02:37 -0500 Subject: [PATCH 01/13] fix(tasks): T009, T001 - audio tasks carry their sample rate, and loop_audio T009: slice_audio read the source's sample rate and dropped it, so a resample_audio chained off it failed until the rate was restated by hand. Every audio task now returns an AudioTrack - the waveform with the rate it is at - which _waveform_and_rate, pair_audio and the save path all already read. A rate declared on the step or its result still wins. T001: loop_audio makes a bed of a requested length (seconds, or frames at an fps) out of a short recording, laps joined with an equal-power crossfade so the loop point neither clicks nor ticks. It is the missing half of the fix for the hole under a cut: slice tone out of a shot, loop it to the length of the episode, mix_audio it under and pair_audio it back on. Wiring it into dialogue-short waits on a bed measured from real H3 output. Co-Authored-By: Claude Opus 5 --- docs/TASKS.md | 71 ++++++++++++++- dw/tasks/audio_utils.py | 131 +++++++++++++++++++++++---- dw/tasks/task.py | 9 ++ tests/test_audio_utils.py | 173 ++++++++++++++++++++++++++++++------ tests/test_concat_videos.py | 8 +- tests/test_mix_audio.py | 28 ++++-- 6 files changed, 365 insertions(+), 55 deletions(-) diff --git a/docs/TASKS.md b/docs/TASKS.md index 688d513a..462590bb 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -267,14 +267,17 @@ A bleed works because it copies ambience, which has no pitch and no attacks to give the copy away. It is the wrong tool for anything tonal - a copied musical phrase or half-spoken word reads as a stutter whichever direction it runs. When a shot ends on something tonal, either give the cut a continuous bed with -`slice_audio` + `pair_audio`, which leaves no seam to treat at all, or fade the +`slice_audio` + `loop_audio` + `mix_audio` + `pair_audio`, which leaves no seam +to treat at all, or fade the seam gracefully with `seam_fade_ms` (a hundred or so milliseconds) and accept the cut. That advice inverts on a continuous bed - a laugh track, room tone - where a longer fade only digs the hole deeper (the same sitcom cut measured 54-59 dB holes with a 250-500 ms fade and no bleed). `audio_bleed_ms` wins where both are set and there is material to bleed. A bleed covers the gap but cannot fill it: the silence is inside the incoming shot's own head, and the only complete fix is -a continuous bed under the whole cut with `slice_audio` + `pair_audio`. +a continuous bed under the whole cut: `slice_audio` a few seconds of tone out of +a shot, [`loop_audio`](#loop_audio) it to the length of the cut, `mix_audio` it +under the episode and `pair_audio` it back onto the picture. ### dissolve_videos @@ -552,6 +555,64 @@ rescaled - follow it with `normalize_audio` to bring the peak back down. **Example:** [dissolve-between-shots.json](../workflows/templates/dissolve-between-shots.json) — a generated score mixed under the shots' own audio. +### loop_audio + +Make a bed of a given length out of a short recording — the room tone laid +under a whole cut, which is the only complete fix for the hole at a seam. Each +shot in a cut carries its own room and nothing runs underneath the join; +a continuous bed does, the way a location's room tone is laid under a dialogue +scene so the edits stop being audible: + +```json +{ + "task": { + "command": "loop_audio", + "arguments": { + "audio": "previous_result:room_tone", + "target_frames": 620, + "fps": 24, + "crossfade_ms": 250 + } + } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | Path or URL of an audio or video file, a video generated with a soundtrack (which brings its sample rate along), or a waveform | +| `duration_seconds` | One of | How long the bed should be, in seconds | +| `target_frames` / `fps` | One of | How long the bed should be, in video frames — how a bed is matched to a cut exactly | +| `crossfade_ms` | No | Crossfade at each loop point, clamped to the material available (default 250) | +| `sample_rate` | With a waveform | Sample rate of a waveform passed directly; given for a file or a video it overrides the rate they carry | + +Laps are joined with an equal-power crossfade rather than butted together, so +the loop point is not a click and a tone with movement in it does not tick once +a second. The source is used whole every lap and only the last one is trimmed, +so the bed lands exactly on the requested length; a source longer than the +request is trimmed to it. + +The bed is laid under the cut with `mix_audio` and attached to the picture with +`pair_audio`: + +```json +{ "name": "bed", "task": { "command": "loop_audio", + "arguments": { "audio": "previous_result:room_tone", + "target_frames": 620, "fps": 24 } } }, +{ "name": "mixed", "task": { "command": "mix_audio", + "arguments": { "audios": ["previous_result:episode", + "previous_result:bed"], + "gains": [1.0, 0.25] } } }, +{ "name": "cut", "task": { "command": "pair_audio", + "arguments": { "video": "previous_result:episode", + "audio": "previous_result:mixed" } }, + "result": { "content_type": "video/mp4", "fps": 24 } } +``` + +Where the bed itself comes from is the open question: a few seconds of a +generated shot's own ambience, cut out with `slice_audio` from a stretch with +nothing tonal in it, is the material that matches — the room the shots were +generated in. + ### resample_audio Convert a track to a different sample rate. A pipeline that conditions on audio @@ -579,6 +640,12 @@ supplied recording once, up front, feeds it what it already wants: A track already at the target rate is returned untouched. The conversion is PyAV's, which dw already needs for video - no torchaudio dependency. +Every audio task returns the waveform *and* the rate it is at, so one chains +into the next without the rate being restated: a `resample_audio` fed +`previous_result:` from a `slice_audio` takes the source rate from the slice. A +`sample_rate` given on the step still wins, and one declared on the step's +`result` still decides what is written to disk. + **Example:** [assemble-and-score.json](../workflows/templates/assemble-and-score.json) ## Data Gathering diff --git a/dw/tasks/audio_utils.py b/dw/tasks/audio_utils.py index e82ccf3a..946317e5 100644 --- a/dw/tasks/audio_utils.py +++ b/dw/tasks/audio_utils.py @@ -235,6 +235,22 @@ def load_audio(location, base_dir=None): return as_channels_samples(data), sample_rate +def _as_track(waveform, sample_rate): + """An audio task's return value: the waveform with the rate it is at. + + Every one of these commands already knows the rate - it was given, or it + came off the file or the video the track was taken from - and dropping it + on the way out made the next command in the chain ask for it again. A + resample fed straight from a slice failed for want of a number the slice + had read and thrown away (2026-09-11). An AudioTrack carries it, and + everything downstream of audio reads '.audio'/'.sample_rate' already; a + 'sample_rate' the workflow declares on the result still wins at save. + """ + from ..result import AudioTrack + + return AudioTrack(numpy.ascontiguousarray(waveform), int(sample_rate)) + + def _as_number(value, kind, name): """Coerce a numeric slice argument given as a string, leaving None alone.""" if not isinstance(value, str): @@ -276,8 +292,8 @@ def slice_audio( file or a video it overrides the rate they carry Returns: - The slice as a (samples, channels) float32 array - the layout audio - results are saved in + An AudioTrack holding the slice and the rate it is at, so the next + audio command in the chain does not have to be told the rate again """ # A variable a workflow declares null carries no type, so a value given for # it on the command line arrives as a string - the same coercion the upscale @@ -313,7 +329,7 @@ def slice_audio( "'start_frame'/'num_frames'/'fps'" ) - return slice_samples(waveform, start, length).T + return _as_track(slice_samples(waveform, start, length), sample_rate) def resample_audio(audio, target_sample_rate, sample_rate=None): @@ -335,12 +351,12 @@ def resample_audio(audio, target_sample_rate, sample_rate=None): file or a video it overrides the rate they carry Returns: - The resampled track as a (samples, channels) float32 array + An AudioTrack holding the resampled waveform and its new rate """ waveform, sample_rate = _waveform_and_rate(audio, sample_rate, "resample_audio") if sample_rate == target_sample_rate: - return waveform.T + return _as_track(waveform, sample_rate) import av from av.audio.resampler import AudioResampler @@ -363,7 +379,9 @@ def resample_audio(audio, target_sample_rate, sample_rate=None): f"Resampled {waveform.shape[1]} samples at {sample_rate}Hz " f"to {target_sample_rate}Hz" ) - return numpy.concatenate(converted, axis=1).astype(numpy.float32).T + return _as_track( + numpy.concatenate(converted, axis=1).astype(numpy.float32), target_sample_rate + ) def crossfade_audio(audios, crossfade_ms=75, sample_rate=None): @@ -380,7 +398,7 @@ def crossfade_audio(audios, crossfade_ms=75, sample_rate=None): brings its own; given here it wins Returns: - The joined track as a (samples, channels) float32 array + An AudioTrack holding the joined waveform and its rate """ if not isinstance(audios, list) or not audios: raise ValueError("crossfade_audio needs a non-empty list of audio tracks") @@ -400,7 +418,9 @@ def crossfade_audio(audios, crossfade_ms=75, sample_rate=None): f"crossfade_audio needs one sample rate, got {sorted(rates)}" ) sample_rate = rates.pop() - return crossfade_concat(waveforms, sample_rate, crossfade_ms).T + return _as_track( + crossfade_concat(waveforms, sample_rate, crossfade_ms), sample_rate + ) def mix_audio(audios, gains=None, sample_rate=None): @@ -427,7 +447,7 @@ def mix_audio(audios, gains=None, sample_rate=None): brings its own; given here it wins Returns: - The mixed track as a (samples, channels) float32 array + An AudioTrack holding the mixed waveform and its rate """ if not isinstance(audios, list) or not audios: raise ValueError("mix_audio needs a non-empty list of audio tracks") @@ -460,7 +480,86 @@ def mix_audio(audios, gains=None, sample_rate=None): for index, waveform in enumerate(waveforms): gain = 1.0 if gains is None else float(gains[index]) mixed[:, : waveform.shape[1]] += waveform * gain - return mixed.T + return _as_track(mixed, sample_rate) + + +def loop_audio( + audio, + duration_seconds=None, + target_frames=None, + fps=None, + crossfade_ms=250, + sample_rate=None, +): + """Task command: make a bed of a given length out of a short recording. + + A cut between two independently generated shots has a hole in it: each + shot carries its own room, and nothing runs underneath the seam. A + continuous bed laid under the whole cut is what fills it - the way a + location's room tone is laid under a dialogue scene so the edits stop + being audible - and a bed is made by looping a few seconds of tone to + the length of the picture. + + Laps are joined with an equal-power crossfade rather than butted + together, so the loop point is not a click and a tone with any movement + in it does not tick once a second. The source is used whole every lap; + only the last one is trimmed, to land exactly on the requested length. A + source longer than the request is trimmed to it. + + Args: + audio: Path or URL of an audio file (or of a video file, whose + soundtrack is taken), a video generated with a soundtrack, or a + waveform (which needs sample_rate alongside it) + duration_seconds: How long the bed should be, in seconds + target_frames: How long the bed should be, in video frames - needs + 'fps', and is how a bed is matched to a cut exactly + fps: Frame rate 'target_frames' is counted at + crossfade_ms: Length of the crossfade at each loop point, clamped to + the material available + sample_rate: Sample rate of a waveform passed directly; given for a + file or a video it overrides the rate they carry + + Returns: + An AudioTrack holding the bed and the rate it is at + """ + duration_seconds = _as_number(duration_seconds, float, "duration_seconds") + target_frames = _as_number(target_frames, int, "target_frames") + fps = _as_number(fps, Fraction, "fps") + crossfade_ms = _as_number(crossfade_ms, float, "crossfade_ms") + + waveform, sample_rate = _waveform_and_rate(audio, sample_rate, "loop_audio") + if waveform.size == 0: + raise ValueError("loop_audio needs a source with samples in it") + + if duration_seconds is not None: + length = int(round(duration_seconds * sample_rate)) + elif target_frames is not None: + if fps is None: + raise ValueError("loop_audio needs 'fps' to count a length in frames") + length = frames_to_samples(target_frames, fps, sample_rate) + else: + raise ValueError( + "loop_audio needs either 'duration_seconds' or 'target_frames'/'fps' " + "to know how long a bed to make" + ) + if length <= 0: + raise ValueError(f"loop_audio needs a length above zero, got {length} samples") + if crossfade_ms < 0: + raise ValueError("loop_audio 'crossfade_ms' cannot be negative") + + window = min(int(crossfade_ms / 1000.0 * sample_rate), waveform.shape[1] // 2) + bed = waveform + # Each lap after the first overlaps the one before it by the crossfade, so + # a lap adds (source - window) samples rather than a whole source + while bed.shape[1] < length: + bed = crossfade_concat( + [bed, waveform], sample_rate, window / sample_rate * 1000.0 + ) + logger.debug( + f"loop_audio: {waveform.shape[1]} samples at {sample_rate}Hz looped to " + f"{length} ({bed.shape[1]} before trimming)" + ) + return _as_track(bed[:, :length], sample_rate) def _equal_power_ramps(window): @@ -517,7 +616,7 @@ def fade_audio(audio, fade_in_ms=0, fade_out_ms=0, sample_rate=None): sample_rate: Sample rate of a waveform passed directly Returns: - The faded track as a (samples, channels) float32 array + An AudioTrack holding the faded waveform and its rate """ waveform, sample_rate = _waveform_and_rate(audio, sample_rate, "fade_audio") if fade_in_ms < 0 or fade_out_ms < 0: @@ -531,7 +630,7 @@ def fade_audio(audio, fade_in_ms=0, fade_out_ms=0, sample_rate=None): fade_out = min(int(round(fade_out_ms / 1000 * sample_rate)), length) if fade_out: faded[:, length - fade_out :] *= _fade_curve(fade_out) - return faded.T + return _as_track(faded, sample_rate) def normalize_audio(audio, peak_dbfs=-1.0, sample_rate=None): @@ -551,21 +650,21 @@ def normalize_audio(audio, peak_dbfs=-1.0, sample_rate=None): sample_rate: Sample rate of a waveform passed directly Returns: - The scaled track as a (samples, channels) float32 array; a silent + An AudioTrack holding the scaled waveform and its rate; a silent track is returned unchanged """ - waveform, _ = _waveform_and_rate(audio, sample_rate, "normalize_audio") + waveform, sample_rate = _waveform_and_rate(audio, sample_rate, "normalize_audio") if peak_dbfs > 0: raise ValueError("normalize_audio 'peak_dbfs' cannot be above full scale (0)") peak = float(numpy.abs(waveform).max()) if waveform.size else 0.0 if peak == 0.0: logger.warning("normalize_audio: the track is silent - left unchanged") - return waveform.T + return _as_track(waveform, sample_rate) gain = 10 ** (peak_dbfs / 20) / peak logger.debug( f"normalize_audio: peak {peak:.3f}, gain {20 * numpy.log10(gain):+.1f} dB" ) - return (waveform * gain).astype(numpy.float32).T + return _as_track((waveform * gain).astype(numpy.float32), sample_rate) def _fade_curve(window): diff --git a/dw/tasks/task.py b/dw/tasks/task.py index 6467c4e1..83d8f270 100644 --- a/dw/tasks/task.py +++ b/dw/tasks/task.py @@ -198,6 +198,15 @@ def _handle_crossfade_audio(task, arguments, previous_pipelines): return crossfade_audio(**arguments) +@register_command("loop_audio", implementation="dw.tasks.audio_utils.loop_audio") +def _handle_loop_audio(task, arguments, previous_pipelines): + """Loop a short recording into a bed of a given length""" + logger.debug("Looping audio") + from .audio_utils import loop_audio + + return loop_audio(**arguments) + + @register_command( "stabilize_video", implementation="dw.tasks.stabilize.stabilize_video" ) diff --git a/tests/test_audio_utils.py b/tests/test_audio_utils.py index 15134d72..997c1b87 100644 --- a/tests/test_audio_utils.py +++ b/tests/test_audio_utils.py @@ -19,6 +19,15 @@ ) +def samples(track): + """The (samples, channels) waveform an audio task's AudioTrack carries. + + The tasks return an AudioTrack so the rate travels with the waveform; + these shape assertions are written in the save layout, so they unwrap it. + """ + return numpy.asarray(track.audio).T + + class TestAsChannelsSamples: def test_a_mono_vector_gains_a_channel_axis(self): assert as_channels_samples(numpy.zeros(100)).shape == (1, 100) @@ -244,12 +253,15 @@ def test_every_audio_task_takes_a_video_path(self, tmp_path): path = self._write_video(tmp_path / "cut.mp4") - assert resample_audio(path, 4000).shape[1] == 2 - assert slice_audio(path, start_seconds=0, duration_seconds=1).shape[0] == 8000 - assert fade_audio(path, fade_out_ms=100).shape[1] == 2 - assert normalize_audio(path).shape[1] == 2 - assert mix_audio([path, path]).shape[1] == 2 - assert crossfade_audio([path, path], crossfade_ms=10).shape[1] == 2 + assert samples(resample_audio(path, 4000)).shape[1] == 2 + assert ( + samples(slice_audio(path, start_seconds=0, duration_seconds=1)).shape[0] + == 8000 + ) + assert samples(fade_audio(path, fade_out_ms=100)).shape[1] == 2 + assert samples(normalize_audio(path)).shape[1] == 2 + assert samples(mix_audio([path, path])).shape[1] == 2 + assert samples(crossfade_audio([path, path], crossfade_ms=10)).shape[1] == 2 class TestBleedJoin: @@ -319,27 +331,32 @@ def test_it_scales_the_length_to_the_new_rate(self): result = resample_audio(waveform, 32000, sample_rate=44100) - assert result.shape == (32000, 2) + assert samples(result).shape == (32000, 2) + assert result.sample_rate == 32000 def test_it_returns_samples_by_channels(self): waveform = numpy.zeros((1, 44100), dtype=numpy.float32) - assert resample_audio(waveform, 22050, sample_rate=44100).shape == (22050, 1) + assert samples(resample_audio(waveform, 22050, sample_rate=44100)).shape == ( + 22050, + 1, + ) def test_matching_rates_pass_through_untouched(self): waveform = numpy.linspace(-1, 1, 1000, dtype=numpy.float32)[None, :] result = resample_audio(waveform, 8000, sample_rate=8000) - assert result.shape == (1000, 1) - assert numpy.allclose(result[:, 0], waveform[0]) + assert samples(result).shape == (1000, 1) + assert result.sample_rate == 8000 + assert numpy.allclose(samples(result)[:, 0], waveform[0]) def test_a_tone_keeps_its_level_and_duration(self): rate, seconds = 44100, 0.5 t = numpy.arange(int(rate * seconds)) / rate tone = numpy.sin(2 * numpy.pi * 440 * t).astype(numpy.float32)[None, :] - result = resample_audio(tone, 32000, sample_rate=44100) + result = samples(resample_audio(tone, 32000, sample_rate=44100)) assert result.shape[0] == pytest.approx(32000 * seconds, rel=0.01) level = float(numpy.sqrt((result[:, 0] ** 2).mean())) @@ -354,7 +371,10 @@ def test_a_raw_waveform_needs_its_rate(self): def test_it_accepts_a_torch_waveform(self): waveform = torch.zeros(2, 44100) - assert resample_audio(waveform, 32000, sample_rate=44100).shape == (32000, 2) + assert samples(resample_audio(waveform, 32000, sample_rate=44100)).shape == ( + 32000, + 2, + ) class TestFadeAudio: @@ -363,7 +383,9 @@ def test_fades_end_on_silence_and_leave_the_middle_alone(self): track = numpy.ones((2, 1000), dtype=numpy.float32) - faded = fade_audio(track, fade_in_ms=100, fade_out_ms=200, sample_rate=1000) + faded = samples( + fade_audio(track, fade_in_ms=100, fade_out_ms=200, sample_rate=1000) + ) assert faded.shape == (1000, 2) assert faded[0, 0] == pytest.approx(0.0, abs=1e-6) @@ -385,7 +407,9 @@ def test_the_input_is_not_modified(self): def test_a_fade_longer_than_the_track_is_clamped(self): from dw.tasks.audio_utils import fade_audio - faded = fade_audio(numpy.ones((1, 10)), fade_in_ms=5000, sample_rate=100) + faded = samples( + fade_audio(numpy.ones((1, 10)), fade_in_ms=5000, sample_rate=100) + ) assert faded.shape == (10, 1) @@ -408,7 +432,7 @@ def test_the_peak_lands_on_the_target(self): track = numpy.array([[0.1, -0.25, 0.05]], dtype=numpy.float32) - scaled = normalize_audio(track, peak_dbfs=-6.0, sample_rate=100) + scaled = samples(normalize_audio(track, peak_dbfs=-6.0, sample_rate=100)) assert scaled.shape == (3, 1) assert numpy.abs(scaled).max() == pytest.approx(10 ** (-6 / 20), abs=1e-6) @@ -420,7 +444,9 @@ def test_the_peak_lands_on_the_target(self): def test_silence_is_left_alone(self): from dw.tasks.audio_utils import normalize_audio - assert numpy.all(normalize_audio(numpy.zeros((1, 10)), sample_rate=100) == 0) + assert numpy.all( + samples(normalize_audio(numpy.zeros((1, 10)), sample_rate=100)) == 0 + ) def test_a_target_above_full_scale_is_refused(self): from dw.tasks.audio_utils import normalize_audio @@ -445,22 +471,34 @@ def test_slice_audio_takes_the_rate_from_the_video(self): sliced = slice_audio(self.video(), start_seconds=1.0, duration_seconds=2.0) - assert sliced.shape == (200, 2) + assert samples(sliced).shape == (200, 2) + # The rate it read travels with the slice, so a resample chained off it + # does not have to be told the rate again + assert sliced.sample_rate == 100 def test_no_duration_slices_to_the_end_of_the_track(self): """A workflow that trims only when given a length still yields the track - the missing half of the pair means 'to the end', not an error.""" from dw.tasks.audio_utils import slice_audio - assert slice_audio(self.video(), start_seconds=1.0).shape == (300, 2) - assert slice_audio(self.video(), start_seconds=0).shape == (400, 2) + assert samples(slice_audio(self.video(), start_seconds=1.0)).shape == (300, 2) + assert samples(slice_audio(self.video(), start_seconds=0)).shape == (400, 2) def test_no_start_slices_from_the_head_of_the_track(self): from dw.tasks.audio_utils import slice_audio - assert slice_audio(self.video(), duration_seconds=2.0).shape == (200, 2) - assert slice_audio(self.video(), start_frame=12, fps=24).shape == (350, 2) - assert slice_audio(self.video(), num_frames=24, fps=24).shape == (100, 2) + assert samples(slice_audio(self.video(), duration_seconds=2.0)).shape == ( + 200, + 2, + ) + assert samples(slice_audio(self.video(), start_frame=12, fps=24)).shape == ( + 350, + 2, + ) + assert samples(slice_audio(self.video(), num_frames=24, fps=24)).shape == ( + 100, + 2, + ) def test_a_slice_in_frames_still_needs_the_frame_rate(self): from dw.tasks.audio_utils import slice_audio @@ -481,20 +519,24 @@ def test_a_given_rate_overrides_the_video_rate(self): self.video(), start_seconds=0.0, duration_seconds=1.0, sample_rate=50 ) - assert sliced.shape == (50, 2) + assert samples(sliced).shape == (50, 2) + assert sliced.sample_rate == 50 def test_resample_audio_takes_the_rate_from_the_video(self): from dw.tasks.audio_utils import resample_audio - assert resample_audio(self.video(), target_sample_rate=100).shape == (400, 2) + assert samples(resample_audio(self.video(), target_sample_rate=100)).shape == ( + 400, + 2, + ) def test_fade_and_normalize_take_a_video(self): from dw.tasks.audio_utils import fade_audio, normalize_audio - faded = fade_audio(self.video(), fade_out_ms=1000) + faded = samples(fade_audio(self.video(), fade_out_ms=1000)) assert faded.shape == (400, 2) and faded[-1, 0] == pytest.approx(0.0, abs=1e-6) - scaled = normalize_audio(self.video(level=0.5), peak_dbfs=0.0) + scaled = samples(normalize_audio(self.video(level=0.5), peak_dbfs=0.0)) assert numpy.abs(scaled).max() == pytest.approx(1.0) def test_crossfade_audio_joins_videos_at_their_own_rate(self): @@ -503,7 +545,8 @@ def test_crossfade_audio_joins_videos_at_their_own_rate(self): joined = crossfade_audio([self.video(), self.video()], crossfade_ms=1000) # 4 s + 4 s - 1 s overlap = 7 s at 100 Hz - assert joined.shape == (700, 2) + assert samples(joined).shape == (700, 2) + assert joined.sample_rate == 100 def test_crossfade_audio_refuses_mixed_rates(self): from dw.tasks.audio_utils import crossfade_audio @@ -523,3 +566,79 @@ def test_a_silent_video_is_refused(self): with pytest.raises(ValueError, match="carries none"): fade_audio(AudioVideo([], None, None), fade_in_ms=10) + + +class TestLoopAudio: + """A bed made from a short recording - the room tone laid under a cut so + the seam between two independently generated shots is not a hole.""" + + def tone(self, samples=100, rate=100, level=0.5, channels=1): + return numpy.full((channels, samples), level, dtype=numpy.float32) + + def test_it_makes_a_bed_of_the_requested_seconds(self): + from dw.tasks.audio_utils import loop_audio + + bed = loop_audio(self.tone(), duration_seconds=3.5, sample_rate=100) + + assert samples(bed).shape == (350, 1) + assert bed.sample_rate == 100 + + def test_a_length_in_frames_matches_a_cut_exactly(self): + from dw.tasks.audio_utils import loop_audio + + bed = loop_audio(self.tone(), target_frames=48, fps=24, sample_rate=100) + + assert samples(bed).shape == (200, 1) + + def test_a_source_longer_than_the_bed_is_trimmed(self): + from dw.tasks.audio_utils import loop_audio + + bed = loop_audio(self.tone(samples=400), duration_seconds=1.0, sample_rate=100) + + assert samples(bed).shape == (100, 1) + + def test_the_loop_point_is_crossfaded_rather_than_butted(self): + from dw.tasks.audio_utils import loop_audio + + # A ramp ends far from where it starts, so a butt join would step; + # the crossfade has to leave the seam continuous + ramp = numpy.linspace(-1, 1, 200, dtype=numpy.float32)[None, :] + bed = samples( + loop_audio(ramp, duration_seconds=4.0, crossfade_ms=500, sample_rate=100) + ) + + steps = numpy.abs(numpy.diff(bed[:, 0])) + assert steps.max() < 0.1 + + def test_it_takes_the_rate_from_a_generated_video(self): + from dw.result import AudioVideo + from dw.tasks.audio_utils import loop_audio + + video = AudioVideo([], numpy.zeros((2, 400), dtype=numpy.float32), 100) + + bed = loop_audio(video, duration_seconds=10.0) + + assert samples(bed).shape == (1000, 2) + assert bed.sample_rate == 100 + + def test_it_needs_to_be_told_how_long_a_bed_to_make(self): + from dw.tasks.audio_utils import loop_audio + + with pytest.raises(ValueError, match="how long a bed"): + loop_audio(self.tone(), sample_rate=100) + + def test_a_length_in_frames_still_needs_the_frame_rate(self): + from dw.tasks.audio_utils import loop_audio + + with pytest.raises(ValueError, match="fps"): + loop_audio(self.tone(), target_frames=48, sample_rate=100) + + def test_an_empty_source_is_refused(self): + from dw.tasks.audio_utils import loop_audio + + with pytest.raises(ValueError, match="samples in it"): + loop_audio( + numpy.zeros((1, 0), dtype=numpy.float32), + duration_seconds=1.0, + sample_rate=100, + ) diff --git a/tests/test_concat_videos.py b/tests/test_concat_videos.py index 94623b3f..9fdb49f5 100644 --- a/tests/test_concat_videos.py +++ b/tests/test_concat_videos.py @@ -123,7 +123,8 @@ def test_slice_audio_runs_through_task(self): } ) - assert result.shape == (200, 2) + assert result.audio.shape == (2, 200) + assert result.sample_rate == 100 def test_crossfade_audio_runs_through_task(self): task = Task({"command": "crossfade_audio", "arguments": {}}, "cpu") @@ -139,7 +140,7 @@ def test_crossfade_audio_runs_through_task(self): } ) - assert result.shape == (390, 2) + assert result.audio.shape == (2, 390) class TestAudioBleed: @@ -275,7 +276,8 @@ def test_resample_audio_runs_through_task(self): } ) - assert result.shape == (32000, 2) + assert result.audio.shape == (2, 32000) + assert result.sample_rate == 32000 class TestVideoFiles: diff --git a/tests/test_mix_audio.py b/tests/test_mix_audio.py index 8747cca0..e9c6979e 100644 --- a/tests/test_mix_audio.py +++ b/tests/test_mix_audio.py @@ -10,34 +10,48 @@ def _tone(samples, level, channels=2): return numpy.full((channels, samples), level, dtype=numpy.float32) +def _samples(track): + """The (samples, channels) waveform mix_audio's AudioTrack carries.""" + return numpy.asarray(track.audio).T + + class TestMixAudio: def test_tracks_are_summed(self): - mixed = mix_audio([_tone(100, 0.2), _tone(100, 0.3)], sample_rate=44100) + mixed = _samples( + mix_audio([_tone(100, 0.2), _tone(100, 0.3)], sample_rate=44100) + ) assert mixed.shape == (100, 2) assert mixed[0, 0] == pytest.approx(0.5) def test_gains_are_plain_multipliers(self): - mixed = mix_audio( - [_tone(100, 0.4), _tone(100, 0.4)], gains=[1.0, 0.5], sample_rate=44100 + mixed = _samples( + mix_audio( + [_tone(100, 0.4), _tone(100, 0.4)], gains=[1.0, 0.5], sample_rate=44100 + ) ) assert mixed[0, 0] == pytest.approx(0.6) def test_the_shorter_track_is_padded_with_silence(self): - mixed = mix_audio([_tone(100, 0.5), _tone(40, 0.25)], sample_rate=44100) + mixed = _samples( + mix_audio([_tone(100, 0.5), _tone(40, 0.25)], sample_rate=44100) + ) assert mixed.shape == (100, 2) assert mixed[0, 0] == pytest.approx(0.75) assert mixed[-1, 0] == pytest.approx(0.5) def test_mono_is_tiled_up_to_the_widest_track(self): - mixed = mix_audio( - [_tone(50, 0.5, channels=1), _tone(50, 0.25, channels=2)], sample_rate=44100 + mixed = _samples( + mix_audio( + [_tone(50, 0.5, channels=1), _tone(50, 0.25, channels=2)], + sample_rate=44100, + ) ) assert mixed.shape == (50, 2) assert mixed[0, 0] == pytest.approx(0.75) assert mixed[0, 1] == pytest.approx(0.75) def test_the_sum_is_not_rescaled(self): - mixed = mix_audio([_tone(10, 0.8), _tone(10, 0.8)], sample_rate=44100) + mixed = _samples(mix_audio([_tone(10, 0.8), _tone(10, 0.8)], sample_rate=44100)) assert mixed.max() == pytest.approx(1.6) def test_an_empty_list_is_refused(self): From 28afbcf300a450928b897aa953f1c25c1e71fb2d Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Fri, 11 Sep 2026 10:05:11 -0500 Subject: [PATCH 02/13] feat(media): T004 - opt-in per-second level envelope from probe_media probe_media(path, envelope=True) reports the soundtrack's level second by second - rms_dbfs and peak_dbfs, one entry each per second - from the decode pass it already makes. That is what locates something in a track: whether a shot is still sounding at its last frame, how deep the hole at a seam goes, where a score goes quiet. Off by default, since a ten-minute track is 600 numbers and the default metadata call has to stay small. Carried by GET /api/gallery/{name}/metadata?envelope=true and MCP get_gallery_metadata(name, envelope=True). Co-Authored-By: Claude Opus 5 --- docs/MCP.md | 2 +- dw/media_info.py | 86 +++++++++++++++++++++++++++++++++++++++- dw/server/app.py | 16 ++++++-- dw_mcp/catalog.py | 14 +++++-- dw_mcp/server.py | 12 ++++-- tests/test_media_info.py | 74 ++++++++++++++++++++++++++++++++++ tests/test_server.py | 23 +++++++++++ 7 files changed, 216 insertions(+), 11 deletions(-) diff --git a/docs/MCP.md b/docs/MCP.md index fb9946b1..5c7c973f 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -227,7 +227,7 @@ when no single workflow covers it. | `get_server_info()` | — | What this installation can do and where it keeps things: `device` (the accelerator a run will use), `version`, the `workspace` this session is working in and the workflow/asset/output/prompt `directories` of *that* workspace, the bind address and port, whether a token is required, and whether MCP is mounted. Check the device before authoring - a CUDA-only choice (bitsandbytes, `torch.compile`, flash attention) is not available on an `mps` or `cpu` server | | `list_jobs()` | — | List queued, running and recent jobs. In a named workspace, that workspace's jobs; in the default one, every job the server holds | | `list_gallery(limit=50)` | `limit` | List generated output files, newest first. A name is `//`; each entry also carries a ready-made `url`, already scoped to the workspace that made it - a hand-built `/outputs/` URL 404s for anything but the default workspace | -| `get_gallery_metadata(name)` | `name` | Get the metadata embedded in a generated file: the exact workflow and arguments that produced it, and, for audio/video, a `media` block (duration, rate, channels, fps, size, peak/mean dBFS) | +| `get_gallery_metadata(name, envelope=False)` | `name` | Get the metadata embedded in a generated file: the exact workflow and arguments that produced it, and, for audio/video, a `media` block (duration, rate, channels, fps, size, peak/mean dBFS). `envelope=true` adds `media.envelope` — `rms_dbfs` and `peak_dbfs` one entry per second — which is what locates something in a track rather than measuring the whole of it | ### Media diff --git a/dw/media_info.py b/dw/media_info.py index ebeeb67f..b8b5b087 100644 --- a/dw/media_info.py +++ b/dw/media_info.py @@ -23,7 +23,7 @@ def _dbfs(value): return max(SILENCE_DBFS, 20.0 * math.log10(float(value))) -def probe_media(path): +def probe_media(path, envelope=False): """Duration, format and level of an audio or video file, or None. Video answers fps, frame_count, width and height, plus the soundtrack's @@ -36,6 +36,22 @@ def probe_media(path): whichever streams are involved - `container.decode()` demuxes to EOF, so two separate passes (count video, then decode audio) would leave the second one nothing to read. + + With `envelope=True` the same decode also reports the level second by + second, as `envelope: {"interval_seconds": 1.0, "rms_dbfs": [...], + "peak_dbfs": [...]}` - which is what tells an agent *where* in a track + something is, rather than only how loud the whole thing was: whether a + shot is still voiced at its last frame, where a score's quiet passage + sits, how deep the hole at a seam goes. Off by default, because a + ten-minute track is 600 numbers nobody asked for and the default + metadata call has to stay small. + + Args: + path: The file to probe + envelope: Also report the per-second level of the soundtrack. The + list covers what decodes, which for a lossy codec can run a + fraction of a second past the reported duration - its own + priming and padding """ try: container = av.open(path) @@ -73,6 +89,11 @@ def probe_media(path): peak = 0.0 total = 0.0 count = 0 + # One bin per second of the soundtrack, filled as frames decode: + # [sum of squares, sample count, peak] - the same numbers the + # whole-track level is made of, kept per second instead of once + bins = [] if envelope and audio is not None else None + elapsed = 0 # samples of the soundtrack seen so far streams = [ s for s in ((video if need_frame_count else None), audio) @@ -97,6 +118,10 @@ def probe_media(path): peak = max(peak, float(numpy.abs(samples).max(initial=0.0))) total += float(numpy.square(samples).sum()) count += samples.size + if bins is not None: + elapsed = _fill_envelope( + bins, samples, elapsed, audio.rate, int(audio.channels) + ) except Exception as e: # A track that opens fine can still fail mid-decode (damage # past the header); the fields already gathered - duration, @@ -111,4 +136,63 @@ def probe_media(path): rms = math.sqrt(total / count) if count else 0.0 info["peak_dbfs"] = _dbfs(peak) info["mean_dbfs"] = _dbfs(rms) + if bins is not None: + info["envelope"] = _as_envelope(bins) return info + + +def _as_frame_samples(samples, channels): + """One decoded audio frame as a (samples, channels) array. + + A planar format decodes to (channels, samples); a packed one decodes to + (1, samples * channels) interleaved. Both have to become a run of + samples before they can be cut on a second boundary, or a stereo packed + frame would be counted as twice as much time as it holds. + """ + if samples.ndim == 1: + return samples[:, numpy.newaxis] + if samples.shape[0] == channels and channels > 1: + return samples.T + if samples.shape[0] == 1 and channels > 1: + return samples.reshape(-1, channels) + return samples.T if samples.shape[0] < samples.shape[1] else samples + + +def _fill_envelope(bins, samples, elapsed, rate, channels): + """Add a decoded audio frame's samples to the per-second bins. + + A bin covers one second of the track regardless of how the decoder + happened to chop it, so a frame straddling a second boundary is split + across the two bins rather than counted in whichever one it started in. + `elapsed` is how many samples of the track came before this frame; the + new total is returned. + """ + frame = _as_frame_samples(samples, channels) + length = frame.shape[0] + start = 0 + while start < length: + second = (elapsed + start) // rate + while len(bins) <= second: + bins.append([0.0, 0, 0.0]) + # How much of this frame still belongs to the second it is in + room = int((second + 1) * rate - (elapsed + start)) + stop = min(length, start + max(room, 1)) + piece = frame[start:stop] + entry = bins[second] + entry[0] += float(numpy.square(piece).sum()) + entry[1] += int(piece.size) + entry[2] = max(entry[2], float(numpy.abs(piece).max(initial=0.0))) + start = stop + return elapsed + length + + +def _as_envelope(bins): + """The per-second bins as the levels an agent reads.""" + return { + "interval_seconds": 1.0, + "rms_dbfs": [ + _dbfs(math.sqrt(total / count) if count else 0.0) + for total, count, _peak in bins + ], + "peak_dbfs": [_dbfs(peak) for _total, _count, peak in bins], + } diff --git a/dw/server/app.py b/dw/server/app.py index 195936d6..7dfbb0a6 100644 --- a/dw/server/app.py +++ b/dw/server/app.py @@ -1819,12 +1819,22 @@ def gallery( } @app.get("/api/gallery/{name:path}/metadata") - def gallery_metadata(name: str, ws: Workspace = Depends(selected_workspace)): + def gallery_metadata( + name: str, + envelope: bool = False, + ws: Workspace = Depends(selected_workspace), + ): """Generation metadata embedded in a saved image ('workflow' inside it is the full definition the editor can reopen), plus the job that produced the file when history remembers one, plus - for audio and video - what the file itself holds: duration, format and level, - which is how an agent that cannot listen checks a track.""" + which is how an agent that cannot listen checks a track. + + `envelope=true` adds the soundtrack's level second by second, which + is what says *where* in a track something is - whether a shot is + still voiced at its last frame, how deep the hole at a seam goes. + Opt-in: a ten-minute track is 600 numbers, and the default call has + to stay small.""" path = _output_file(name, ws.outputs) metadata = read_embedded_metadata(path) try: @@ -1836,7 +1846,7 @@ def gallery_metadata(name: str, ws: Workspace = Depends(selected_workspace)): job = None extension = os.path.splitext(path)[1].lower() media = ( - probe_media(path) + probe_media(path, envelope=envelope) if MEDIA_KINDS.get(extension) in ("audio", "video") else None ) diff --git a/dw_mcp/catalog.py b/dw_mcp/catalog.py index 2a48d1f9..4d13301a 100644 --- a/dw_mcp/catalog.py +++ b/dw_mcp/catalog.py @@ -101,12 +101,20 @@ def list_gallery(client, limit=50): return client.get_json("/api/gallery", params={"limit": limit}) -def get_gallery_metadata(client, name): +def get_gallery_metadata(client, name, envelope=False): """Metadata embedded in a saved file: the full workflow that made it, plus the job that produced it when history remembers one, plus for audio and video what the file holds - duration, sample rate, channels, - fps, size, peak and mean level in dBFS.""" - body = client.get_json(api_path("api", "gallery", name, "metadata")) + fps, size, peak and mean level in dBFS. + + With envelope=True the soundtrack's level is reported second by second + as well, which is what locates something in a track rather than only + measuring the whole of it. Opt-in: it is one number per second per + measure, and the default answer has to stay small.""" + body = client.get_json( + api_path("api", "gallery", name, "metadata"), + params={"envelope": "true"} if envelope else None, + ) media = body.get("media") if media and media.get("kind") in ("audio", "video"): body["next"] = ( diff --git a/dw_mcp/server.py b/dw_mcp/server.py index 49ceb286..cfc3ed2b 100644 --- a/dw_mcp/server.py +++ b/dw_mcp/server.py @@ -284,15 +284,21 @@ def list_gallery(limit: int = 50) -> dict: name.""" return catalog.list_gallery(client, limit=limit) - def get_gallery_metadata(name: str) -> dict: + def get_gallery_metadata(name: str, envelope: bool = False) -> dict: """Get the metadata embedded in a generated file: the exact workflow, arguments and seed that produced it. Use this to reproduce a result, or to see what a run that went wrong actually ran - it is the definition, not a summary, so it can be edited and re-run. For audio and video the `media` block carries duration, sample rate, channels, fps, size and level - the checks an agent - that cannot listen makes on a deliverable.""" - return catalog.get_gallery_metadata(client, name) + that cannot listen makes on a deliverable. `envelope=true` adds + that level second by second (`media.envelope.rms_dbfs` / + `peak_dbfs`, one entry per second), which is what says *where* in a + track something is: whether a shot is still sounding at its last + frame, how deep the hole at a seam goes, where a score goes quiet. + Leave it off unless you are asking a question about a position in + the track - a long track is a long list.""" + return catalog.get_gallery_metadata(client, name, envelope=envelope) def list_guides() -> dict: """List the documentation the engine serves: each guide's diff --git a/tests/test_media_info.py b/tests/test_media_info.py index aef76bde..b45d925b 100644 --- a/tests/test_media_info.py +++ b/tests/test_media_info.py @@ -179,3 +179,77 @@ def test_a_damaged_track_still_reports_header_fields(tmp_path): assert info["kind"] == "video" assert "peak_dbfs" not in info assert "frame_count" not in info + + +class TestEnvelope: + """The per-second level: what says *where* in a track something is, + rather than only how loud the whole of it was.""" + + def write_gapped_wav(self, path, seconds=4.0, sample_rate=8000, silent_second=2): + """A tone with one second of silence punched out of the middle of it.""" + import wave + + t = numpy.arange(int(seconds * sample_rate)) / sample_rate + samples = (numpy.sin(2 * numpy.pi * 220 * t) * 0.5 * 32767).astype(" -20.0 + + def test_a_video_soundtrack_gets_one_too(self, tmp_path): + write_mp4(tmp_path / "shot.mp4", frames=12, fps=6) + + info = probe_media(str(tmp_path / "shot.mp4"), envelope=True) + + assert info["kind"] == "video" + assert info["frame_count"] == 12 + # 2 s of soundtrack - and a third, near-silent bin is allowed: a + # lossy codec decodes its own priming and padding past the nominal + # duration, and the envelope reports what actually decoded + assert len(info["envelope"]["rms_dbfs"]) in (2, 3) + assert info["envelope"]["rms_dbfs"][0] > -30.0 + assert info["peak_dbfs"] < 0.0 + + def test_a_silent_video_has_no_envelope_to_report(self, tmp_path): + write_mp4(tmp_path / "mute.mp4", frames=12, fps=6, with_audio=False) + + assert "envelope" not in probe_media(str(tmp_path / "mute.mp4"), envelope=True) + + def test_the_seconds_sum_back_to_the_whole_track(self, tmp_path): + """A bin holds the same sums the whole-track level is made of, so + recombining them has to land on the level the track reports.""" + self.write_gapped_wav(tmp_path / "score.wav", seconds=4.0) + + info = probe_media(str(tmp_path / "score.wav"), envelope=True) + + assert max(info["envelope"]["peak_dbfs"]) == pytest.approx( + info["peak_dbfs"], abs=0.01 + ) + power = numpy.mean([10 ** (db / 10) for db in info["envelope"]["rms_dbfs"]]) + assert 10 * math.log10(power) == pytest.approx(info["mean_dbfs"], abs=0.1) diff --git a/tests/test_server.py b/tests/test_server.py index 32332b5a..7b32f6be 100644 --- a/tests/test_server.py +++ b/tests/test_server.py @@ -1136,6 +1136,29 @@ def test_gallery_metadata_describes_audio_and_video(server, tmp_path): assert still["media"] is None +def test_gallery_metadata_reports_a_level_envelope_on_request(server, tmp_path): + """Per-second level is opt-in: the default answer stays small, and + `envelope=true` says where in the track the level sits.""" + from tests.test_media_info import write_wav + + with server(success_script) as client: + outputs = tmp_path / "outputs" + write_wav(outputs / "score-gen.0-0.0.wav", seconds=3.0) + + plain = client.get("/api/gallery/score-gen.0-0.0.wav/metadata").json() + assert "envelope" not in plain["media"] + + detailed = client.get( + "/api/gallery/score-gen.0-0.0.wav/metadata", params={"envelope": "true"} + ).json() + envelope = detailed["media"]["envelope"] + assert envelope["interval_seconds"] == 1.0 + assert len(envelope["rms_dbfs"]) == 3 + assert max(envelope["peak_dbfs"]) == pytest.approx( + detailed["media"]["peak_dbfs"], abs=0.01 + ) + + def test_gallery_metadata_survives_a_damaged_track(server, tmp_path): """A track that opens fine but fails partway through decode (damage past the header) must not 500 the metadata route - it should fall back From 011d0fc2f0319159af151f0ca96109e6e1651cfa Mon Sep 17 00:00:00 2001 From: Don Kackman Date: Fri, 11 Sep 2026 10:08:47 -0500 Subject: [PATCH 03/13] feat(mcp): T013, T006 - name an upload, and read variable defaults cheaply T013: upload_asset(file_path, asset_name=...) and POST /api/uploads?asset_name= store an upload under a readable name instead of a random hex one. A recurring cast referenced as asset:uploads/084eaecc....wav in every workflow cannot be told apart without opening each file. The name may carry folders, is validated by validate_asset_reference and confined to the library the way keep_output's is, and takes the uploaded file's extension when it has none. No name given means the old random one, so two uploads of the same file still never collide. T006: GET /api/workflows/{name}/variables and get_workflow(name, variables_only=True) answer with a workflow's variables and their defaults and nothing else - confirming audio_bleed_ms defaults to 1800 otherwise meant pulling the whole definition, SDNQ quantization blocks and all, for one integer. String defaults over 200 characters are cut and named in `truncated`; full=true returns them whole. Co-Authored-By: Claude Opus 5 --- docs/MCP.md | 4 +- docs/SERVER.md | 8 +++- dw/server/app.py | 83 ++++++++++++++++++++++++++++++++++++++- dw_mcp/assets.py | 16 ++++++-- dw_mcp/catalog.py | 13 +++++- dw_mcp/server.py | 21 +++++++--- tests/test_mcp_assets.py | 18 +++++++++ tests/test_mcp_catalog.py | 4 ++ tests/test_mcp_server.py | 2 + tests/test_server.py | 74 ++++++++++++++++++++++++++++++++++ 10 files changed, 226 insertions(+), 17 deletions(-) diff --git a/docs/MCP.md b/docs/MCP.md index 5c7c973f..7f8d56fc 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -213,7 +213,7 @@ when no single workflow covers it. | `list_guides()` | — | List the documentation the engine serves: each guide's name, what it covers, and its section headings. The index is the routing table - match a request's shape against a heading rather than guessing | | `get_guide(name, section=None)` | `name`, `section` | Get one guide whole, or one section of it. Prefer a section: a guide runs to thousands of lines. Section names match loosely, so a heading copied approximately still resolves | | `list_workflows(shape=None, traits=None, configures=None, include_models=False)` | `shape`, `traits`, `configures`, `include_models` | List stored workflows. Always the server's compact view: each entry carries `summary`, `shape`, `traits`, `cost`, `kinds`, `variable_names`, and `configures` only when set - `get_workflow` has the full description and definition. `shape` keeps one of `image`, `image-set`, `image-edit`, `shot`, `sequence`, `audio`, `text`, `utility`; `traits` is comma-separated and every one listed must match (`has-audio`, `chained`, `image-conditioned`, `identity-referenced`, `needs-input-media`, `composes-workflows`); an unknown value in either is a 400 listing the vocabulary. Templates only by default - `configures=