diff --git a/CHANGELOG.md b/CHANGELOG.md index e3065d4..5d4ce3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,19 +9,47 @@ its series' file in [changelog/](changelog/) when a newer one ships. ## [Unreleased] -## [0.68.0] - 2026-11-17 +## [0.69.0] - 2026-11-17 ### Added -- `ForEachSubtitleBitmap` and `ForEachSubtitleBitmapFrom` (`matroska`, - `mkv/ops`) give a bitmap subtitle track as timed pictures whatever its codec, - PGS, VobSub or DVB, walking the file or served from a `SubtitleIndex`; - `BitmapCue` is the type `PGSCue` names. `ForEachSubtitlePGS` and its - siblings are unchanged. -- `ForEachSubtitleBlock` and `ForEachSubtitleBlockFrom` give the blocks of a - subtitle track of any codec (`SubtitleBlock`: time, duration, decoded - payload), walked or served from an index under the checks of the other - indexed reads, a damaged region passed over as a `Lost` call. -- `extract-subtitle -format png` writes any bitmap track (PGS, VobSub, DVB) as - PNGs and a `cues.json` manifest, with `-index` too; `-format pgs` is - unchanged. +- `ForEachSubtitleBitmapFile` (`matroska`, `mkv/ops`) gives a bitmap subtitle + file that sits beside a video - a PGS `.sup`, or a VobSub `.idx`/`.sub` + pair - as cues, timed and ended as `ForEachSubtitleBitmap` does a track's, + a unit that does not decode lost alone. `extract-subtitle` reads such a file + as `-format png`, `-t` picking a `.idx`'s language from 1. +- `DamagedRange.Err` (`mkv`, `matroska`): set on the range `Options.OnSkip` + gets for a bitmap subtitle block that was read but did not decode, to the + decoder's error; nil for a damaged region. `extract-subtitle` names such a + block on stderr by its time and that error, in a message of its own. + +### Breaking + +- Bytes written: `from-mp4` of a fragmented MP4 without `mehd` writes the + Matroska `Duration` the probe now reports, the end of the latest track - a + track ending where it starts plus what its samples last. A video track that + starts after zero (a composition offset, no edit list) and outlasts the + audio was left out, the duration being the longest sum of sample durations + (2.763 s for a file whose last frame ends at 2.803 s; the probe of 17 of + the 732 files of the public test corpora changes, none shortening). A + track's own duration does not change. +- `subtitle.DVBSubDecoder` and `subtitle.PGSDecoder` keep the picture of a + region or an object from one display set to the next, until its pixels or + its colours change: display sets that show the same picture now share one + image, as its placements within a set did, so a caller that wrote into a + returned image changes the next sets' too - treat the images as read-only. + A display set that only names an unchanged region again no longer reads it + through its colour table again (a 14-byte block made a 4096x4096 region be + rendered whole). + +### Fixed + +- A bitmap cue whose block states a duration ends no later than its DVB + page's time-out; the block's duration won. +- `ForEachSubtitleBitmap`, `ForEachSubtitlePGS`, their `From` forms and + `extract-subtitle -format png|pgs`: a block that reads but does not decode + no longer fails the whole track. It is lost alone, the cue on screen ending + at its time, and reported through `Options.OnSkip` (`DamagedRange.Err` + says why); the rest of the track is delivered. `Options.DamageBytes` + negative (`--damage-bytes -1`) still fails the read on it. A PGS track with + such a block failed since PGS extraction exists. diff --git a/README.md b/README.md index 2a9b142..c248733 100644 --- a/README.md +++ b/README.md @@ -189,7 +189,7 @@ mkvgo [options] # global: -json, -f/--force, --version | | `playability` · `ladder` | Direct-play verdict per client target / recommended ABR rungs capped at the source | | | `ingest` | One-call serving plan for a file: direct-play, remux to HLS, or transcode | | **Extract** | `demux` | Extract tracks to raw streams (MKV/WebM or MP4) | -| | `extract-subtitle` · `to-vtt` | Subtitle track → SRT/ASS/WebVTT, or a bitmap track (PGS, VobSub, DVB) → PNGs + manifest; external sidecar → WebVTT | +| | `extract-subtitle` · `to-vtt` | Subtitle track → SRT/ASS/WebVTT, or a bitmap track (PGS, VobSub, DVB) → PNGs + manifest; external sidecar → WebVTT, or a `.sup`/`.idx`+`.sub` → PNGs + manifest | | | `extract-attachment` · `add-attachment` · `remove-attachment` | Manage attachments (MIME sniffed) | | | `extract-frame` | Keyframe nearest a time, decoder-ready - thumbnail/storyboard pipelines | | | `extract-frames` | Keyframe-only Matroska stream (every N seconds or at given times) | diff --git a/changelog/0.60-0.69.md b/changelog/0.60-0.69.md index b3791e4..237360a 100644 --- a/changelog/0.60-0.69.md +++ b/changelog/0.60-0.69.md @@ -2,6 +2,23 @@ Archived entries of mkvgo 0.60.x to 0.69.x, as they were published; a version moves here once a newer one ships. The current versions are in [CHANGELOG.md](../CHANGELOG.md). +## [0.68.0] - 2026-11-17 + +### Added + +- `ForEachSubtitleBitmap` and `ForEachSubtitleBitmapFrom` (`matroska`, + `mkv/ops`) give a bitmap subtitle track as timed pictures whatever its codec, + PGS, VobSub or DVB, walking the file or served from a `SubtitleIndex`; + `BitmapCue` is the type `PGSCue` names. `ForEachSubtitlePGS` and its + siblings are unchanged. +- `ForEachSubtitleBlock` and `ForEachSubtitleBlockFrom` give the blocks of a + subtitle track of any codec (`SubtitleBlock`: time, duration, decoded + payload), walked or served from an index under the checks of the other + indexed reads, a damaged region passed over as a `Lost` call. +- `extract-subtitle -format png` writes any bitmap track (PGS, VobSub, DVB) as + PNGs and a `cues.json` manifest, with `-index` too; `-format pgs` is + unchanged. + ## [0.67.0] - 2026-11-17 ### Added diff --git a/cmd/mkvgo/commands/extract.go b/cmd/mkvgo/commands/extract.go index e57852b..e567714 100644 --- a/cmd/mkvgo/commands/extract.go +++ b/cmd/mkvgo/commands/extract.go @@ -46,13 +46,12 @@ func CmdExtractAttachment(args []string) { func CmdExtractSubtitle(args []string) { usage := CmdUsage["extract-subtitle"] - if len(args) < 5 { + if len(args) < 3 { FatalUsage("usage: " + usage) } var outPath, format, indexPath string var trackID uint64 var damageBytes int64 - format = "srt" fl := newFlags("extract-subtitle") fl.String(&outPath, "-o") @@ -67,6 +66,13 @@ func CmdExtractSubtitle(args []string) { }, "-t") fl.String(&format, "-format") source, _ := fl.ParseSource(args, usage) + if isBitmapSubtitleFile(source) { + extractBitmapSubtitleFile(source, outPath, format, indexPath, trackID, damageBytes, usage) + return + } + if format == "" { + format = "srt" + } if outPath == "" || trackID == 0 { FatalUsage("usage: " + usage) } @@ -198,6 +204,43 @@ type pgsCueManifest struct { File string `json:"file"` } +// isBitmapSubtitleFile reports a subtitle file that sits beside a video: a PGS .sup, a VobSub .idx or .sub. +func isBitmapSubtitleFile(path string) bool { + switch strings.ToLower(filepath.Ext(path)) { + case ".sup", ".idx", ".sub": + return true + } + return false +} + +// extractBitmapSubtitleFile writes the cues of a .sup or a .idx/.sub pair as PNGs and a manifest. -t picks the +// language of a .idx by its position in it, from 1; a .sup holds one. +func extractBitmapSubtitleFile(source, outPath, format, indexPath string, stream uint64, damageBytes int64, usage string) { + if outPath == "" { + FatalUsage("usage: " + usage) + } + pgsOfSup := format == "pgs" && strings.EqualFold(filepath.Ext(source), ".sup") + if format != "" && format != "png" && !pgsOfSup { + FatalUsage(fmt.Sprintf("%s holds bitmap subtitles: -format png writes them, -format %s does not apply", filepath.Base(source), format)) + } + if indexPath != "" { + FatalUsage("-index applies to a subtitle track of an MKV/WebM, not to a subtitle file beside the video") + } + if stream == 0 { + stream = 1 + } + guardOutput(outPath, source) + opts := subtitleReadOptions(damageBytes) + opts.SyncOutput = SyncOutput + n, err := writePGSDir(outPath, func(fn func(matroska.BitmapCue) error) error { + return matroska.ForEachSubtitleBitmapFile(cliCtx, source, int(stream-1), fn, opts) + }) + if err != nil { + FatalErr(err) + } + fmt.Printf("extracted %d bitmap cue(s) from %s (stream %d) → %s\n", n, source, stream, outPath) +} + // writePGSDir writes one PNG per cue into outDir plus a cues.json manifest. It // drives the streaming extractor, so a two-hour track costs one picture of // memory rather than the whole track's worth. @@ -333,6 +376,15 @@ func subtitleIndexFromCues(source string, trackIDs []uint64, opts ...matroska.Op func subtitleReadOptions(damageBytes int64) matroska.Options { return matroska.Options{DamageBytes: damageBytes, OnSkip: func(r matroska.DamagedRange) { noteDamage() + if r.Err != nil && r.StartOffset < 0 { // a unit of a file beside the video: no byte range there + fmt.Fprintf(os.Stderr, "lost the subtitle at %d ms, which does not decode: %v\n", r.ApproxStartMs, r.Err) + return + } + if r.Err != nil { // a block that was read: its time and what its decoder said + fmt.Fprintf(os.Stderr, "lost the subtitle block at %d ms (bytes %d to %d), which does not decode: %v\n", + r.ApproxStartMs, r.StartOffset, r.EndOffset, r.Err) + return + } fmt.Fprintf(os.Stderr, "skipped a damaged region at bytes %d to %d (about %d to %d ms): the subtitles in it are lost\n", r.StartOffset, r.EndOffset, r.ApproxStartMs, r.ApproxEndMs) }} diff --git a/cmd/mkvgo/commands/extract_pgs_test.go b/cmd/mkvgo/commands/extract_pgs_test.go index a98381d..0f77682 100644 --- a/cmd/mkvgo/commands/extract_pgs_test.go +++ b/cmd/mkvgo/commands/extract_pgs_test.go @@ -231,3 +231,81 @@ func TestCmdExtractSubtitle_PNGWritesAVobSubTrack(t *testing.T) { } } } + +// On stderr a block that does not decode is named by its time and its error, in words of its own: a region the read +// jumped over keeps the message it had. +func TestCmdExtractSubtitle_NamesABlockThatDoesNotDecode(t *testing.T) { + data, err := os.ReadFile("../../../internal/testdata/vobsub.mkv") + if err != nil { + t.Fatal(err) + } + clear(data[len(data)/2 : len(data)/2+4096]) + src := filepath.Join(t.TempDir(), "zeroed.mkv") + if err := os.WriteFile(src, data, 0o644); err != nil { + t.Fatal(err) + } + var out string + errOut := stderrOf(t, func() { + out = capture(t, func() { + commands.CmdExtractSubtitle([]string{src, "-t", "1", "-o", filepath.Join(t.TempDir(), "subs"), "-format", "png"}) + }) + }) + if !strings.Contains(out, "extracted 41 bitmap cue(s)") { + t.Errorf("stdout = %q, want 41 cues", out) + } + lines := strings.Split(strings.TrimSpace(errOut), "\n") + if len(lines) != 2 { + t.Fatalf("stderr holds %d line(s), want the region and the block:\n%s", len(lines), errOut) + } + // The block sits ahead of the zeroed bytes it runs into: it is read, then the region is jumped over. + want := "lost the subtitle block at 230872 ms (bytes " + if !strings.HasPrefix(lines[0], want) || !strings.HasSuffix(lines[0], "), which does not decode: VobSub: the control sequence at byte 2979 names byte 0 as the next one, which is not after it") { + t.Errorf("the block is announced as %q, want its time and the decoder's error", lines[0]) + } + if !strings.HasPrefix(lines[1], "skipped a damaged region at bytes ") || strings.Contains(lines[1], "decode") { + t.Errorf("the region is announced as %q", lines[1]) + } +} + +// A .sup or a .idx/.sub pair beside the video is read as -format png without one, -t picking a .idx's language from +// 1; a unit that does not decode is named on stderr by its time and its error. +func TestCmdExtractSubtitle_BitmapFileBesideTheVideo(t *testing.T) { + dir := t.TempDir() + out := capture(t, func() { + commands.CmdExtractSubtitle([]string{"../../../internal/testdata/pgs_sub.sup", "-o", filepath.Join(dir, "sup")}) + }) + if !strings.Contains(out, "extracted 2 bitmap cue(s) from ../../../internal/testdata/pgs_sub.sup (stream 1)") { + t.Errorf("stdout = %q, want the two display sets of the .sup", out) + } + sub, err := os.ReadFile("../../../internal/testdata/vobsub.sub") + if err != nil { + t.Fatal(err) + } + idx, err := os.ReadFile("../../../internal/testdata/vobsub.idx") + if err != nil { + t.Fatal(err) + } + // The unit of the first entry, at 132499 ms, its control sequences placed inside its header. + unit := 14 + int(sub[13]&7) + 6 + 3 + int(sub[14+int(sub[13]&7)+8]) + 1 + sub[unit+2], sub[unit+3] = 0, 2 + for name, data := range map[string][]byte{"film.idx": idx, "film.sub": sub} { + if err := os.WriteFile(filepath.Join(dir, name), data, 0o644); err != nil { + t.Fatal(err) + } + } + errOut := stderrOf(t, func() { + out = capture(t, func() { + commands.CmdExtractSubtitle([]string{filepath.Join(dir, "film.sub"), "-t", "1", "-o", filepath.Join(dir, "vob"), "-format", "png"}) + }) + }) + if !strings.Contains(out, "extracted 42 bitmap cue(s)") { + t.Errorf("stdout = %q, want 42 cues", out) + } + if want := "lost the subtitle at 132499 ms, which does not decode: VobSub: the subpicture places its control sequences at byte 2"; !strings.HasPrefix(strings.TrimSpace(errOut), want) { + t.Errorf("stderr = %q, want it to start with %q", errOut, want) + } + blob, err := os.ReadFile(filepath.Join(dir, "vob", "cues.json")) + if err != nil || !strings.Contains(string(blob), `"start_ms": 147355`) { + t.Errorf("the manifest does not hold the second cue: %v", err) + } +} diff --git a/cmd/mkvgo/commands/helpers.go b/cmd/mkvgo/commands/helpers.go index cfee9bc..e2a1015 100644 --- a/cmd/mkvgo/commands/helpers.go +++ b/cmd/mkvgo/commands/helpers.go @@ -113,7 +113,7 @@ var CmdUsage = map[string]string{ "set-chapters": "mkvgo set-chapters -o (OGM format: CHAPTER01=00:00:00.000 / CHAPTER01NAME=Intro)", "extract-chapters": "mkvgo extract-chapters [-o ] (OGM format, stdout by default)", "remove-attachment": "mkvgo remove-attachment -o ", - "extract-subtitle": "mkvgo extract-subtitle -t -o [-format srt|ass|vtt|png|pgs (default: srt; png writes a bitmap track - PGS, VobSub, DVB - as a directory of PNGs + cues.json, pgs the same for a PGS track only)] [-index (vtt/png/pgs, MKV only)] [--damage-bytes n (MKV: pass over a damaged region within n bytes, its cues lost; default 64 MiB)]", + "extract-subtitle": "mkvgo extract-subtitle -t -o [-format srt|ass|vtt|png|pgs (default: srt; png writes a bitmap track - PGS, VobSub, DVB - as a directory of PNGs + cues.json, pgs the same for a PGS track only; a .sup or a .idx/.sub pair beside the video is read as png, -t picking a .idx's language from 1)] [-index (vtt/png/pgs, MKV only)] [--damage-bytes n (MKV: pass over a damaged region within n bytes, its cues lost; default 64 MiB)]", "subtitle-index": "mkvgo subtitle-index -o [-t 4,5 (default: every subtitle track)] [-from-cues] [--damage-bytes n]", "split": "mkvgo split -o [-chapters | -range 0-5:00,5:00-0 | -every 6:00] [-pattern part_%03d.mkv|.mp4 ({title} = chapter title)]", "join": "mkvgo join -o ... (all MKV/WebM or all MP4/MOV)", diff --git a/docs/cli.md b/docs/cli.md index 8ad7500..3c0a435 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -439,10 +439,10 @@ mkvgo remove-attachment video.mkv -o out.mkv 2 ### extract-subtitle -Extract an embedded text subtitle track as SRT, ASS or WebVTT, or a bitmap one as pictures. SRT/ASS apply to MKV/WebM; WebVTT (`-format vtt`) also works on MP4 (tx3g/wvtt). `-format png` reads a bitmap track of an MKV/WebM (PGS, VobSub or DVB, by its codec) and writes into the directory `-o` one PNG per cue and a `cues.json` manifest: for each cue its file, `start_ms`, `end_ms`, its position `x`, `y` on the subtitle plane `screen_w` x `screen_h`, and `forced`. `-format pgs` is the same for a PGS track, any other codec refused. +Extract an embedded text subtitle track as SRT, ASS or WebVTT, or a bitmap one as pictures. SRT/ASS apply to MKV/WebM; WebVTT (`-format vtt`) also works on MP4 (tx3g/wvtt). `-format png` reads a bitmap track of an MKV/WebM (PGS, VobSub or DVB, by its codec) and writes into the directory `-o` one PNG per cue and a `cues.json` manifest: for each cue its file, `start_ms`, `end_ms`, its position `x`, `y` on the subtitle plane `screen_w` x `screen_h`, and `forced`. `-format pgs` is the same for a PGS track, any other codec refused. A bitmap subtitle file that sits beside the video - a PGS `.sup`, or a VobSub pair named by its `.idx` or its `.sub` - is written the same way, `-format png` being its only format and its default; `-t` picks the language of a `.idx` by its position in it, from 1 (default 1). ``` -mkvgo extract-subtitle -t -o [-format srt|ass|vtt|png|pgs] [-index ] [--damage-bytes ] +mkvgo extract-subtitle -t -o [-format srt|ass|vtt|png|pgs] [-index ] [--damage-bytes ] ``` | Flag | Description | @@ -451,7 +451,7 @@ mkvgo extract-subtitle -t -o [-format srt|ass|vt | `-o` | Output file path, or directory for `png` and `pgs` (required) | | `-format` | Output format: `srt` (default), `ass`, `vtt`, `png` or `pgs` | | `-index` | Serve from a `subtitle-index` file instead of walking the source (`-format vtt`, `png` and `pgs`, MKV/WebM only) | -| `--damage-bytes` | How far past an unreadable element the read looks for the next Cluster (MKV/WebM). A damaged region within it is passed over, its cues lost and named on stderr; a longer one fails the extraction. With `--exit-damaged`, an extraction that passed over a region exits 3. `-1` refuses any damage. Default 64 MiB | +| `--damage-bytes` | How far past an unreadable element the read looks for the next Cluster (MKV/WebM). A damaged region within it is passed over, its cues lost and named on stderr; a longer one fails the extraction. With `png` and `pgs`, a block that reads but does not decode is lost too, and named on stderr by its time and its decoding error (`lost the subtitle block at ms (bytes to ), which does not decode: `; for a file beside the video, `lost the subtitle at ms, which does not decode: `). With `--exit-damaged`, an extraction that passed over a region exits 3. `-1` refuses any damage. Default 64 MiB | ```bash mkvgo extract-subtitle video.mkv -t 3 -o subs.srt @@ -461,6 +461,8 @@ mkvgo extract-subtitle movie.mp4 -t 3 -o subs.vtt -format vtt mkvgo extract-subtitle video.mkv -t 3 -o subs.vtt -format vtt -index video.mkvsix mkvgo extract-subtitle dvdrip.mkv -t 4 -o subs/ -format png mkvgo extract-subtitle dvdrip.mkv -t 4 -o subs/ -format png -index dvdrip.mkvsix +mkvgo extract-subtitle dvdrip.idx -t 2 -o subs/ # the second language of a .idx/.sub pair +mkvgo extract-subtitle movie.sup -o subs/ ``` ### subtitle-index diff --git a/docs/library.md b/docs/library.md index 28d1046..7d87081 100644 --- a/docs/library.md +++ b/docs/library.md @@ -1459,7 +1459,7 @@ fallback is a `data:` URI embedding `Key`, the raw key in the playlist text. err := mp4.RemuxFromMP4(ctx, "in.mp4", "out.mkv") ``` -Reads `avc1`/`avc3`, `hvc1`/`hev1`, `av01`, `vp09`, `mp4v` (MPEG-4 Part 2 → `V_MPEG4/ISO/ASP`, its `esds` DecoderSpecificInfo the CodecPrivate), `mp4a` (AAC, MP3 or DTS, by `esds` object type), `Opus`, `ac-3`, `ec-3`, `mlpa` (TrueHD → `A_TRUEHD`), `alac` (→ `A_ALAC`, the 24-byte ALACSpecificConfig), `fLaC`, `tx3g` (→ SRT) and `wvtt` (→ WebVTT). Every configuration record is carried in the form the target container specifies, never as the source's: a `vp09` entry's `vpcC` becomes the Matroska VP9 Codec Feature Metadata records (profile, level, bit depth, chroma; its colour goes to the Colour element), `dOps` becomes an `OpusHead`, `dfLa` gets its `fLaC` marker back; the reverse remux turns an `OpusHead` into a `dOps`, feature metadata or a missing CodecPrivate into a `vpcC` derived from the first keyframe, and refuses a CodecPrivate that is not the record its box would claim (an Annex-B parameter-set stream where an `avcC` is expected, a bare OBU where an `av1C` is) rather than wrap it and produce an MP4 no player decodes. An `avcC`/`hvcC` that carries no parameter sets (they travel in the samples) is declared as such: the sample entry and codec string are `avc3`/`hev1`, not `avc1`/`hvc1`. Colour code points, chapters, the movie title (`udta/meta/ilst/©nam` → `Info.Title`), the other global tags (`©ART`/`©alb`/`©gen`/… → `Tags`) and per-track names (`udta/name`, else an `hdlr` name that is not a muxer's default → `Track.Name`; the `hdlr` name itself is `Track.HandlerName`, the probe JSON's `handler_name`, and a muxer's default is never a title: a control character, the track type's fourcc, a `Handler` or `Handle` suffix, a single word tagged with `@` or `#`, a bare media word such as `Video`, a known tool name - a title with spaces, as HandBrake writes them, stays one) round-trip back to Matroska - and back out to MP4 with `RemuxToMP4`. Audio decodes bit-identically across the round trip for AAC/AC-3/E-AC-3/FLAC; Opus and MP3 stay in sync (the Opus pre-skip travels as the edit list <-> `CodecDelay`, like AAC; MP3's delay is handled by the decoder from the bitstream). PCM entries (`twos`, `sowt`, `raw `, `in24`/`in32`/`fl32`/`fl64` with the `enda` byte order, `ipcm`/`fpcm` by their `pcmC`, the QuickTime version 2 `lpcm`) become `A_PCM/INT/BIG`, `A_PCM/INT/LIT` or `A_PCM/FLOAT/IEEE` with the bit depth, their one-frame samples gathered into blocks of at most 1024 frames that never cross a chunk - the packets the mainstream demuxer reads such a track in, so the remux is packet-identical. The one rule of every remux, in both directions: an audio or video track Matroska cannot carry - an encrypted one (`encv`/`enca`; `Track.Encryption` names the scheme), a sample entry with no Matroska form (MPEG-H, AC-4, IAMF, AMR, a PCM layout Matroska has no form for; MPEG-4 Part 2, TrueHD and ALAC are carried, as the mainstream muxer carries them), one whose configuration is not what its codec needs - fails the remux by name unless `Options.SkipUnsupported` drops it (reported via `OnDrop`); a subtitle track it cannot carry (TTML `stpp`, VobSub, an encrypted one) never fails the remux and is dropped with its reason, like `RemuxToMP4` drops ASS or PGS; non-audio/video/subtitle tracks and still pictures are dropped. A fragmented source's duration is its `mehd`'s, else the end of its fragments; a source without samples (an initialisation segment) is refused. An incomplete source (its `moov` intact, its sample tables placing media past the end of the file) remuxes what arrived and is reported once through `Options.OnTruncated` (`Truncation`: the file size, how far the media that arrived plays, the declared duration), so a caller can tell that output from the whole file's. `RemuxToMP4` and `RemuxToHLS` report the same way a Matroska that stops before the end its Segment declares (a Segment of unknown size declares none), and `RemuxToHLS` an MP4 cut short. +Reads `avc1`/`avc3`, `hvc1`/`hev1`, `av01`, `vp09`, `mp4v` (MPEG-4 Part 2 → `V_MPEG4/ISO/ASP`, its `esds` DecoderSpecificInfo the CodecPrivate), `mp4a` (AAC, MP3 or DTS, by `esds` object type), `Opus`, `ac-3`, `ec-3`, `mlpa` (TrueHD → `A_TRUEHD`), `alac` (→ `A_ALAC`, the 24-byte ALACSpecificConfig), `fLaC`, `tx3g` (→ SRT) and `wvtt` (→ WebVTT). Every configuration record is carried in the form the target container specifies, never as the source's: a `vp09` entry's `vpcC` becomes the Matroska VP9 Codec Feature Metadata records (profile, level, bit depth, chroma; its colour goes to the Colour element), `dOps` becomes an `OpusHead`, `dfLa` gets its `fLaC` marker back; the reverse remux turns an `OpusHead` into a `dOps`, feature metadata or a missing CodecPrivate into a `vpcC` derived from the first keyframe, and refuses a CodecPrivate that is not the record its box would claim (an Annex-B parameter-set stream where an `avcC` is expected, a bare OBU where an `av1C` is) rather than wrap it and produce an MP4 no player decodes. An `avcC`/`hvcC` that carries no parameter sets (they travel in the samples) is declared as such: the sample entry and codec string are `avc3`/`hev1`, not `avc1`/`hvc1`. Colour code points, chapters, the movie title (`udta/meta/ilst/©nam` → `Info.Title`), the other global tags (`©ART`/`©alb`/`©gen`/… → `Tags`) and per-track names (`udta/name`, else an `hdlr` name that is not a muxer's default → `Track.Name`; the `hdlr` name itself is `Track.HandlerName`, the probe JSON's `handler_name`, and a muxer's default is never a title: a control character, the track type's fourcc, a `Handler` or `Handle` suffix, a single word tagged with `@` or `#`, a bare media word such as `Video`, a known tool name - a title with spaces, as HandBrake writes them, stays one) round-trip back to Matroska - and back out to MP4 with `RemuxToMP4`. Audio decodes bit-identically across the round trip for AAC/AC-3/E-AC-3/FLAC; Opus and MP3 stay in sync (the Opus pre-skip travels as the edit list <-> `CodecDelay`, like AAC; MP3's delay is handled by the decoder from the bitstream). PCM entries (`twos`, `sowt`, `raw `, `in24`/`in32`/`fl32`/`fl64` with the `enda` byte order, `ipcm`/`fpcm` by their `pcmC`, the QuickTime version 2 `lpcm`) become `A_PCM/INT/BIG`, `A_PCM/INT/LIT` or `A_PCM/FLOAT/IEEE` with the bit depth, their one-frame samples gathered into blocks of at most 1024 frames that never cross a chunk - the packets the mainstream demuxer reads such a track in, so the remux is packet-identical. The one rule of every remux, in both directions: an audio or video track Matroska cannot carry - an encrypted one (`encv`/`enca`; `Track.Encryption` names the scheme), a sample entry with no Matroska form (MPEG-H, AC-4, IAMF, AMR, a PCM layout Matroska has no form for; MPEG-4 Part 2, TrueHD and ALAC are carried, as the mainstream muxer carries them), one whose configuration is not what its codec needs - fails the remux by name unless `Options.SkipUnsupported` drops it (reported via `OnDrop`); a subtitle track it cannot carry (TTML `stpp`, VobSub, an encrypted one) never fails the remux and is dropped with its reason, like `RemuxToMP4` drops ASS or PGS; non-audio/video/subtitle tracks and still pictures are dropped. A fragmented source's duration is its `mehd`'s, else the end of its latest track, a track ending where it starts plus what its fragments' samples last (one whose first sample carries a composition offset, without an edit list, starts that much after zero); a source without samples (an initialisation segment) is refused. An incomplete source (its `moov` intact, its sample tables placing media past the end of the file) remuxes what arrived and is reported once through `Options.OnTruncated` (`Truncation`: the file size, how far the media that arrived plays, the declared duration), so a caller can tell that output from the whole file's. `RemuxToMP4` and `RemuxToHLS` report the same way a Matroska that stops before the end its Segment declares (a Segment of unknown size declares none), and `RemuxToHLS` an MP4 cut short. #### Subtitles @@ -1628,7 +1628,8 @@ within `Options.DamageBytes` of it, 64 MiB by default (`mkv.DefaultDamageBytes`) as the on-demand HLS windows do. The cues inside the region are lost and the rest of the track is delivered; a bitmap cue the region left on screen ends with the default duration (or at the start of the cluster the region begins -in) instead of lasting until the next cue past it. Each region is reported once +in) instead of lasting until the next cue past it. The bitmap reads lose the +same way, alone, a block that reads but does not decode. Each region is reported once through `Options.OnSkip`, with its byte range and the timestamps of the clusters around it. A longer region fails the read with `reader.ErrDamagedRegion`, and a negative `DamageBytes` refuses any damage. @@ -1731,13 +1732,27 @@ err := matroska.ForEachSubtitleBitmap(ctx, "movie.mkv", trackID, func(c matroska err = matroska.ForEachSubtitleBitmapFrom(ctx, "movie.mkv", trackID, ix, fn) ``` -`BitmapCue` is the type `PGSCue` names. One cue's picture is alive at a time. +`BitmapCue` is the type `PGSCue` names. One cue's picture is alive at a time; +a cue that shows the same picture as an earlier one (a DVB region, a PGS +object, that did not change) is handed the same image, so treat `Image` as +read-only. A cue starts where its display set takes effect and ends, in this order: where the stream says (a VobSub stop command), at the duration the muxer wrote for -its block, or where the next display set begins - no later than a DVB page's -time-out, and 3 s after its start for a last cue nothing closes. A display set -of several pictures (two PGS windows, two DVB regions) is one cue, composited -on the smallest rectangle that holds them. A text track is refused by name, as +its block, or where the next display set begins - in every case no later than +a DVB page's time-out, and 3 s after its start for a last cue nothing closes. +A display set of several pictures (two PGS windows, two DVB regions) is one cue, composited +on the smallest rectangle that holds them. A block that reads but does not +decode - its payload does not decompress, its bytes are not what the codec +reads, its pictures cannot be composited - is lost alone: the cue on screen +ends at the block's time, the decoder starts afresh, the block is reported +through `Options.OnSkip` and the read goes on. The `DamagedRange` of such a +block carries its byte range, its time and, in `Err`, the decoder's error; +`Err` is nil for a damaged region, bytes the read jumped over. A +DVB decoder that starts afresh holds no region and no colour table, so nothing +shows until the stream's next acquisition point or mode change. With a +negative `Options.DamageBytes`, which refuses any damage, such a block fails +the read by its time. An error the callback returns and a cancelled context +always stop it. A text track is refused by name, as is a bitmap codec without a decoder here. `ForEachSubtitlePGS` and its siblings below are the same read for a track that must be PGS. CLI: `mkvgo extract-subtitle -format png`, with `-index` for the indexed form. @@ -2015,8 +2030,10 @@ What differs from PGS: - **DVB** keeps regions, their pixels and the colour tables between blocks: a display set sends only what changed, until a page composition declares a mode change or an acquisition point. A set's objects are the regions on - screen, one picture each; a region the page places several times is read - once, its placements sharing the picture. The CodecPrivate names the + screen, one picture each; a region is read once for as long as it shows the + same thing, its placements and the display sets that follow sharing the + picture until its pixels or its colours change (the PGS decoder keeps an + object's picture the same way): treat the images as read-only. The CodecPrivate names the service's composition and ancillary pages, the segments of other pages being ignored; with one too short to name them every page is decoded. 2, 4 and 8-bit pixels, map tables, the default colour tables and the display @@ -2040,8 +2057,26 @@ error. ### Bitmap subtitles beside the video: .idx/.sub and .sup The same pictures also travel as files of their own: VobSub as a `.idx` and -a `.sub`, PGS as a `.sup`. Two readers turn them into what the decoders take, -one unit or display set at a time, so a whole file is never held. +a `.sub`, PGS as a `.sup`. One call gives them as cues, timed and ended by the +rules of `ForEachSubtitleBitmap`: + +```go +// A .sup, or a VobSub pair named by its .idx or its .sub (the other beside it, same name); +// the stream picks the .idx's language by its position, from 0. +err := matroska.ForEachSubtitleBitmapFile(ctx, "movie.idx", 0, func(c matroska.BitmapCue) error { + return png.Encode(w, c.Image) // c.StartMs, c.EndMs, c.X, c.Y, c.ScreenW, c.ScreenH, c.Forced +}) +``` + +A unit or a display set that does not decode is lost alone and reported +through `Options.OnSkip` with its time and error and no byte range +(`StartOffset` and `EndOffset` -1), unless a negative `Options.DamageBytes` +refuses it. A file the reader cannot go through to its end - a `.sup` cut in a +segment, a unit the `.sub` holds in part - gives the cues ahead of the fault, +then its error. CLI: `mkvgo extract-subtitle movie.idx -o `. + +Under it, two readers turn the files into what the decoders take, one unit or +display set at a time, so a whole file is never held. ```go ix, err := subtitle.ParseVobSubIndex(idxFile) // the .idx: header, languages, times diff --git a/matroska/matroska.go b/matroska/matroska.go index f6a674b..c9ddaee 100644 --- a/matroska/matroska.go +++ b/matroska/matroska.go @@ -1185,6 +1185,16 @@ func ForEachSubtitleBitmapFrom(ctx context.Context, srcPath string, trackID uint return ops.ForEachSubtitleBitmapFrom(ctx, srcPath, trackID, ix, fn, opts...) } +// ForEachSubtitleBitmapFile calls fn with each cue of a bitmap subtitle file beside a video - a PGS .sup, or a +// VobSub .idx/.sub pair, stream picking the .idx's language from 0 - timed as ForEachSubtitleBitmap times a track's. +// See ops.ForEachSubtitleBitmapFile. +func ForEachSubtitleBitmapFile(ctx context.Context, path string, stream int, fn func(BitmapCue) error, opts ...Options) error { + if err := oneopt.Check(opts); err != nil { + return err + } + return ops.ForEachSubtitleBitmapFile(ctx, path, stream, fn, opts...) +} + // SubtitleBlock is one block of a subtitle track, its payload as the codec reads it. See ops.SubtitleBlock. type SubtitleBlock = ops.SubtitleBlock diff --git a/mkv/ops/pgs_fuzz_test.go b/mkv/ops/pgs_fuzz_test.go index 4a15454..fd23f4e 100644 --- a/mkv/ops/pgs_fuzz_test.go +++ b/mkv/ops/pgs_fuzz_test.go @@ -15,11 +15,11 @@ func FuzzPGSAssembler(f *testing.F) { f.Fuzz(func(t *testing.T, first, second []byte, durMs int64) { costguard.Fuzz(t, len(first)+len(second), func() { var cues []PGSCue - a, err := newBitmapAssembler(func(c PGSCue) error { cues = append(cues, c); return nil }, &mkv.Track{Codec: "pgs"}) + a, err := newBitmapAssembler(func(c PGSCue) error { cues = append(cues, c); return nil }, &mkv.Track{Codec: "pgs"}, nil) if err != nil { t.Fatal(err) } - if a.block(1000, durMs%5000, first) != nil || a.block(2000, 0, second) != nil { + if a.block(mkv.Block{Timecode: 1000, Duration: durMs % 5000, Data: first}) != nil || a.block(mkv.Block{Timecode: 2000, Data: second}) != nil { return } for _, c := range cues { @@ -43,17 +43,17 @@ func FuzzBitmapAssembler(f *testing.F) { track = &mkv.Track{Codec: "dvbsub"} } var cues []BitmapCue - a, err := newBitmapAssembler(func(c BitmapCue) error { cues = append(cues, c); return nil }, track) + a, err := newBitmapAssembler(func(c BitmapCue) error { cues = append(cues, c); return nil }, track, nil) if err != nil { t.Fatal(err) } - if a.block(1000, durMs%5000, first) != nil { + if a.block(mkv.Block{Timecode: 1000, Duration: durMs % 5000, Data: first}) != nil { return } if lost && a.lost(1500) != nil { return } - if a.block(2000, 0, second) != nil || a.finish() != nil { + if a.block(mkv.Block{Timecode: 2000, Data: second}) != nil || a.finish() != nil { return } for _, c := range cues { diff --git a/mkv/ops/subbitmap.go b/mkv/ops/subbitmap.go index e21c5f6..00c11aa 100644 --- a/mkv/ops/subbitmap.go +++ b/mkv/ops/subbitmap.go @@ -20,17 +20,23 @@ type BitmapCue = PGSCue // ForEachSubtitleBitmap walks srcPath once and calls fn with each cue of the bitmap subtitle track trackID, in // presentation order: a PGS, VobSub or DVB track, decoded by its codec. Only one cue's picture is alive at a time, -// so a whole track is written out in bounded memory. An error from fn stops the walk and is returned as is. +// so a whole track is written out in bounded memory; a cue that shows the picture of an earlier one is handed the +// same image, to be read and not written. An error from fn stops the walk and is returned as is. // // A cue starts where its display set takes effect and ends, in this order: where the stream says (a VobSub stop -// command), at the duration the muxer wrote for its block, or where the next display set begins - no later than a -// DVB page's time-out, and 3 s after its start for a last cue nothing closes. A display set of several pictures is +// command), at the duration the muxer wrote for its block, or where the next display set begins - in every case no +// later than a DVB page's time-out, and 3 s after its start for a last cue nothing closes. A display set of several pictures is // one cue, composited on the smallest rectangle that holds them. +// +// A block that reads but does not decode is lost alone: the cue on screen ends at its time, the decoder starts +// afresh and the block is reported through Options.OnSkip (DamagedRange.Err says why), the read going on - unless Options.DamageBytes refuses +// any damage, which fails the read on it. A DVB decoder that starts afresh holds no region and no colour table, so +// nothing shows until the stream's next acquisition point or mode change. func ForEachSubtitleBitmap(ctx context.Context, srcPath string, trackID uint64, fn func(BitmapCue) error, opts ...mkv.Options) error { if err := oneopt.Check(opts); err != nil { return err } - return walkSubtitleBlocks(ctx, srcPath, trackID, bitmapCues(fn, nil), opts) + return walkSubtitleBlocks(ctx, srcPath, trackID, bitmapCues(fn, nil, opts), opts) } // ForEachSubtitleBitmapFrom is ForEachSubtitleBitmap served from a prebuilt index, under the checks of @@ -40,7 +46,7 @@ func ForEachSubtitleBitmapFrom(ctx context.Context, srcPath string, trackID uint if err := oneopt.Check(opts); err != nil { return err } - return indexedSubtitleBlocks(ctx, srcPath, trackID, ix, bitmapCues(fn, nil), opts) + return indexedSubtitleBlocks(ctx, srcPath, trackID, ix, bitmapCues(fn, nil, opts), opts) } // bitmapDecoder is what the three codecs' decoders share. @@ -67,7 +73,7 @@ func newBitmapDecoder(t *mkv.Track) (bitmapDecoder, string, error) { } // bitmapCues is the sink that decodes a bitmap track's blocks into cues; check, when set, refuses the track first. -func bitmapCues(fn func(BitmapCue) error, check func(*mkv.Track) error) subtitleSink { +func bitmapCues(fn func(BitmapCue) error, check func(*mkv.Track) error, opts []mkv.Options) subtitleSink { var asm *bitmapAssembler return subtitleSink{ accept: func(_ *mkv.Container, t *mkv.Track) error { @@ -76,11 +82,11 @@ func bitmapCues(fn func(BitmapCue) error, check func(*mkv.Track) error) subtitle return err } } - a, err := newBitmapAssembler(fn, t) + a, err := newBitmapAssembler(fn, t, opts) asm = a return err }, - block: func(blk mkv.Block) error { return asm.block(blk.Timecode, blk.Duration, blk.Data) }, + block: func(blk mkv.Block) error { return asm.block(blk) }, lost: func(fromMs int64) error { return asm.lost(fromMs) }, end: func() error { return asm.finish() }, } @@ -98,61 +104,118 @@ type bitmapAssembler struct { open *BitmapCue latest int64 // when the open cue ends at the latest: its page's time-out, 0 for none unitMs int64 // the time of the block that started the unit being decoded + strict bool // a block that does not decode fails the read: Options.DamageBytes refuses any damage + onSkip func(mkv.DamagedRange) } -func newBitmapAssembler(fn func(BitmapCue) error, t *mkv.Track) (*bitmapAssembler, error) { +func newBitmapAssembler(fn func(BitmapCue) error, t *mkv.Track, opts []mkv.Options) (*bitmapAssembler, error) { dec, name, err := newBitmapDecoder(t) if err != nil { return nil, err } - return &bitmapAssembler{dec: dec, name: name, emit: fn, track: t}, nil + return &bitmapAssembler{dec: dec, name: name, emit: fn, track: t, + strict: mkv.DamageBytesFrom(opts) < 0, onSkip: mkv.OnSkipFrom(opts)}, nil } -func (a *bitmapAssembler) block(timeMs, durMs int64, data []byte) error { - data, err := decompressSubtitleBlock(a.track, data) +// block decodes one block and emits the cues it completes. A block that does not decode is lost alone: decoding +// it is one thing, and what the caller's function returns for a cue is another, which always stops the read. +func (a *bitmapAssembler) block(blk mkv.Block) error { + shown, err := a.decode(blk) if err != nil { - return fmt.Errorf("subtitle block at %d ms: %w", timeMs, err) + return a.undecodable(blk, err) + } + for i := range shown { + s := &shown[i] + // Whatever is on screen ends where this display set begins, whether the new one clears the screen or + // replaces it. + if err := a.closeAt(s.cue.StartMs); err != nil { + return err + } + if s.clear { + continue + } + if s.cue.EndMs > s.cue.StartMs { // the stream or the muxer gave the end: emit and hold nothing + if err := a.emit(s.cue); err != nil { + return err + } + continue + } + cue := s.cue + a.open, a.latest = &cue, s.latest + } + return nil +} + +// decodedSet is one display set of a block, decoded and ready to be timed against what is on screen. +type decodedSet struct { + cue BitmapCue // StartMs set; EndMs when the stream or the muxer gave it, the picture unless clear + clear bool + latest int64 // when the cue ends at the latest, 0 for none +} + +// decode turns a block into its display sets, flattened; nothing is emitted, so an error loses the block whole. +func (a *bitmapAssembler) decode(blk mkv.Block) ([]decodedSet, error) { + timeMs := blk.Timecode + data, err := decompressSubtitleBlock(a.track, blk.Data) + if err != nil { + return nil, err } // A unit spread over several blocks (VobSub) is dated by the first of them. if c, ok := a.dec.(interface{ Continues([]byte) bool }); !ok || !c.Continues(data) { a.unitMs = timeMs } sets, err := a.dec.Decode(data) - if err != nil { - return fmt.Errorf("subtitle block at %d ms: %w", timeMs, err) + if err != nil { // the sets a decoder hands back with its error go with the block + return nil, err } + shown := make([]decodedSet, 0, len(sets)) for i := range sets { ds := &sets[i] start := a.unitMs + ds.Delay.Milliseconds() - // Whatever is on screen ends where this display set begins, whether the new one clears the screen or - // replaces it. - if err := a.closeAt(start); err != nil { - return err - } if ds.Clear() { + shown = append(shown, decodedSet{cue: BitmapCue{StartMs: start}, clear: true}) continue } cue, err := flattenDisplaySet(ds, start, a.name) if err != nil { - return fmt.Errorf("subtitle block at %d ms: %w", timeMs, err) + return nil, err } - // The stream's own end, then the muxer's, are known now: emit and hold nothing. + // The stream's own end, then the muxer's. switch { case ds.Duration > 0: cue.EndMs = start + ds.Duration.Milliseconds() - case durMs > 0: - cue.EndMs = timeMs + durMs + case blk.Duration > 0: + cue.EndMs = timeMs + blk.Duration } - if cue.EndMs > cue.StartMs { - if err := a.emit(cue); err != nil { - return err + s := decodedSet{cue: cue} + if ds.Timeout > 0 { + s.latest = start + ds.Timeout.Milliseconds() + if cue.EndMs > s.latest { // the end the block states, when the page's time-out comes first + s.cue.EndMs = s.latest } - continue } - a.open, a.latest = &cue, 0 - if ds.Timeout > 0 { - a.latest = start + ds.Timeout.Milliseconds() + shown = append(shown, s) + } + return shown, nil +} + +// undecodable loses a block that reads but does not decode: the decoder forgets its state, the cue on screen ends +// at the block's time and the block is reported through OnSkip with the decoder's error. A read that refuses any +// damage fails on it. +func (a *bitmapAssembler) undecodable(blk mkv.Block, err error) error { + if a.strict { + return fmt.Errorf("subtitle block at %d ms: %w", blk.Timecode, err) + } + a.dec.Reset() + if err := a.closeAt(blk.Timecode); err != nil { + return err + } + if a.onSkip != nil { + r := mkv.DamagedRange{StartOffset: -1, EndOffset: -1, ApproxStartMs: blk.Timecode, ApproxEndMs: blk.Timecode, Err: err} + if blk.DataOffset >= 0 { // a block of the file; a unit of a file beside it has no byte range here + r.StartOffset, r.EndOffset = blk.DataOffset, blk.DataOffset+int64(len(blk.Data)) } + a.onSkip(r) } return nil } diff --git a/mkv/ops/subbitmap_test.go b/mkv/ops/subbitmap_test.go index b9d4dbe..002d32d 100644 --- a/mkv/ops/subbitmap_test.go +++ b/mkv/ops/subbitmap_test.go @@ -5,6 +5,8 @@ import ( "context" "errors" "image" + "os" + "path/filepath" "strings" "testing" "time" @@ -186,7 +188,7 @@ func TestForEachSubtitleBlock_LostRegion(t *testing.T) { if err != nil { t.Fatalf("%s: %v", tc.name, err) } - if len(skips) != 1 || skips[0].EndOffset != clusters[6] || skips[0].ApproxStartMs != tc.lostMs { + if len(skips) != 1 || skips[0].EndOffset != clusters[6] || skips[0].ApproxStartMs != tc.lostMs || skips[0].Err != nil { t.Errorf("%s: OnSkip got %+v, want once a region from %d ms ending at byte %d", tc.name, skips, tc.lostMs, clusters[6]) } if lost != 1 || lostAt != 4 || times[3] != 3200 || times[4] != tc.nextMs { @@ -381,7 +383,7 @@ func TestBitmapAssembler_CueEnds(t *testing.T) { {30040, 0, "f"}, // completes it: dated 30000, ended by the next set at 31000 {31000, 0, "g"}, // never closed: its time-out, shorter than the default: 31000..33000 } { - if err := a.block(b.timeMs, b.durMs, []byte(b.data)); err != nil { + if err := a.block(mkv.Block{Timecode: b.timeMs, Duration: b.durMs, Data: []byte(b.data)}); err != nil { t.Fatal(err) } } @@ -414,13 +416,13 @@ func TestBitmapAssembler_LostRegion(t *testing.T) { got = append(got, cueShape{startMs: c.StartMs, endMs: c.EndMs}) return nil }} - if err := a.block(1000, 0, []byte("b")); err != nil { + if err := a.block(mkv.Block{Timecode: 1000, Data: []byte("b")}); err != nil { t.Fatal(err) } if err := a.lost(1800); err != nil { t.Fatal(err) } - if err := a.block(9000, 0, []byte("b")); err != nil { + if err := a.block(mkv.Block{Timecode: 9000, Data: []byte("b")}); err != nil { t.Fatal(err) } if err := a.finish(); err != nil { @@ -431,3 +433,234 @@ func TestBitmapAssembler_LostRegion(t *testing.T) { t.Errorf("%d reset(s), cues %+v; want one reset and %+v", dec.resets, got, want) } } + +// zeroedCopy copies a sample with n bytes zeroed from its middle. +func zeroedCopy(t *testing.T, src string, n int) string { + t.Helper() + data, err := os.ReadFile(src) + if err != nil { + t.Fatal(err) + } + clear(data[len(data)/2 : len(data)/2+n]) + path := filepath.Join(t.TempDir(), "zeroed.mkv") + if err := os.WriteFile(path, data, 0o644); err != nil { + t.Fatal(err) + } + return path +} + +// A block that reads but does not decode is lost alone: a VobSub track with 4096 bytes zeroed in the middle of the +// file gives every cue but the two the zeros hit, where it gave none. +func TestForEachSubtitleBitmap_UndecodableBlockIsLostAlone(t *testing.T) { + ctx := context.Background() + bad := zeroedCopy(t, vobsubSampleMKV, 4096) + whole := collectCues(t, func(fn func(BitmapCue) error) error { return ForEachSubtitleBitmap(ctx, vobsubSampleMKV, 1, fn) }) + ix, err := BuildSubtitleIndex(ctx, vobsubSampleMKV, nil) + if err != nil { + t.Fatal(err) + } + for name, read := range map[string]func(fn func(BitmapCue) error, o mkv.Options) error{ + "walk": func(fn func(BitmapCue) error, o mkv.Options) error { return ForEachSubtitleBitmap(ctx, bad, 1, fn, o) }, + "index": func(fn func(BitmapCue) error, o mkv.Options) error { + return ForEachSubtitleBitmapFrom(ctx, bad, 1, ix, fn, o) + }, + } { + var skips []mkv.DamagedRange + var cues []BitmapCue + if err := read(func(c BitmapCue) error { cues = append(cues, c); return nil }, onSkipInto(&skips)); err != nil { + t.Fatalf("%s: %v", name, err) + } + if len(cues) != 41 { + t.Fatalf("%s: %d cues of %d, want 41", name, len(cues), len(whole)) + } + // What OnSkip reports tells the two losses apart: bytes the read jumped over, and a block that was read. + regions, blocks := 0, 0 + for _, r := range skips { + if r.Err == nil { + regions++ + continue + } + blocks++ + if r.ApproxStartMs != 230872 || r.ApproxEndMs != 230872 || !strings.HasPrefix(r.Err.Error(), "VobSub: the control sequence at byte 2979") { + t.Errorf("%s: the block that does not decode is reported at %d ms as %q", name, r.ApproxStartMs, r.Err) + } + } + if regions != 1 || blocks != 1 { + t.Errorf("%s: OnSkip got %d damaged region(s) and %d undecodable block(s), want one of each: %+v", name, regions, blocks, skips) + } + // Every cue that came out is one of the whole track's, pixels included. + byStart := map[int64]BitmapCue{} + for _, c := range whole { + byStart[c.StartMs] = c + } + for _, c := range cues { + w, ok := byStart[c.StartMs] + if !ok || shapeOf(c) != shapeOf(w) || !bytes.Equal(c.Image.Pix, w.Image.Pix) { + t.Errorf("%s: the cue at %d ms is not the whole track's", name, c.StartMs) + } + } + // A read that refuses any damage still fails, on the block and by its time. + err := read(func(BitmapCue) error { return nil }, mkv.Options{DamageBytes: -1}) + if err == nil { + t.Errorf("%s: a read refusing any damage passed over the zeroed bytes", name) + } + } +} + +// pgsDamagedFixture is buildPGSFixture's track with a block that reads but is no display set between its cues. +func pgsDamagedFixture(t *testing.T) string { + t.Helper() + tracks := []mkv.Track{subtitleTrack(1, "pgs")} + blocks := []mkv.Block{ + {TrackNumber: 1, Timecode: 0, Data: pgsShow(10, 100, 900, true)}, // open: ended by the block that does not decode + {TrackNumber: 1, Timecode: 700, Data: []byte{0x16, 0xff, 0xff, 0x00}}, + {TrackNumber: 1, Timecode: 1000, Duration: 400, Data: pgsShow(11, 200, 950, true)}, + {TrackNumber: 1, Timecode: 2000, Data: pgsShow(12, 300, 960, false)}, + } + return buildMinimalMKV(t, t.TempDir(), "pgsbad.mkv", tracks, blocks, 3000) +} + +func TestForEachSubtitleBitmap_UndecodablePGSBlock(t *testing.T) { + ctx := context.Background() + path := pgsDamagedFixture(t) + want := []cueShape{{0, 700, 100, 900, 4, 2}, {1000, 1400, 200, 950, 4, 2}, {2000, 2000 + defaultSubDurationMs, 300, 960, 4, 2}} + ix, err := BuildSubtitleIndex(ctx, path, nil) + if err != nil { + t.Fatal(err) + } + for name, read := range map[string]func(fn func(PGSCue) error, o mkv.Options) error{ + "bitmap walk": func(fn func(PGSCue) error, o mkv.Options) error { return ForEachSubtitleBitmap(ctx, path, 1, fn, o) }, + "pgs walk": func(fn func(PGSCue) error, o mkv.Options) error { return ForEachSubtitlePGS(ctx, path, 1, fn, o) }, + "bitmap index": func(fn func(PGSCue) error, o mkv.Options) error { + return ForEachSubtitleBitmapFrom(ctx, path, 1, ix, fn, o) + }, + "pgs index": func(fn func(PGSCue) error, o mkv.Options) error { + return ForEachSubtitlePGSFrom(ctx, path, 1, ix, fn, o) + }, + } { + var skips []mkv.DamagedRange + var got []cueShape + if err := read(func(c PGSCue) error { got = append(got, shapeOf(c)); return nil }, onSkipInto(&skips)); err != nil { + t.Fatalf("%s: %v", name, err) + } + if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] || got[2] != want[2] { + t.Errorf("%s: cues %+v, want %+v", name, got, want) + } + if len(skips) != 1 || skips[0].ApproxStartMs != 700 || skips[0].ApproxEndMs != 700 || skips[0].EndOffset-skips[0].StartOffset != 4 || skips[0].StartOffset <= 0 { + t.Errorf("%s: OnSkip got %+v, want once the 4 bytes of the block at 700 ms", name, skips) + } else if skips[0].Err == nil || skips[0].Err.Error() != "PGS: a 0x16 segment declares 65535 bytes, 1 remain in the block" { + t.Errorf("%s: the block is reported with the error %v, want the decoder's own", name, skips[0].Err) + } + err := read(func(PGSCue) error { return nil }, mkv.Options{DamageBytes: -1}) + if err == nil || !strings.Contains(err.Error(), "subtitle block at 700 ms: PGS: a 0x16 segment declares 65535 bytes") { + t.Errorf("%s: refusing any damage, err = %v; want the block's own", name, err) + } + // What the caller's function returns stays fatal, also from the cue the lost block closes. + stop := errors.New("stop") + if err := read(func(PGSCue) error { return stop }, mkv.Options{}); err != stop { + t.Errorf("%s: err = %v, want the callback's own", name, err) + } + } + // So does a cancelled context. + cancelled, cancel := context.WithCancel(ctx) + cancel() + if err := ForEachSubtitleBitmap(cancelled, path, 1, func(PGSCue) error { return nil }); !errors.Is(err, context.Canceled) { + t.Errorf("a cancelled read: err = %v, want context.Canceled", err) + } + if err := ForEachSubtitleBitmapFrom(cancelled, path, 1, ix, func(PGSCue) error { return nil }); !errors.Is(err, context.Canceled) { + t.Errorf("a cancelled indexed read: err = %v, want context.Canceled", err) + } +} + +// failingDecoder fails on blocks starting with '!' after handing back the sets it had, as the DVB decoder does. +type failingDecoder struct { + scriptedDecoder +} + +func (d *failingDecoder) Decode(block []byte) ([]subtitle.BitmapDisplaySet, error) { + if block[0] == '!' { + return d.sets['b'], errors.New("a segment that cannot be read") + } + return d.scriptedDecoder.Decode(block) +} + +// The sets a decoder returns with its error are dropped with the block, the decoder is reset and the cue on screen +// ends at the block's time; a display set that cannot be composited is lost the same way. +func TestBitmapAssembler_UndecodableBlock(t *testing.T) { + far := subtitle.BitmapDisplaySet{ScreenW: 720, ScreenH: 576, Objects: []subtitle.BitmapObject{ + {X: 0, Y: 0, Image: image.NewNRGBA(image.Rect(0, 0, 1, 1))}, + {X: bitmapMaxPlane + 1, Y: 0, Image: image.NewNRGBA(image.Rect(0, 0, 1, 1))}, + }} + dec := &failingDecoder{scriptedDecoder{sets: map[byte][]subtitle.BitmapDisplaySet{ + 'b': {shownSet(2, subtitle.BitmapDisplaySet{})}, + 'f': {shownSet(3, subtitle.BitmapDisplaySet{}), far}, + }}} + var got []cueShape + var skips []mkv.DamagedRange + a := &bitmapAssembler{dec: dec, name: "test", track: &mkv.Track{}, onSkip: func(r mkv.DamagedRange) { skips = append(skips, r) }, + emit: func(c BitmapCue) error { + got = append(got, cueShape{startMs: c.StartMs, endMs: c.EndMs, x: c.X}) + return nil + }} + for _, b := range []mkv.Block{ + {Timecode: 1000, Data: []byte("b"), DataOffset: 100}, + {Timecode: 1500, Data: []byte("!x"), DataOffset: 200}, // fails: its sets dropped, the cue of 1000 ends here + {Timecode: 2000, Data: []byte("b"), DataOffset: 300}, + {Timecode: 2600, Data: []byte("fyz"), DataOffset: 400}, // its second set cannot be composited: the whole block is lost + {Timecode: 3000, Data: []byte("b"), DataOffset: 500}, + } { + if err := a.block(b); err != nil { + t.Fatal(err) + } + } + if err := a.finish(); err != nil { + t.Fatal(err) + } + want := []cueShape{{startMs: 1000, endMs: 1500, x: 2}, {startMs: 2000, endMs: 2600, x: 2}, {startMs: 3000, endMs: 3000 + defaultSubDurationMs, x: 2}} + if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] || got[2] != want[2] { + t.Errorf("cues %+v, want %+v", got, want) + } + wantSkips := []mkv.DamagedRange{{StartOffset: 200, EndOffset: 202, ApproxStartMs: 1500, ApproxEndMs: 1500}, {StartOffset: 400, EndOffset: 403, ApproxStartMs: 2600, ApproxEndMs: 2600}} + wantErrs := []string{"a segment that cannot be read", "test: a display set spreads its objects over"} + if dec.resets != 2 || len(skips) != 2 { + t.Fatalf("%d reset(s), OnSkip %+v; want 2 and %+v", dec.resets, skips, wantSkips) + } + for i, r := range skips { + if r.Err == nil || !strings.HasPrefix(r.Err.Error(), wantErrs[i]) { + t.Errorf("skip %d carries the error %v, want one starting with %q", i, r.Err, wantErrs[i]) + } + r.Err = nil + if r != wantSkips[i] { + t.Errorf("skip %d = %+v, want %+v", i, r, wantSkips[i]) + } + } + + a.strict = true + if err := a.block(mkv.Block{Timecode: 4000, Data: []byte("!")}); err == nil || !strings.Contains(err.Error(), "subtitle block at 4000 ms: a segment that cannot be read") { + t.Errorf("refusing any damage, err = %v; want the block's own", err) + } +} + +// A cue whose block states a duration still ends no later than its page's time-out. +func TestBitmapAssembler_BlockDurationPastTheTimeout(t *testing.T) { + dec := &scriptedDecoder{sets: map[byte][]subtitle.BitmapDisplaySet{ + 'c': {shownSet(3, subtitle.BitmapDisplaySet{Timeout: 2 * time.Second})}, + }} + var got []cueShape + a := &bitmapAssembler{dec: dec, name: "test", track: &mkv.Track{}, emit: func(c BitmapCue) error { + got = append(got, cueShape{startMs: c.StartMs, endMs: c.EndMs}) + return nil + }} + for _, b := range []mkv.Block{ + {Timecode: 1000, Duration: 9000, Data: []byte("c")}, // the time-out comes first: 1000..3000 + {Timecode: 20000, Duration: 500, Data: []byte("c")}, // the duration comes first: 20000..20500 + } { + if err := a.block(b); err != nil { + t.Fatal(err) + } + } + want := []cueShape{{startMs: 1000, endMs: 3000}, {startMs: 20000, endMs: 20500}} + if len(got) != 2 || got[0] != want[0] || got[1] != want[1] { + t.Errorf("cues %+v, want %+v", got, want) + } +} diff --git a/mkv/ops/subpgs.go b/mkv/ops/subpgs.go index 9b4a31e..6144215 100644 --- a/mkv/ops/subpgs.go +++ b/mkv/ops/subpgs.go @@ -67,7 +67,7 @@ func ForEachSubtitlePGS(ctx context.Context, srcPath string, trackID uint64, fn if err := oneopt.Check(opts); err != nil { return err } - return walkSubtitleBlocks(ctx, srcPath, trackID, bitmapCues(fn, checkPGSTrack), opts) + return walkSubtitleBlocks(ctx, srcPath, trackID, bitmapCues(fn, checkPGSTrack, opts), opts) } // ExtractSubtitlePGSFrom is ExtractSubtitlePGS served from a prebuilt index: @@ -95,7 +95,7 @@ func ForEachSubtitlePGSFrom(ctx context.Context, srcPath string, trackID uint64, if err := oneopt.Check(opts); err != nil { return err } - return indexedSubtitleBlocks(ctx, srcPath, trackID, ix, bitmapCues(fn, checkPGSTrack), opts) + return indexedSubtitleBlocks(ctx, srcPath, trackID, ix, bitmapCues(fn, checkPGSTrack, opts), opts) } // checkPGSTrack refuses a subtitle track that is not PGS. The message names the codec it did find, because the diff --git a/mkv/ops/subsidecar.go b/mkv/ops/subsidecar.go new file mode 100644 index 0000000..f04145e --- /dev/null +++ b/mkv/ops/subsidecar.go @@ -0,0 +1,140 @@ +package ops + +// subsidecar.go - the bitmap subtitles that sit beside a video, a VobSub .idx/.sub pair or a PGS .sup, as the same +// timed pictures ForEachSubtitleBitmap gives for a track of the file: the readers of mkv/subtitle hand the units and +// display sets over, the assembler of subbitmap.go dates and ends the cues, under the same rules. + +import ( + "context" + "fmt" + "io" + "path/filepath" + "strings" + + "github.com/gravity-zero/mkvgo/internal/oneopt" + "github.com/gravity-zero/mkvgo/mkv" + "github.com/gravity-zero/mkvgo/mkv/subtitle" +) + +// ForEachSubtitleBitmapFile calls fn with each cue of a bitmap subtitle file that sits beside a video, in +// presentation order: a PGS .sup, or a VobSub pair named by its .idx or its .sub, the other found beside it under +// the same name. stream picks the language of a .idx, by its position in it from 0 (VobSubIndex.Streams); a .sup +// holds one, 0. The cues are timed and ended as ForEachSubtitleBitmap times and ends a track's, a unit or a display +// set that does not decode is lost alone and reported through Options.OnSkip (without a byte range: StartOffset and +// EndOffset are -1), and a negative Options.DamageBytes fails the read on it instead. A file the reader cannot go +// through to its end (cut short, a unit the .sub holds in part) gives the cues ahead of the fault, then the error. +// One cue's picture is alive at a time; an error from fn stops the read and is returned as is. +func ForEachSubtitleBitmapFile(ctx context.Context, path string, stream int, fn func(BitmapCue) error, opts ...mkv.Options) error { + if err := oneopt.Check(opts); err != nil { + return err + } + fs := mkv.FSFrom(opts) + switch strings.ToLower(filepath.Ext(path)) { + case ".sup": + if stream != 0 { + return fmt.Errorf("%s holds one subtitle stream, 0, not %d", path, stream) + } + return supCues(ctx, fs, path, fn, opts) + case ".idx", ".sub": + return vobsubPairCues(ctx, fs, path, stream, fn, opts) + } + return fmt.Errorf("%s is not a bitmap subtitle file this reads: a PGS .sup, or a VobSub .idx with its .sub", path) +} + +// supCues reads a .sup display set by display set. +func supCues(ctx context.Context, fs *mkv.FS, path string, fn func(BitmapCue) error, opts []mkv.Options) error { + f, err := fs.DoOpen(path) + if err != nil { + return err + } + defer f.Close() + emit, stopped := callerErrors(fn) + asm, err := newBitmapAssembler(emit, &mkv.Track{Codec: "pgs"}, opts) + if err != nil { + return err + } + readErr := subtitle.ForEachSUPDisplaySet(f, func(timeMs int64, set []byte) error { + if err := ctx.Err(); err != nil { + return err + } + return asm.block(mkv.Block{Timecode: timeMs, Data: set, DataOffset: -1}) + }) + return finishSidecar(ctx, asm, readErr, *stopped) +} + +// vobsubPairCues reads one language of a .idx/.sub pair unit by unit. +func vobsubPairCues(ctx context.Context, fs *mkv.FS, path string, stream int, fn func(BitmapCue) error, opts []mkv.Options) error { + ext := filepath.Ext(path) + base := strings.TrimSuffix(path, ext) + idxPath, subPath := base+".idx", base+".sub" + if ext == strings.ToUpper(ext) { // a pair named in capitals, as DVD tools write it + idxPath, subPath = base+".IDX", base+".SUB" + } + idx, err := fs.DoOpen(idxPath) + if err != nil { + return fmt.Errorf("a VobSub .sub is read with the .idx beside it: %w", err) + } + ix, err := subtitle.ParseVobSubIndex(idx) + idx.Close() + if err != nil { + return err + } + if stream < 0 || stream >= len(ix.Streams) { + return fmt.Errorf("%s holds %d subtitle stream(s), from 0: no stream %d", idxPath, len(ix.Streams), stream) + } + sub, err := fs.DoOpen(subPath) + if err != nil { + return fmt.Errorf("a VobSub .idx is read with the .sub beside it: %w", err) + } + defer sub.Close() + emit, stopped := callerErrors(fn) + asm, err := newBitmapAssembler(emit, &mkv.Track{Codec: "vobsub", CodecPrivate: ix.Header}, opts) + if err != nil { + return err + } + readErr := ix.ForEachUnit(seekReaderAt{sub}, stream, func(timeMs int64, unit []byte) error { + if err := ctx.Err(); err != nil { + return err + } + return asm.block(mkv.Block{Timecode: timeMs, Data: unit, DataOffset: -1}) + }) + return finishSidecar(ctx, asm, readErr, *stopped) +} + +// callerErrors wraps fn so that what it returns is kept: the read tells a fault of the file from the caller's stop. +func callerErrors(fn func(BitmapCue) error) (func(BitmapCue) error, *error) { + stopped := new(error) + return func(c BitmapCue) error { + err := fn(c) + if err != nil { + *stopped = err + } + return err + }, stopped +} + +// finishSidecar gives the cue still on screen once the file is read, or once the reader stopped on a fault of the +// file, whose error then follows; the caller's own error and a cancelled read stop at once. +func finishSidecar(ctx context.Context, asm *bitmapAssembler, readErr, stopped error) error { + if stopped != nil || ctx.Err() != nil { + return readErr + } + if err := asm.finish(); err != nil { + return err + } + return readErr +} + +// seekReaderAt reads at an offset through a file that only seeks, io.EOF for fewer bytes than asked, as os.File.ReadAt. +type seekReaderAt struct{ f io.ReadSeeker } + +func (r seekReaderAt) ReadAt(p []byte, off int64) (int, error) { + if _, err := r.f.Seek(off, io.SeekStart); err != nil { + return 0, err + } + n, err := io.ReadFull(r.f, p) + if err == io.ErrUnexpectedEOF { + err = io.EOF + } + return n, err +} diff --git a/mkv/ops/subsidecar_test.go b/mkv/ops/subsidecar_test.go new file mode 100644 index 0000000..34a215a --- /dev/null +++ b/mkv/ops/subsidecar_test.go @@ -0,0 +1,190 @@ +package ops + +import ( + "bytes" + "context" + "encoding/binary" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/gravity-zero/mkvgo/mkv" + "github.com/gravity-zero/mkvgo/mkv/subtitle" +) + +const ( + vobsubSampleIdx = "../../internal/testdata/vobsub.idx" + vobsubSampleSub = "../../internal/testdata/vobsub.sub" + pgsSampleSup = "../../internal/testdata/pgs_sub.sup" +) + +// copyPair copies the sample .idx/.sub pair into dir under name, the .sub through edit when it is set. +func copyPair(t *testing.T, dir, name string, edit func(sub []byte)) string { + t.Helper() + for _, f := range []struct{ from, ext string }{{vobsubSampleIdx, ".idx"}, {vobsubSampleSub, ".sub"}} { + data, err := os.ReadFile(f.from) + if err != nil { + t.Fatal(err) + } + if f.ext == ".sub" && edit != nil { + edit(data) + } + if err := os.WriteFile(filepath.Join(dir, name+f.ext), data, 0o644); err != nil { + t.Fatal(err) + } + } + return filepath.Join(dir, name+".idx") +} + +// The pair the VobSub sample was muxed from gives the cues its Matroska track gives, pixels included, named by its +// .idx or by its .sub. +func TestForEachSubtitleBitmapFile_VobSubPair(t *testing.T) { + ctx := context.Background() + want := collectCues(t, func(fn func(BitmapCue) error) error { return ForEachSubtitleBitmap(ctx, vobsubSampleMKV, 1, fn) }) + dir := t.TempDir() + idx := copyPair(t, dir, "film", nil) + for _, path := range []string{idx, strings.TrimSuffix(idx, ".idx") + ".sub"} { + got := collectCues(t, func(fn func(BitmapCue) error) error { return ForEachSubtitleBitmapFile(ctx, path, 0, fn) }) + sameCues(t, got, want, filepath.Base(path)) + } +} + +// A .sup: two display sets of two objects each, the second cut short by the end of the file without its end +// segment, which is given as it stands and closed by the default duration. +func TestForEachSubtitleBitmapFile_Sup(t *testing.T) { + got := collectCues(t, func(fn func(BitmapCue) error) error { + return ForEachSubtitleBitmapFile(context.Background(), pgsSampleSup, 0, fn) + }) + want := []cueShape{ + {67, 401, 100, 956, 1439, 67}, // two objects, composited on the rectangle that holds them + {401, 401 + defaultSubDurationMs, 100, 956, 1439, 67}, + } + if len(got) != len(want) { + t.Fatalf("%d cues, want %d", len(got), len(want)) + } + for i := range want { + if s := shapeOf(got[i]); s != want[i] || got[i].ScreenW != 1920 || got[i].ScreenH != 1080 { + t.Errorf("cue %d = %+v on %dx%d, want %+v on 1920x1080", i, s, got[i].ScreenW, got[i].ScreenH, want[i]) + } + } +} + +// unitStart is the offset in a .sub of the subpicture unit the pack at pos starts: past the pack header, the +// private stream packet's header and its substream byte. +func unitStart(t *testing.T, sub []byte, pos int64) int { + t.Helper() + p := int(pos) + if !bytes.Equal(sub[p:p+4], []byte{0, 0, 1, 0xba}) { + t.Fatalf("no pack at %d", p) + } + p += 14 + int(sub[p+13]&7) + if !bytes.Equal(sub[p:p+4], []byte{0, 0, 1, 0xbd}) { + t.Fatalf("no private stream packet at %d", p) + } + return p + 6 + 3 + int(sub[p+8]) + 1 +} + +// A unit of the pair that does not decode is lost alone, reported with its time and error and no byte range; a read +// that refuses any damage fails on it. +func TestForEachSubtitleBitmapFile_UndecodableUnit(t *testing.T) { + ctx := context.Background() + idxData, err := os.ReadFile(vobsubSampleIdx) + if err != nil { + t.Fatal(err) + } + ix, err := subtitle.ParseVobSubIndex(bytes.NewReader(idxData)) + if err != nil { + t.Fatal(err) + } + e := ix.Streams[0].Entries[10] + idx := copyPair(t, t.TempDir(), "bad", func(sub []byte) { + u := unitStart(t, sub, e.Pos) + binary.BigEndian.PutUint16(sub[u+2:], 2) // control sequences at byte 2: inside the header + }) + var skips []mkv.DamagedRange + var got []BitmapCue + err = ForEachSubtitleBitmapFile(ctx, idx, 0, func(c BitmapCue) error { got = append(got, c); return nil }, onSkipInto(&skips)) + if err != nil { + t.Fatal(err) + } + if len(got) != 42 { + t.Errorf("%d cues, want the 42 of the 43 that decode", len(got)) + } + if len(skips) != 1 || skips[0].ApproxStartMs != e.TimeMs || skips[0].StartOffset != -1 || skips[0].EndOffset != -1 || + skips[0].Err == nil || !strings.Contains(skips[0].Err.Error(), "places its control sequences at byte 2") { + t.Errorf("OnSkip got %+v, want once the unit at %d ms, its error, no byte range", skips, e.TimeMs) + } + err = ForEachSubtitleBitmapFile(ctx, idx, 0, func(BitmapCue) error { return nil }, mkv.Options{DamageBytes: -1}) + if err == nil || !strings.Contains(err.Error(), "places its control sequences at byte 2") { + t.Errorf("refusing any damage: err = %v, want the unit's own", err) + } +} + +// A file the reader cannot go through gives the cues ahead of the fault, then its error; the caller's own error +// stops the read at once. +func TestForEachSubtitleBitmapFile_FaultsAndStops(t *testing.T) { + ctx := context.Background() + data, err := os.ReadFile(pgsSampleSup) + if err != nil { + t.Fatal(err) + } + cut := filepath.Join(t.TempDir(), "cut.sup") + second := bytes.Index(data[13:], []byte("PG")) + 13 // the second segment of the file, inside the first display set + for end := len(data) - 1; end > second; end-- { // cut inside the last segment of the second display set + if !bytes.Equal(data[end-1:end+1], []byte("PG")) { + continue + } + if err := os.WriteFile(cut, data[:end+20], 0o644); err != nil { + t.Fatal(err) + } + break + } + var got []BitmapCue + err = ForEachSubtitleBitmapFile(ctx, cut, 0, func(c BitmapCue) error { got = append(got, c); return nil }) + if err == nil || !strings.Contains(err.Error(), "cut short") || len(got) != 1 || got[0].StartMs != 67 || got[0].EndMs != 67+defaultSubDurationMs { + t.Errorf("a .sup cut short: %d cue(s) %+v, err %v; want the first display set, then the error", len(got), got, err) + } + + stop := errors.New("stop") + calls := 0 + halt := func(BitmapCue) error { calls++; return stop } + if err := ForEachSubtitleBitmapFile(ctx, pgsSampleSup, 0, halt); err != stop || calls != 1 { + t.Errorf(".sup: err = %v after %d call(s), want the callback's own after one", err, calls) + } + calls = 0 + idx := copyPair(t, t.TempDir(), "film", nil) + if err := ForEachSubtitleBitmapFile(ctx, idx, 0, halt); err != stop || calls != 1 { + t.Errorf("pair: err = %v after %d call(s), want the callback's own after one", err, calls) + } + cancelled, cancel := context.WithCancel(ctx) + cancel() + if err := ForEachSubtitleBitmapFile(cancelled, idx, 0, func(BitmapCue) error { return nil }); !errors.Is(err, context.Canceled) { + t.Errorf("a cancelled read: err = %v", err) + } +} + +func TestForEachSubtitleBitmapFile_Refusals(t *testing.T) { + ctx := context.Background() + dir := t.TempDir() + idx := copyPair(t, dir, "film", nil) + if err := os.WriteFile(filepath.Join(dir, "alone.idx"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + none := func(BitmapCue) error { return nil } + for name, tc := range map[string]struct { + err error + want string + }{ + "no such stream": {ForEachSubtitleBitmapFile(ctx, idx, 1, none), "holds 1 subtitle stream(s), from 0: no stream 1"}, + "a .sup's stream 1": {ForEachSubtitleBitmapFile(ctx, pgsSampleSup, 1, none), "holds one subtitle stream, 0, not 1"}, + "another file": {ForEachSubtitleBitmapFile(ctx, vobsubSampleMKV, 0, none), "is not a bitmap subtitle file this reads"}, + "a .sub without .idx": {ForEachSubtitleBitmapFile(ctx, filepath.Join(dir, "lost.sub"), 0, none), "read with the .idx beside it"}, + "a .idx without .sub": {ForEachSubtitleBitmapFile(ctx, filepath.Join(dir, "alone.idx"), 0, none), "names no stream"}, + } { + if tc.err == nil || !strings.Contains(tc.err.Error(), tc.want) { + t.Errorf("%s: err = %v, want one naming %q", name, tc.err, tc.want) + } + } +} diff --git a/mkv/progress.go b/mkv/progress.go index f7c3525..49d54ab 100644 --- a/mkv/progress.go +++ b/mkv/progress.go @@ -20,6 +20,10 @@ type DamagedRange struct { EndOffset int64 `json:"end_offset"` ApproxStartMs int64 `json:"approx_start_ms"` ApproxEndMs int64 `json:"approx_end_ms"` + // Err is set when the range is one block that was read but did not decode, where a damaged region is bytes a + // read jumped over: the decoder's error. The block's time is ApproxStartMs; StartOffset and EndOffset are -1 + // for a unit of a subtitle file read beside the video, which has no byte range there. + Err error `json:"-"` } // RepairedRange records one region that a tolerant walk (ops.Salvage, or diff --git a/mkv/subtitle/bitmap_cache_test.go b/mkv/subtitle/bitmap_cache_test.go new file mode 100644 index 0000000..b097540 --- /dev/null +++ b/mkv/subtitle/bitmap_cache_test.go @@ -0,0 +1,167 @@ +package subtitle + +import ( + "image" + "runtime" + "testing" +) + +// allocated runs fn and returns the bytes it allocated. +func allocated(fn func()) uint64 { + var before, after runtime.MemStats + runtime.GC() + runtime.ReadMemStats(&before) + fn() + runtime.ReadMemStats(&after) + return after.TotalAlloc - before.TotalAlloc +} + +// A display set that only names a region again costs nothing of the region: 50 page-only blocks of 14 bytes each +// made a 2048x2048 region be read through its CLUT 50 times, 16 MiB a time. +func TestDVBSubUnchangedRegionIsNotRenderedAgain(t *testing.T) { + d := newDVBSub(t) + at := dvbPlacement{region: 0, x: 10, y: 20} + first := decodeOneDVB(t, d, concat(dvbPCS(2, at), dvbRCS(0, true, 2048, 2048, 3, 0, 1), dvbEDS())) + if len(first.Objects) != 1 { + t.Fatalf("%d objects, want the region", len(first.Objects)) + } + page := dvbPCS(0, at) // a page composition and nothing else: the display set ends with the block + if len(page) != 14 { + t.Fatalf("the page-only block is %d bytes, want 14", len(page)) + } + var last BitmapDisplaySet + alloc := allocated(func() { + for i := 0; i < 50; i++ { + last = decodeOneDVB(t, d, page) + } + }) + if len(last.Objects) != 1 || last.Objects[0].Image != first.Objects[0].Image { + t.Fatalf("a display set naming the unchanged region again does not hand back its picture") + } + if alloc > 1<<20 { + t.Errorf("50 page-only blocks allocated %d KiB, want next to nothing: the region did not change", alloc>>10) + } +} + +// The picture kept for a region is dropped as soon as what it shows changes: its pixels, or its colours. +func TestDVBSubRegionPictureFollowsItsPixelsAndColours(t *testing.T) { + at := dvbPlacement{region: 0} + show := concat(dvbPCS(2, at), dvbRCS(0, true, 4, 2, 2, 0, 0, dvbObjectRef{id: 7}), dvbTestCLUT, dvbODS(7, false, dvbTop4x2, dvbBottom4x2), dvbEDS()) + page := concat(dvbPCS(0, at), dvbEDS()) + white, red := [4]uint8{255, 255, 255, 255}, [4]uint8{} + red[0], red[1], red[2] = ycrcbToRGB(81, 240, 90) + red[3] = 255 + for name, tc := range map[string]struct { + change []byte + want [4]uint8 // the top-left pixel afterwards + }{ + "a CLUT entry redefined": {dvbCDS(0, [6]byte{1, 0x41, 81, 240, 90, 0}), red}, + "an object drawn over": {dvbODS(7, false, []byte{dvbData4Bit, 0x22, 0x22, 0x00, dvbDataEndRow}, nil), [4]uint8{}}, + "the region filled": {dvbRCS(0, true, 4, 2, 2, 0, 1), white}, + "another CLUT": {concat(dvbCDS(5, [6]byte{1, 0x41, 81, 240, 90, 0}), dvbRCS(0, false, 4, 2, 2, 5, 0, dvbObjectRef{id: 7})), red}, + } { + d := newDVBSub(t) + before := decodeOneDVB(t, d, show) + if got := nrgbaAt(t, before.Objects[0].Image, 0, 0); got != white { + t.Fatalf("%s: the top-left pixel starts as %v, want white", name, got) + } + after := decodeOneDVB(t, d, concat(tc.change, page)) + if len(after.Objects) != 1 || after.Objects[0].Image == before.Objects[0].Image { + t.Errorf("%s: the display set after the change shows the picture from before it", name) + continue + } + want := tc.want + if want == ([4]uint8{}) { // the object drawn over paints entry 2: the test CLUT's blue + want[0], want[1], want[2] = ycrcbToRGB(82, 90, 240) + want[3] = 255 + } + if got := nrgbaAt(t, after.Objects[0].Image, 0, 0); got != want { + t.Errorf("%s: the top-left pixel is %v, want %v", name, got, want) + } + if got := nrgbaAt(t, before.Objects[0].Image, 0, 0); got != white { + t.Errorf("%s: the picture handed back before the change was written over: %v", name, got) + } + } +} + +// A composition that names an object again costs nothing of the object: 50 blocks of a composition and an end +// segment made a 1024x1024 object be expanded 50 times, 4 MiB a time. +func TestPGSUnchangedObjectIsNotDecodedAgain(t *testing.T) { + d := NewPGSDecoder() + show := concat(pgsPCS(1920, 1080, pgsEpochStart, 0, synthObject{id: 1, x: 10, y: 20}), pgsPDS(0, palWhite), + pgsODS(1, 1024, 1024, []byte{0x00, 0xc4, 0x00, 0x01, 0x00, 0x00}), pgsEND()) + again := concat(pgsPCS(1920, 1080, 0, 0, synthObject{id: 1, x: 30, y: 40}), pgsEND()) + first, err := d.Decode(show) + if err != nil || len(first) != 1 || len(first[0].Objects) != 1 { + t.Fatalf("%d sets, err %v", len(first), err) + } + var last []PGSDisplaySet + alloc := allocated(func() { + for i := 0; i < 50; i++ { + if last, err = d.Decode(again); err != nil { + t.Fatal(err) + } + } + }) + if len(last) != 1 || len(last[0].Objects) != 1 || last[0].Objects[0].Image != first[0].Objects[0].Image || last[0].Objects[0].X != 30 { + t.Fatalf("a composition naming the unchanged object again does not hand back its picture, placed anew") + } + if alloc > 1<<20 { + t.Errorf("50 compositions of an unchanged object allocated %d KiB, want next to nothing", alloc>>10) + } +} + +// The picture kept for an object is dropped when the object, its palette or the crop asked of it changes. +func TestPGSObjectPictureFollowsItsDataAndPalette(t *testing.T) { + show := concat(pgsPCS(1920, 1080, pgsEpochStart, 0, synthObject{id: 1}), pgsPDS(0, palWhite, palBlue), pgsODS(1, 4, 2, rle4x2), pgsEND()) + again := concat(pgsPCS(1920, 1080, 0, 0, synthObject{id: 1}), pgsEND()) + white := [4]uint8{255, 255, 255, 255} + var blue [4]uint8 + blue[0], blue[1], blue[2] = ycrcbToRGB(82, 90, 240) + blue[3] = 255 + crop := image.Rect(2, 0, 4, 2) + for name, tc := range map[string]struct { + change []byte + then []byte + want [4]uint8 // the top-left pixel afterwards + }{ + "a palette entry redefined": {pgsPDS(0, [5]byte{1, 82, 90, 240, 255}), again, blue}, + "the object redefined": {pgsODS(1, 4, 2, []byte{0x00, 0x84, 0x02, 0x00, 0x00, 0x00, 0x84, 0x02, 0x00, 0x00}), again, blue}, + "another palette": {pgsPDS(3, [5]byte{1, 82, 90, 240, 255}), concat(pgsPCS(1920, 1080, 0, 3, synthObject{id: 1}), pgsEND()), blue}, + "a crop of it": {nil, concat(pgsPCS(1920, 1080, 0, 0, synthObject{id: 1, crop: &crop}), pgsEND()), white}, + } { + d := NewPGSDecoder() + before, err := d.Decode(show) + if err != nil || len(before) != 1 { + t.Fatal(err) + } + after, err := d.Decode(concat(tc.change, tc.then)) + if err != nil || len(after) != 1 || len(after[0].Objects) != 1 { + t.Fatalf("%s: %d sets, err %v", name, len(after), err) + } + if after[0].Objects[0].Image == before[0].Objects[0].Image { + t.Errorf("%s: the display set after the change shows the picture from before it", name) + } + if got := nrgbaAt(t, after[0].Objects[0].Image, 0, 0); got != tc.want { + t.Errorf("%s: the top-left pixel is %v, want %v", name, got, tc.want) + } + if got := nrgbaAt(t, before[0].Objects[0].Image, 0, 0); got != white { + t.Errorf("%s: the picture handed back before the change was written over: %v", name, got) + } + } +} + +// What the decoder keeps of pictures is bounded: objects that together decode past one full plane are not all kept. +func TestPGSKeptPicturesBounded(t *testing.T) { + d := NewPGSDecoder() + parts := [][]byte{pgsPCS(1920, 1080, pgsEpochStart, 0, synthObject{id: 1}, synthObject{id: 2}, synthObject{id: 3}), pgsPDS(0, palWhite)} + for id := uint16(1); id <= 3; id++ { + parts = append(parts, pgsODS(id, 4096, 2048, []byte{0, 0})) + } + if _, err := d.Decode(concat(append(parts, pgsEND())...)); err != nil { + t.Fatal(err) + } + if d.keptPixels > pgsMaxKeptPixels { + t.Errorf("the decoder keeps %d pixels of pictures, past its bound of %d", d.keptPixels, pgsMaxKeptPixels) + } +} diff --git a/mkv/subtitle/dvbsub.go b/mkv/subtitle/dvbsub.go index 1755e22..206ca2a 100644 --- a/mkv/subtitle/dvbsub.go +++ b/mkv/subtitle/dvbsub.go @@ -57,6 +57,7 @@ type dvbCLUT struct { c4 [4][4]uint8 c16 [16][4]uint8 c256 [256][4]uint8 + gen uint64 // moves each time a definition changes an entry: a picture read through it is then out of date } var dvbDefaultCLUT = newDVBDefaultCLUT() @@ -116,6 +117,17 @@ type dvbRegion struct { clut uint8 pix []byte objects []dvbObjectRef + // shown is the picture last read from the region, handed back until its pixels or its colours change; nil + // once a pixel is written. It holds up to four bytes for each of the region's. + shown *dvbPicture +} + +// dvbPicture is a region as a display set shows it, and the CLUT it was read through. +type dvbPicture struct { + img *image.NRGBA // nil when the region shows nothing + at image.Point // where the picture sits in the region + clut *dvbCLUT + gen uint64 } type dvbPlacement struct { @@ -154,8 +166,9 @@ func (d *DVBSubDecoder) Reset() { // Decode decodes one Matroska block into the display sets it completes, almost always one: a set's objects are the // regions on screen, each cut down to its visible pixels, and a set with none clears the screen; its Timeout is the -// page's, the end of a set being the next one. The images are freshly allocated and owned by the caller; a region -// a page places more than once shares one image between its placements. Bytes that are no +// page's, the end of a set being the next one. An image is shared by every placement, and by every display set, +// that shows the same picture: a region is read once and its image handed back for as long as neither its pixels +// nor its colours change. Treat the images as read-only. Bytes that are no // segment end the block. A segment that cannot be read is an error, returned with the sets completed ahead of it // and the one being composed, which show the state the decoder is left in. // @@ -341,6 +354,9 @@ func (d *DVBSubDecoder) parseRegion(p []byte) error { } d.held = held r.clut, r.objects = p[7], refs + if fill { + r.shown = nil + } if fill && (!fresh || background != 0) { for i := range r.pix { r.pix[i] = background @@ -392,6 +408,7 @@ func (d *DVBSubDecoder) parseCLUT(p []byte) error { if flags&0x20 != 0 { c.c256[entry] = col } + c.gen++ } return nil } @@ -424,6 +441,7 @@ func (d *DVBSubDecoder) parseObject(p []byte) error { if ref.id != id { continue } + r.shown = nil pen := dvbPen{d: d, r: r, skipOne: p[2]&2 != 0} if err := pen.field(top, ref.x, ref.y); err != nil { return fmt.Errorf("DVB subtitle: object %d: %w", id, err) @@ -653,35 +671,33 @@ func (p *dvbPen) string8(data []byte) int { } // compose turns the page into a display set: each region on screen, read through its CLUT, where the page puts it. -// A region the page places several times is read once, its placements sharing the picture. +// A region is read once for as long as it shows the same thing: the placements of one display set share its +// picture, and so do the display sets that follow until its pixels or its colours change. func (d *DVBSubDecoder) compose() (BitmapDisplaySet, error) { ds := BitmapDisplaySet{ScreenW: d.screenW, ScreenH: d.screenH, Timeout: d.timeout} if ds.ScreenW == 0 { // no display definition: the format's standard-definition plane ds.ScreenW, ds.ScreenH = 720, 576 } - type picture struct { - img *image.NRGBA // nil when the region shows nothing - at image.Point - } - var seen map[uint8]picture for _, pl := range d.page { - pic, ok := seen[pl.region] - if !ok { - r := d.regions[pl.region] - if r == nil { - if err := d.spend(1); err != nil { - return ds, err - } - continue + r := d.regions[pl.region] + if r == nil { + if err := d.spend(1); err != nil { + return ds, err } + continue + } + clut := d.cluts[r.clut] + if clut == nil { + clut = dvbDefaultCLUT + } + pic := r.shown + if pic == nil || pic.clut != clut || pic.gen != clut.gen { if err := d.spend(r.w * r.h); err != nil { return ds, err } - pic.img, pic.at = trimTransparent(d.render(r)) - if seen == nil { - seen = make(map[uint8]picture, len(d.page)) - } - seen[pl.region] = pic + pic = &dvbPicture{clut: clut, gen: clut.gen} + pic.img, pic.at = trimTransparent(renderDVBRegion(r, clut)) + r.shown = pic } else if err := d.spend(1); err != nil { return ds, err } @@ -692,12 +708,8 @@ func (d *DVBSubDecoder) compose() (BitmapDisplaySet, error) { return ds, nil } -// render reads a region's pixels through its CLUT, or the default one when the epoch defined none of that ID. -func (d *DVBSubDecoder) render(r *dvbRegion) *image.NRGBA { - clut := d.cluts[r.clut] - if clut == nil { - clut = dvbDefaultCLUT - } +// renderDVBRegion reads a region's pixels through a CLUT. +func renderDVBRegion(r *dvbRegion, clut *dvbCLUT) *image.NRGBA { colours := clut.c256[:] switch r.depth { case 4: diff --git a/mkv/subtitle/pgs.go b/mkv/subtitle/pgs.go index ba49ec8..a17e528 100644 --- a/mkv/subtitle/pgs.go +++ b/mkv/subtitle/pgs.go @@ -63,6 +63,8 @@ const ( // pgsMaxBlockPixels bounds the pixels one block's display sets decode: four 4096x4096 objects, 256 MiB. A real // block places one or two subtitle-sized objects; only a forged one names objects over and over. pgsMaxBlockPixels = 4 * pgsMaxObjectDim * pgsMaxObjectDim + // pgsMaxKeptPixels bounds the pictures kept from one display set to the next: one 4096x4096 object, 64 MiB. + pgsMaxKeptPixels = pgsMaxObjectDim * pgsMaxObjectDim ) // PGSObject is one picture of a display set, already composed: the palette is @@ -106,7 +108,28 @@ func (ds *PGSDisplaySet) Clear() bool { return len(ds.Objects) == 0 } // pgsPalette holds a palette's 256 entries, already converted to straight RGBA. // Entries default to fully transparent: a PDS updates only the entries it // carries, and an index the palette never defined must not paint anything. -type pgsPalette [256][4]uint8 +type pgsPalette struct { + rgba [256][4]uint8 + gen uint64 // moves each time a definition changes an entry: a picture rendered under it is then out of date +} + +// pgsNoPalette stands for a palette the epoch never defined: every entry transparent. +var pgsNoPalette = &pgsPalette{} + +// pgsPick names what a composition asks of an object: the object, or one crop of it. +type pgsPick struct { + id uint16 + cropped bool + crop image.Rectangle +} + +// pgsPicture is a picture the decoder keeps: what it was rendered from, to tell when it is out of date. +type pgsPicture struct { + def *pgsObjectDef + pal *pgsPalette + gen uint64 + img *image.NRGBA +} // pgsObjectDef is a defined object, kept as its RLE rather than as a decoded // bitmap. Two reasons: an epoch's objects stay alive across display sets, and @@ -128,6 +151,10 @@ type PGSDecoder struct { pendID uint16 objBytes int // what the epoch's objects hold (pgsMaxEpochBytes) pixels int // what the block being decoded may still decode (pgsMaxBlockPixels) + // kept holds the picture last rendered for each object or crop of one, handed back as long as neither the + // object nor its palette changed; keptPixels is what they hold together (pgsMaxKeptPixels). + kept map[pgsPick]pgsPicture + keptPixels int } // NewPGSDecoder returns a decoder with no epoch state. @@ -138,14 +165,16 @@ func NewPGSDecoder() *PGSDecoder { return &PGSDecoder{} } // against pictures that belong to a different part of the file. func (d *PGSDecoder) Reset() { d.palettes, d.objects, d.pending, d.objBytes = nil, nil, nil, 0 + d.kept, d.keptPixels = nil, 0 } // Decode decodes one Matroska block into the display sets it holds. Almost // always that is exactly one; the slice exists because the format allows a // block to carry several, and dropping the extras would silently lose cues. // -// The returned images are freshly allocated and owned by the caller; an object a display set places more than once -// shares one image between its placements. +// An image is shared by every placement, and by every display set, that shows the same picture: an object, or a +// crop of one, is rendered once and its image handed back for as long as neither the object nor its palette +// changes. Treat the images as read-only. func (d *PGSDecoder) Decode(block []byte) ([]PGSDisplaySet, error) { var out []PGSDisplaySet var comp *pgsComposition @@ -183,7 +212,7 @@ func (d *PGSDecoder) Decode(block []byte) ([]PGSDisplaySet, error) { // An epoch start invalidates everything carried over. It comes // BEFORE the PDS/ODS of its own display set, so clearing here // discards the old epoch without touching the new one. - d.palettes, d.objects, d.pending, d.objBytes = nil, nil, nil, 0 + d.Reset() } comp = c case pgsSegPDS: @@ -328,7 +357,10 @@ func (d *PGSDecoder) parsePalette(p []byte) error { for off := 0; off < len(entries); off += entryLen { e := entries[off : off+entryLen] r, g, bl := ycrcbToRGB(e[1], e[2], e[3]) - pal[e[0]] = [4]uint8{r, g, bl, e[4]} + pal.rgba[e[0]] = [4]uint8{r, g, bl, e[4]} + } + if len(entries) > 0 { + pal.gen++ } return nil } @@ -414,49 +446,51 @@ func (d *PGSDecoder) compose(c *pgsComposition) (PGSDisplaySet, error) { // A composition against a palette this epoch never defined paints // nothing at all. Treating it as a clear is the honest reading - there // is no colour to invent - and it keeps the timeline consistent. - pal = &pgsPalette{} + pal = pgsNoPalette } - // A composition may name one object, or one crop of it, many times: each is decoded once, its references - // sharing the bitmap, so a few bytes naming an object again never cost another picture. - type pick struct { - id uint16 - crop image.Rectangle - } - decoded := map[pick]*image.NRGBA{} for _, co := range c.objects { - key := pick{id: co.objectID} - if co.cropped { - key.crop = image.Rect(co.cropX, co.cropY, co.cropX+co.cropW, co.cropY+co.cropH) - } - if img, ok := decoded[key]; ok { - if img != nil { - ds.Objects = append(ds.Objects, PGSObject{X: co.x, Y: co.y, Forced: co.forced, Image: img}) - } - continue - } - decoded[key] = nil def := d.objects[co.objectID] - if def != nil { - if d.pixels -= def.w * def.h; d.pixels < 0 { - return ds, fmt.Errorf("PGS: the block's display sets decode to more than %d pixels: the track is damaged or forged", pgsMaxBlockPixels) - } - } if def == nil { // The object was defined in a part of the epoch the caller did not // read (a mid-file start). Skipping keeps the rest of the display // set; inventing a picture would be worse than a missing one. continue } - img, err := decodePGSRLE(def, pal) + key := pgsPick{id: co.objectID, cropped: co.cropped} + if co.cropped { + key.crop = image.Rect(co.cropX, co.cropY, co.cropX+co.cropW, co.cropY+co.cropH) + } + img, err := d.picture(key, def, pal) if err != nil { return ds, fmt.Errorf("PGS: object %d: %w", co.objectID, err) } - x, y := co.x, co.y - if co.cropped { - r := image.Rect(co.cropX, co.cropY, co.cropX+co.cropW, co.cropY+co.cropH).Intersect(img.Bounds()) - if r.Empty() { - continue - } + if img != nil { + ds.Objects = append(ds.Objects, PGSObject{X: co.x, Y: co.y, Forced: co.forced, Image: img}) + } + } + return ds, nil +} + +// picture renders an object, or the crop of it a composition asks for, under a palette; nil when the crop holds +// nothing of the object. A composition may name one object many times, and display set after display set: the +// picture is kept and handed back until the object or the palette changes, so a few bytes naming an object again +// never cost another picture. +func (d *PGSDecoder) picture(key pgsPick, def *pgsObjectDef, pal *pgsPalette) (*image.NRGBA, error) { + if p, ok := d.kept[key]; ok && p.def == def && p.pal == pal && p.gen == pal.gen { + return p.img, nil + } + if d.pixels -= def.w * def.h; d.pixels < 0 { + return nil, fmt.Errorf("the block's display sets decode to more than %d pixels: the track is damaged or forged", pgsMaxBlockPixels) + } + img, err := decodePGSRLE(def, pal) + if err != nil { + return nil, err + } + if key.cropped { + r := key.crop.Intersect(img.Bounds()) + if r.Empty() { + img = nil + } else { // SubImage shares the backing array; re-origin it at 0,0 so the // caller gets a plain top-left-anchored picture and X/Y stay the // composition's own coordinates. @@ -468,10 +502,28 @@ func (d *PGSDecoder) compose(c *pgsComposition) (PGSDisplaySet, error) { } img = cropped } - decoded[key] = img - ds.Objects = append(ds.Objects, PGSObject{X: x, Y: y, Forced: co.forced, Image: img}) } - return ds, nil + d.keep(key, pgsPicture{def: def, pal: pal, gen: pal.gen, img: img}) + return img, nil +} + +// keep holds a picture for the display sets to come, dropping the ones held when together they pass the bound. +func (d *PGSDecoder) keep(key pgsPick, p pgsPicture) { + size := 0 + if p.img != nil { + size = p.img.Rect.Dx() * p.img.Rect.Dy() + } + if old, ok := d.kept[key]; ok && old.img != nil { + d.keptPixels -= old.img.Rect.Dx() * old.img.Rect.Dy() + } + if d.keptPixels+size > pgsMaxKeptPixels { + d.kept, d.keptPixels = nil, 0 + } + if d.kept == nil { + d.kept = make(map[pgsPick]pgsPicture, 2) + } + d.kept[key] = p + d.keptPixels += size } // decodePGSRLE expands one object's run-length data straight into an NRGBA @@ -493,7 +545,7 @@ func decodePGSRLE(def *pgsObjectDef, pal *pgsPalette) (*image.NRGBA, error) { x, y := 0, 0 paint := func(idx byte, run int) { - c := pal[idx] + c := pal.rgba[idx] for run > 0 && y < def.h { if x >= def.w { x, y = 0, y+1 diff --git a/mp4/exotic.go b/mp4/exotic.go index b3efc75..68cbcdf 100644 --- a/mp4/exotic.go +++ b/mp4/exotic.go @@ -227,7 +227,7 @@ func ticksOf(ms int64, timescale uint32) int64 { // not state; media bytes are never read. func fragmentEnds(r io.ReadSeeker, size int64, moovBoxes []memBox, mv *movie) { trex := parseTrexDefaults(moovBoxes) - ends := map[uint32]int64{} + ends, starts := map[uint32]int64{}, map[uint32]int64{} var hdr [16]byte for off := int64(0); off+8 <= size; { if _, err := r.Seek(off, io.SeekStart); err != nil { @@ -257,7 +257,7 @@ func fragmentEnds(r io.ReadSeeker, size int64, moovBoxes []memBox, mv *movie) { if err != nil { break } - moofEnds(payload, trex, ends) + moofEnds(payload, trex, ends, starts) } off += boxSize } @@ -268,12 +268,17 @@ func fragmentEnds(r io.ReadSeeker, size int64, moovBoxes []memBox, mv *movie) { } endMs := ticksToMs(end, t.timescale) t.durationMs = max(t.durationMs, endMs) - mv.durationMs = max(mv.durationMs, endMs) + t.fragEndTicks = max(t.fragEndTicks, end) + if off, ok := starts[id]; ok && !t.fragStarted { + t.fragStartOff, t.fragStarted = off, true + } + mv.durationMs = max(mv.durationMs, endMs, t.shownEndMs()) // the movie ends with its latest track } } -// moofEnds raises each track's decode end from the trafs of one moof. -func moofEnds(moof []byte, trex map[uint32]trexDefault, ends map[uint32]int64) { +// moofEnds raises each track's decode end from the trafs of one moof, and records in starts the composition offset +// of the first sample a track's fragments hold. +func moofEnds(moof []byte, trex map[uint32]trexDefault, ends, starts map[uint32]int64) { boxes, err := iterBoxes(moof) if err != nil { return @@ -319,6 +324,11 @@ func moofEnds(moof []byte, trex map[uint32]trexDefault, ends map[uint32]int64) { if err != nil { return } + if _, seen := starts[h.trackID]; !seen { + if off, ok := trunFirstOffset(tb.payload); ok { + starts[h.trackID] = off + } + } dur += d } } @@ -328,6 +338,25 @@ func moofEnds(moof []byte, trex map[uint32]trexDefault, ends map[uint32]int64) { } } +// trunFirstOffset is the composition offset of the first sample of a trun that trunSpan accepted: 0 when its +// samples carry none, not ok when it holds no sample. +func trunFirstOffset(p []byte) (int64, bool) { + flags := binary.BigEndian.Uint32(p[0:4]) & 0xFFFFFF + if binary.BigEndian.Uint32(p[4:8]) == 0 { + return 0, false + } + if flags&trunSampleCTS == 0 { + return 0, true + } + at := 8 + for _, f := range []uint32{trunDataOffset, trunFirstSampleFlag, trunSampleDuration, trunSampleSize, trunSampleFlags} { + if flags&f != 0 { + at += 4 + } + } + return int64(int32(binary.BigEndian.Uint32(p[at:]))), true // signed in both versions, as parseMoofSamples reads it +} + // dropReason is why a remux to Matroska would not carry t, in words; "" when it carries it. func dropReason(t *inTrack) string { if t.encryption != "" { diff --git a/mp4/exotic_test.go b/mp4/exotic_test.go index 4ab9834..11b3f82 100644 --- a/mp4/exotic_test.go +++ b/mp4/exotic_test.go @@ -808,3 +808,68 @@ func TestShortLPCMEntryIsRefused(t *testing.T) { t.Error("a 16-byte lpcm entry was accepted") } } + +// TestFragmentedDurationIsTheLatestTrackEnd: a fragmented file without an edit list whose track starts after +// zero (its samples carry a composition offset) lasts until that track ends, its start plus the duration of its +// samples, not for the durations alone; the probe, the keyframe probe and the Duration a remux writes agree. +func TestFragmentedDurationIsTheLatestTrackEnd(t *testing.T) { + moof1, mdat1 := oneFragment(1, 0, []uint32{10, 10, 10}, []uint32{1024, 1024, 1024}, []uint32{2048, 2048, 2048}, 1) + moof2, mdat2 := oneFragment(1, 3072, []uint32{10, 10}, []uint32{1024, 1024}, []uint32{3072, 0}, 1) + data := fragmentedAAC(0, moof1, mdat1, moof2, mdat2) + const samplesEnd, trackEnd = 5120 * 1000 / 48000, (2048 + 5120) * 1000 / 48000 // 106 ms, 149 ms + for _, tc := range []struct { + name string + opts Options + }{{"probe", Options{}}, {"keyframe probe", Options{Keyframes: true}}} { + c, _ := probeMovie(t, data, tc.opts) + if c.DurationMs != trackEnd { + t.Errorf("%s: DurationMs = %d, want %d: the track's start plus its samples", tc.name, c.DurationMs, trackEnd) + } + if c.Tracks[0].DurationMs != samplesEnd { + t.Errorf("%s: track DurationMs = %d, want its %d ms of samples", tc.name, c.Tracks[0].DurationMs, samplesEnd) + } + } + dir := t.TempDir() + src, dst := filepath.Join(dir, "late.mp4"), filepath.Join(dir, "late.mkv") + if err := os.WriteFile(src, data, 0o600); err != nil { + t.Fatal(err) + } + if err := RemuxFromMP4(context.Background(), src, dst); err != nil { + t.Fatalf("RemuxFromMP4: %v", err) + } + c, blocks := readMKV(t, dst) + if c.DurationMs != trackEnd { + t.Errorf("remux: the Matroska states %d ms, want %d", c.DurationMs, trackEnd) + } + var last int64 + for _, b := range blocks { + last = max(last, b.Timecode) + } + if last >= c.DurationMs { + t.Errorf("remux: a block is shown at %d ms, at or past the %d ms the file states", last, c.DurationMs) + } +} + +// TestFragmentedDurationIgnoresAStrayOffset: one sample whose composition offset is far from the others' (a bit +// flipped in a damaged file) does not stretch the duration: the track still ends at its start plus its samples. +func TestFragmentedDurationIgnoresAStrayOffset(t *testing.T) { + moof1, mdat1 := oneFragment(1, 0, []uint32{10, 10, 10}, []uint32{1024, 1024, 1024}, []uint32{2048, 2048, 2048}, 1) + moof2, mdat2 := oneFragment(1, 3072, []uint32{10, 10}, []uint32{1024, 1024}, []uint32{2048 | 1<<24, 2048}, 1) + data := fragmentedAAC(0, moof1, mdat1, moof2, mdat2) + const trackEnd = (2048 + 5120) * 1000 / 48000 + for _, tc := range []struct { + name string + opts Options + }{{"probe", Options{}}, {"keyframe probe", Options{Keyframes: true}}} { + if c, _ := probeMovie(t, data, tc.opts); c.DurationMs != trackEnd { + t.Errorf("%s: DurationMs = %d, want %d", tc.name, c.DurationMs, trackEnd) + } + } + mv, err := parseMP4(bytes.NewReader(data), int64(len(data)), sampleFull) + if err != nil { + t.Fatal(err) + } + if c := containerFromMovie(mv); c.DurationMs != trackEnd { + t.Errorf("full parse: DurationMs = %d, want %d", c.DurationMs, trackEnd) + } +} diff --git a/mp4/fragparse.go b/mp4/fragparse.go index b5c2bc2..41f7192 100644 --- a/mp4/fragparse.go +++ b/mp4/fragparse.go @@ -128,7 +128,9 @@ func readFragmentSamples(r io.ReadSeeker, size int64, mv *movie, moovBoxes []mem last := t.samples[n-1] end := last.ctsMs + last.durMs t.durationMs = max(t.durationMs, end) - mv.durationMs = max(mv.durationMs, end) + // The movie lasts until its latest track ends, and a track ends where it starts plus what its + // samples last: one that starts after zero (a composition offset, no edit list) ends that much later. + mv.durationMs = max(mv.durationMs, end, t.shownEndMs()) } } return nil @@ -453,6 +455,10 @@ func appendTrunSamples(p []byte, t *inTrack, base, dataCursor, dts int64, defDur off += 4 } + if !t.fragStarted { + t.fragStartOff, t.fragStarted = ctsOff, true + } + t.fragEndTicks = max(t.fragEndTicks, dts+int64(dur)) cts := ticksToMs(dts+ctsOff, t.timescale) + t.editShiftMs if cts < 0 { cts = 0 @@ -472,3 +478,13 @@ func appendTrunSamples(p []byte, t *inTrack, base, dataCursor, dts int64, defDur } return sampleOff, dts, nil } + +// shownEndMs is when a fragmented track's presentation ends: its start plus the duration of its samples, 0 when +// it has no fragment. The start is the composition offset of its first sample; one stray offset further on (a +// damaged file) does not move it. +func (t *inTrack) shownEndMs() int64 { + if t.fragEndTicks <= 0 || t.timescale == 0 { + return 0 + } + return max(ticksToMs(t.fragEndTicks+max(t.fragStartOff, 0), t.timescale)+t.editShiftMs, 0) +} diff --git a/mp4/parse.go b/mp4/parse.go index 677b758..e26fa2d 100644 --- a/mp4/parse.go +++ b/mp4/parse.go @@ -74,13 +74,18 @@ type inTrack struct { samples []inSample keyframesMs []int64 // sync-sample presentation times (sampleKeyframes mode); nil otherwise sampleEndMs int64 // last sample's cts (sampleKeyframes mode), for the movie duration - sttsClipped bool // the stts states a negative delta: the samples' timeline does not give the duration - unsupported string // why a remux cannot carry the track; "" when it can - encryption string // the protected sample entry's scheme (schm); "" when the samples are clear - bitDepth uint8 // PCM bits per sample; 0 otherwise - videoBitDepth uint16 // the bit depth a video entry's fourcc states (ProRes); 0 when it states none - pcmFrame uint32 // PCM bytes per frame (one sample of every channel); 0 when not PCM - fragDecodeEnd int64 // where the track's last fragment ended (ticks): the next one's base when it has no tfdt + // A fragmented track's presentation end, in its timescale: where its fragments' durations end (fragEndTicks), + // later by the composition offset its first fragment sample starts at (fragStartOff, once fragStarted). + fragEndTicks int64 + fragStartOff int64 + fragStarted bool + sttsClipped bool // the stts states a negative delta: the samples' timeline does not give the duration + unsupported string // why a remux cannot carry the track; "" when it can + encryption string // the protected sample entry's scheme (schm); "" when the samples are clear + bitDepth uint8 // PCM bits per sample; 0 otherwise + videoBitDepth uint16 // the bit depth a video entry's fourcc states (ProRes); 0 when it states none + pcmFrame uint32 // PCM bytes per frame (one sample of every channel); 0 when not PCM + fragDecodeEnd int64 // where the track's last fragment ended (ticks): the next one's base when it has no tfdt // language and selection flags read from the track header / media header. language string // ISO 639-2 from mdhd (e.g. "fre"); "" when absent/"und"