Replace .nbs emote sound with .opus - #897
Open
dima-dencep wants to merge 6 commits into
Open
Conversation
Note Block Studio playback is gone: NoteBlockLib, NbsPlayer, LegacyNBSPacket, MinecraftNbsPlayer, InstrumentConventer and the two note-block SoundInstances are removed. An emote's sound is now a mono Opus stream in <name>.opus next to the emote file, decoded with Concentus and handed to the sound engine as PCM. emotesAPI gains io.github.kosmx.emotes.common.opus: - OggOpusReader streams packets out of an Ogg container. Packets are runs of segments that end at the first one shorter than 255 and may cross pages, so the accumulator lives across page boundaries. OpusHead and OpusTags are read field by field off the stream; comments longer than 128 bytes are skipped unread so cover art never reaches memory. R128_TRACK_GAIN and LOOPSTART are picked up on the way past. - OggOpusWriter muxes packets back out, so an emote's sound survives export. - OpusPackets ports the TOC helpers from libopus opus.c, which is where frame size and sample count come from. - Loudness measures integrated loudness (ITU-R BS.1770 with EBU R128 gating) while the audio is being decoded, not in a second pass over the track. - OpusSound validates, then decodes on a small low-priority pool. The PCM is sized up front from the packet count, the pre-skip is dropped as it decodes, and the result is kept behind a SoftReference so a replay is free but memory pressure can reclaim it. Validation, cheapest first: file size before any parsing, mono only, every packet's TOC readable, duration under 10 minutes, wire size under CommonData.MAX_PACKET_SIZE, and bitrate under min(96 kbps, what fits that limit for the track's length). SongPacket keeps its id and class name. Version 3 carries [u16 preSkip], a varint loop start and a varint-counted list of varint-length-prefixed packets, and nothing else: sample rate and frame size come from the TOC, channel count is fixed by the spec. Version 2 now passes the raw .nbs file through untouched so clients old enough to still play it keep their sound. The packet count is sized against the readable bytes before a list is allocated. Decoding starts when the sub-packet arrives, never at animation start, and only on a client that is about to play the emote: a server or proxy relaying the stream, and a client loading emotes from disk, keep the packets and skip the decode. Playback goes through EmoteSoundInstance and PcmAudioStream, with a mixin handing the engine the decoded buffer in place of SoundBufferLibrary#getStream. The instance is created only once the PCM is ready, so a streaming channel is never taken for a decode that may outlive the emote. The track joins the animation wherever it already is, which covers a late decode, an emote that started mid-way, and resuming after a pause. A LOOPSTART tag makes the sound repeat from that sample, the way NBS loop and loopStartTick used to work. Loudness is normalised to -14 LUFS, from the R128 tag when there is one and from the measurement otherwise. The sound engine clamps volume to [0, 1], so this only ever attenuates. It can be turned off with normalizeSoundVolume, which replaces displayNowPlaying now that there is no song title to show. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XWkKzatVYEvnoQtdH9i9gs
A FILE sub-packet now writes the raw .nbs after the Opus packets, so a client that received an emote as a file still has something to stream to a peer that only speaks version 2. The sub-packet is already framed by its own size, so the tail needs no length of its own, and an emote without a .nbs writes exactly what it wrote before. Streaming still sends one or the other: a live emote goes to a peer that has already told us which format it understands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A three minute track is around 3200 packets. As a List<byte[]> that is 3200 object headers plus the list's own references, roughly 70 KB of overhead on 700 KB of audio, all of it long-lived. They now sit back to back in a single array with an int[] of offsets, so a packet is a slice rather than an object. Nothing along the way needs a byte[] of its own any more: the decoder takes an offset, the muxer appends from one, and the packet writer hands netty a range of the array. Reading from a file bounds the array by the file size and reading from the wire bounds it by what is left of the sub-packet, so neither grows. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Playback: - Emote sound takes a channel from the static pool. The streaming pool is sqrt(channels) capped at eight and shared with music and records, so a crowd emoting would silently lose audio. Only the pool choice moves; playback stays on the streaming path where the engine owns and frees the buffers. - A refused play is retried. The engine turns a sound down rather than queueing it when there is no volume, so what used to be a permanent latch now rebuilds the instance at the current animation time. Attempts are half a second apart, measured on wall time because animation time restarts on every loop, and each one notifies the subtitle overlay. - Whether a sound may be heard at all now lives in one place. It stops a playing sound and refuses to start a new one, so invisibility ending brings the audio back instead of silencing the rest of the emote, and a preview does not outlive its screen. Decoding: - A cleared SoftReference no longer starts a second decode: the running future is held strongly until it completes, and is stored before the completion is registered so an inline finish cannot be overwritten. - A failed decode is remembered and logged once instead of being retried every frame, and the catch covers IllegalArgumentException, which Concentus throws as readily as OpusException. - The pre-skip copy is bounded by the space left, so a decoder that disagrees with the TOC arithmetic cannot run off the end. Serialization: - .nbs goes to a peer that cannot read Opus, or into a stored emote. Requiring an exact version match dropped it from files, which negotiate at version 3. - Trailing bytes are only trusted in a stored emote; a streamed packet cannot smuggle a megabyte of junk that the server would then relay as a song. - A retired sub-packet version says so in the log rather than losing the sound silently. - Built-in emotes load their .opus, and a sound the mod cannot read costs the sound rather than the emote. - Reading from a stream is bounded as it goes; only the file path could check a size up front. - OpusSound.write opens the stream as its own resource, so a throw while writing the headers does not leak the descriptor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The reader and the writer each carried their own copy of the magic bytes, the page flags, the tag names and the number 255, and only the writer knew the CRC polynomial. All of it now lives in OggOpus, which is what the two of them have to agree on. The writer keeps its own page-fill policy: the format caps a page at 255 segments, but how full we make one is our choice, not the format's. Having the checksum on both sides made it worth honouring what the container says. The reader now ends the stream where the stream says it ends, rather than where the bytes run out, so a truncated file is refused instead of importing as a silently shorter sound, and a tagger's trailing bytes no longer discard a sound whose packets all read fine. Page order is checked, and the checksum is computed as the page is read and verified once it is whole, including the last one, which nothing would otherwise come along to check. A flipped byte used to travel all the way to the decoder. The writer stopped stamping one hard-coded serial into every stream, so concatenated files no longer read as one; close() no longer writes a second end-of-stream page when called twice; a packet whose length is out of range is refused before it corrupts the segment table; and a packet the TOC cannot describe is refused rather than sending the granule position backwards. Channel count is a parameter now instead of a literal 1, and OpusSound checks the stereo bit of every packet, so the header can no longer disagree with the audio on the network path, where there is no header to read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Emote sound moves from Note Block Studio to Opus. Put
<name>.opusnext to the emote instead of<name>.nbs; it has to be mono.NoteBlockLib is gone, replaced by Concentus for decoding.
NbsPlayer,LegacyNBSPacket,MinecraftNbsPlayer,InstrumentConventerand the two note-blockSoundInstances are deleted.EmotecraftSoundEventsstays,resolve()still needs it.Parsing
OggOpusReaderstreams packets out of the container. Packet boundaries have nothing to do with page boundaries — a packet is a run of segments ending at the first one shorter than 255 and can continue on the next page — so the accumulator lives across pages.OpusHeadandOpusTagsare read field by field off the stream rather than into a buffer, and comments over 128 bytes are skipped unread, which keeps embedded cover art out of memory.R128_TRACK_GAINandLOOPSTARTare picked up on the way past.OggOpusWritermuxes packets back into an Ogg stream so export keeps the sound. Verified against ffmpeg: a round trip through reader and writer gives back byte-identical packets and PCM, including at a bitrate where packets span page boundaries.Validation
Cheapest check first:
ifbefore any parsingOpusHeadparses and the stream is monoCommonData.MAX_PACKET_SIZEmin(96 kbps, what fits that limit for this length)3–6 run in the constructor, so they cover the file, the network and
.emotecraftalike.Wire format
SongPacketkeeps its id and class name. Version 3 is[u16 preSkip], a varint loop start, then a varint-counted list of varint-length-prefixed packets. Nothing else — sample rate and frame size come from the TOC, channel count is fixed by the spec. The packet count is sized against the readable bytes before any list is allocated.Version 2 now passes the raw
.nbsfile through untouched, so a client old enough to still play it keeps its sound. A new client picks Opus when the peer speaks version 3 and falls back to.nbsotherwise.Playback
The decode starts when the sub-packet arrives, never at animation start, and runs on a small low-priority pool — Concentus is fixed point and does not chew through minutes of audio instantly. The PCM buffer is sized up front from the packet count, pre-skip is dropped as it decodes, and the result sits behind a
SoftReference, so replays are free but memory pressure can reclaim it.Only a client that is about to play the emote decodes. A server or proxy relaying the stream keeps the packets and skips it, and so does a client loading emotes from disk.
A
SoundEnginemixin hands the engine the decoded buffer in place ofSoundBufferLibrary#getStream. The instance is built only once the PCM is ready, so a streaming channel is never taken for a decode that might outlive the emote. The track joins the animation wherever it already is, which covers a late decode, an emote that started mid-way, and resuming after a pause. ALOOPSTARTtag makes the sound repeat from that sample — theloop/loopStartTickpair NBS used to provide; without the tag it plays once.Loudness
Normalised to −14 LUFS, from the R128 tag when present and from an ITU-R BS.1770 measurement taken during the decode otherwise. The engine clamps volume to [0, 1], so this only ever attenuates; no limiter needed. Toggle is
normalizeSoundVolume, which takes the place ofdisplayNowPlaying— there is no song title to display any more.Checked against
ffmpeg -af ebur128: −8.63 LUFS measured here against −8.6 reported by ffmpeg on the same PCM.Notes
LegacyNBSPacket. A peer that only speaks it gets the emote without sound. Reviving it would mean bringing NoteBlockLib back — that format is a note list, not a file, so it cannot be passed through the way.nbsis.output_gainand the R128 tag are applied locally at parse time and not transmitted. Both sides land on −14 LUFS regardless, since normalisation is measured after the gain.🤖 Generated with Claude Code