diff --git a/README.md b/README.md index f44eced..5f49e31 100644 --- a/README.md +++ b/README.md @@ -63,16 +63,84 @@ The editor provides: ## Music and sound effects -Commodore Commander supports [SIDScore](https://github.com/turesheim/SIDScore) which is a DSL and toolchain for composing Commodore 64 SID music and sound effects. It has realtime auditioning and export to ASM, PRG, SID, and WAV. It focuses on playback fidelity by aligning the built-in `sidscore` driver with a realtime player timing while keeping output compatible with PSID players like VICE/VSID. SIDScore playback uses the dedicated instrument panel. And even has support for MIDI instruments. - -![Theia SIDScore player with SID instrument controls](docs/theia-sidscore-player.png) +Commodore Commander supports [SIDScore](https://github.com/turesheim/SIDScore), +a DSL and toolchain for composing Commodore 64 SID music and sound effects. It +provides real-time auditioning, MIDI input, dedicated instrument controls, and +export to ASM, PRG, SID, and WAV. Playback fidelity is improved by aligning the +built-in `sidscore` driver with the timing of the real-time player while keeping +the exported result compatible with PSID players such as VICE and VSID. + +### SIDScore voice visualiser + +The voice visualiser receives ordered audio samples from the SIDScore real-time +renderer and displays voices 1, 2, and 3 independently. The note, waveform, and +envelope level shown beside each plot describe the current voice state. Use the +**View** control to switch between the time-domain **Waveform** view and the +frequency-domain **Spectrogram** view while playback continues. + +#### Waveform + +![Theia SIDScore player with stabilised per-voice waveforms](docs/theia-sidscore-player.png) + +The waveform view plots sample amplitude against time for each voice. At the +usual 44.1 kHz sample rate, the 2,048 displayed samples cover approximately 46 +milliseconds. Each trace is centred on zero and may be enlarged automatically, +up to three times, to make quieter waveforms legible. The vertical scale is +therefore useful for inspecting shape and timing, but it is not a fixed level +scale for comparing loudness between voices. + +**Free** mode always shows the newest samples. This is useful for attacks, +releases, pulse-width sweeps, and other changing signals, but a periodic wave +will normally appear to move horizontally. **Triggered** mode finds the latest +rising midpoint crossing in the selected trigger voice and places it at the +dashed vertical line. The same time offset is applied to all three voices, so +their relative timing is preserved while periodic waveforms appear stationary. +The **V1**, **V2**, and **V3** controls select the common trigger source. If that +voice is silent or has too little signal range, the visualiser falls back to +the newest samples and omits the trigger line. + +#### Spectrogram + +![Theia SIDScore player with per-voice spectrograms](docs/theia-sidscore-spectrogram.png) + +The spectrogram shows how the frequency content of each rendered voice changes +over time: + +- **Horizontal position is time.** Older samples are on the left and the newest + analysis window is on the right. The default history is 16,384 samples, or + approximately 372 milliseconds at 44.1 kHz. +- **Vertical position is frequency on a logarithmic scale.** At 44.1 kHz the + display covers 50 Hz to the 22.05 kHz Nyquist limit. The 100 Hz, 1 kHz, and + 10 kHz labels are reference lines, not the limits: content below 100 Hz and + above 10 kHz is still included. +- **Colour is level in dBFS.** The fixed legend runs from -96 dBFS (dark) to + 0 dBFS (light). dBFS is relative to digital full scale, not acoustic sound + pressure. Because the scale is fixed rather than normalised per frame, colour + can be compared over time and between the three voices. + +Each column is calculated from a 2,048-sample Hann-windowed FFT, with a +256-sample hop between columns. At 44.1 kHz this gives an analysis window of +approximately 46 milliseconds and a new column every 5.8 milliseconds. The +Hann window reduces spectral leakage caused by cutting the signal into short +frames. The window also spreads an abrupt change across a short interval; the +view is therefore a time-frequency analysis rather than an instantaneous list +of oscillator frequencies. + +A pitched waveform normally appears as a fundamental band with harmonic bands +above it. Saw and pulse waves contain stronger high harmonics than a triangle +wave, while noise produces energy across a broad frequency range. Pulse-width +changes alter the harmonic pattern, and filter cutoff or resonance modulation +changes which bands remain strong over time. The display analyses the separate +voice samples produced by SIDScore's digital real-time renderer, including its +filter-routing contribution. It does not show only the SID frequency register, +and it is not a measurement of the analogue output of a physical 6581 or 8580. The SID SFX editor provides a focused workspace for one-shot game sound -effects. It starts from a small preset catalog and exposes SIDScore controls for -waveform, pitch sweep, ADSR envelope, pulse width, priority, voice selection, -and retrigger behavior. The editor visualizes the envelope, pitch movement, and -gate-off point while generating a SIDScore `EFFECT` block that can be previewed, -stopped, or copied into a project source file. +effects. It starts from a small preset catalogue and exposes SIDScore controls +for waveform, pitch sweep, ADSR envelope, pulse width, priority, voice +selection, and retrigger behaviour. The editor visualises the envelope, pitch +movement, and gate-off point while generating a SIDScore `EFFECT` block that +can be previewed, stopped, or copied into a project source file. ![Theia SID SFX editor](docs/theia-sid-sfx-editor.png) diff --git a/docs/theia-sidscore-player.png b/docs/theia-sidscore-player.png index 6244893..19c04b8 100644 Binary files a/docs/theia-sidscore-player.png and b/docs/theia-sidscore-player.png differ diff --git a/docs/theia-sidscore-spectrogram.png b/docs/theia-sidscore-spectrogram.png new file mode 100644 index 0000000..ec1915d Binary files /dev/null and b/docs/theia-sidscore-spectrogram.png differ diff --git a/package-lock.json b/package-lock.json index 9b549a2..01b0bc7 100644 --- a/package-lock.json +++ b/package-lock.json @@ -8804,6 +8804,12 @@ "pend": "~1.2.0" } }, + "node_modules/fft.js": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/fft.js/-/fft.js-4.0.4.tgz", + "integrity": "sha512-f9c00hphOgeQTlDyavwTtu6RiK8AIFjD6+jvXkNkpeQ7rirK3uFWVpalkoS4LAwbdX7mfZ8aoBfFVQX1Re/8aw==", + "license": "MIT" + }, "node_modules/file-icons-js": { "version": "1.0.3", "resolved": "https://registry.npmjs.org/file-icons-js/-/file-icons-js-1.0.3.tgz", @@ -16685,6 +16691,7 @@ "@theia/task": "1.66.2", "@theia/toolbar": "1.66.2", "@theia/workspace": "1.66.2", + "fft.js": "^4.0.4", "jsonc-parser": "^2.3.1", "ws": "^8.20.0" }, diff --git a/packages/theia-extension/package.json b/packages/theia-extension/package.json index 3a3c799..e5616b8 100644 --- a/packages/theia-extension/package.json +++ b/packages/theia-extension/package.json @@ -34,6 +34,7 @@ "@theia/task": "1.66.2", "@theia/toolbar": "1.66.2", "@theia/workspace": "1.66.2", + "fft.js": "^4.0.4", "jsonc-parser": "^2.3.1", "ws": "^8.20.0" }, diff --git a/packages/theia-extension/src/browser/commodore-commander-theme.ts b/packages/theia-extension/src/browser/commodore-commander-theme.ts index aa34f4b..4022ad2 100644 --- a/packages/theia-extension/src/browser/commodore-commander-theme.ts +++ b/packages/theia-extension/src/browser/commodore-commander-theme.ts @@ -356,6 +356,81 @@ export class CommodoreCommanderThemeStyleParticipant implements ColorContributio border-radius: 0 3px 3px 0; } +.cc-sidscore-spectrogram { + background: #1b1f24; + height: 100%; + min-height: 40px; + overflow: hidden; + position: relative; + width: 100%; +} + +.cc-sidscore-spectrogram__canvas { + display: block; + height: 100%; + width: 100%; +} + +.cc-sidscore-spectrogram__frequency { + border-top: 1px solid rgba(245, 239, 202, 0.18); + left: 0; + pointer-events: none; + position: absolute; + right: 0; +} + +.cc-sidscore-spectrogram__frequency span, +.cc-sidscore-spectrogram__time { + background: rgba(27, 31, 36, 0.72); + color: rgba(245, 239, 202, 0.82); + font-size: 10px; + line-height: 14px; + padding: 0 3px; + position: absolute; +} + +.cc-sidscore-spectrogram__frequency span { + left: 3px; + top: 0; + transform: translateY(-50%); +} + +.cc-sidscore-spectrogram__time { + bottom: 2px; + pointer-events: none; +} + +.cc-sidscore-spectrogram__time--start { + left: 48px; +} + +.cc-sidscore-spectrogram__time--end { + right: 3px; +} + +.cc-sidscore-spectrogram-legend { + align-items: center; + display: inline-flex; + font-size: 10px; + gap: 5px; + margin-left: auto; + white-space: nowrap; +} + +.cc-sidscore-spectrogram-legend__scale { + background: linear-gradient( + 90deg, + #1b1f24, + #254a66, + #2f7f75, + #c2a84f, + #f5efca + ); + border: 1px solid rgba(245, 239, 202, 0.35); + height: 10px; + width: 72px; +} + .cc-sidscore-protocol-log-widget { min-width: 360px; } diff --git a/packages/theia-extension/src/browser/sidscore-spectrogram.ts b/packages/theia-extension/src/browser/sidscore-spectrogram.ts new file mode 100644 index 0000000..ca9179c --- /dev/null +++ b/packages/theia-extension/src/browser/sidscore-spectrogram.ts @@ -0,0 +1,230 @@ +import FFT = require('fft.js'); + +export const SID_SCORE_SPECTROGRAM_FFT_SIZE = 2048; +export const SID_SCORE_SPECTROGRAM_HOP_SIZE = 256; +export const SID_SCORE_SPECTROGRAM_HEIGHT = 96; +export const SID_SCORE_SPECTROGRAM_MIN_FREQUENCY = 50; +export const SID_SCORE_SPECTROGRAM_MIN_DECIBELS = -96; +export const SID_SCORE_SPECTROGRAM_MAX_DECIBELS = 0; + +export interface SidScoreSpectrogramOptions { + readonly fftSize?: number; + readonly hopSize?: number; + readonly height?: number; + readonly minFrequency?: number; + readonly minDecibels?: number; + readonly maxDecibels?: number; +} + +export interface SidScoreSpectrogram { + readonly width: number; + readonly height: number; + readonly intensities: Float32Array; + readonly minFrequency: number; + readonly maxFrequency: number; + readonly minDecibels: number; + readonly maxDecibels: number; + readonly timeSpanSeconds: number; +} + +export function createSidScoreSpectrogram( + samples: readonly number[], + sampleRate: number, + options: SidScoreSpectrogramOptions = {} +): SidScoreSpectrogram { + const fftSize = options.fftSize ?? SID_SCORE_SPECTROGRAM_FFT_SIZE; + const hopSize = options.hopSize ?? SID_SCORE_SPECTROGRAM_HOP_SIZE; + const height = options.height ?? SID_SCORE_SPECTROGRAM_HEIGHT; + const minDecibels = + options.minDecibels ?? SID_SCORE_SPECTROGRAM_MIN_DECIBELS; + const maxDecibels = + options.maxDecibels ?? SID_SCORE_SPECTROGRAM_MAX_DECIBELS; + const requestedMinFrequency = + options.minFrequency ?? SID_SCORE_SPECTROGRAM_MIN_FREQUENCY; + validateOptions( + fftSize, + hopSize, + height, + sampleRate, + requestedMinFrequency, + minDecibels, + maxDecibels + ); + + const maxFrequency = sampleRate / 2; + const minFrequency = Math.max( + sampleRate / fftSize, + Math.min(requestedMinFrequency, maxFrequency / 2) + ); + if (samples.length === 0) { + return { + width: 0, + height, + intensities: new Float32Array(0), + minFrequency, + maxFrequency, + minDecibels, + maxDecibels, + timeSpanSeconds: 0 + }; + } + + const frameCount = samples.length <= fftSize + ? 1 + : Math.floor((samples.length - fftSize) / hopSize) + 1; + const coveredSamples = fftSize + (frameCount - 1) * hopSize; + const firstFrameStart = samples.length - coveredSamples; + const intensities = new Float32Array(frameCount * height); + const window = createHannWindow(fftSize); + const windowSum = window.reduce((sum, value) => sum + value, 0); + const fft = new FFT(fftSize); + const input = new Array(fftSize).fill(0); + const spectrum = fft.createComplexArray() as number[]; + const magnitudes = new Float64Array(fftSize / 2 + 1); + + for (let column = 0; column < frameCount; column += 1) { + const frameStart = firstFrameStart + column * hopSize; + for (let index = 0; index < fftSize; index += 1) { + const sourceIndex = frameStart + index; + input[index] = + sourceIndex >= 0 && sourceIndex < samples.length + ? (samples[sourceIndex] ?? 0) * (window[index] ?? 0) + : 0; + } + + fft.realTransform(spectrum, input); + for (let bin = 0; bin < magnitudes.length; bin += 1) { + const real = spectrum[bin * 2] ?? 0; + const imaginary = spectrum[bin * 2 + 1] ?? 0; + const oneSidedScale = bin === 0 || bin === fftSize / 2 ? 1 : 2; + magnitudes[bin] = + oneSidedScale * Math.hypot(real, imaginary) / windowSum; + } + + for (let row = 0; row < height; row += 1) { + const frequency = frequencyForSpectrogramRow( + row, + height, + minFrequency, + maxFrequency + ); + const magnitude = interpolateMagnitude( + magnitudes, + frequency * fftSize / sampleRate + ); + const decibels = 20 * Math.log10(Math.max(magnitude, 1e-12)); + intensities[row * frameCount + column] = clamp01( + (decibels - minDecibels) / (maxDecibels - minDecibels) + ); + } + } + + return { + width: frameCount, + height, + intensities, + minFrequency, + maxFrequency, + minDecibels, + maxDecibels, + timeSpanSeconds: coveredSamples / sampleRate + }; +} + +export function frequencyForSpectrogramRow( + row: number, + height: number, + minFrequency: number, + maxFrequency: number +): number { + if (height <= 1) { + return maxFrequency; + } + const position = Math.max(0, Math.min(1, row / (height - 1))); + return maxFrequency * Math.pow(minFrequency / maxFrequency, position); +} + +export function spectrogramRowForFrequency( + frequency: number, + height: number, + minFrequency: number, + maxFrequency: number +): number { + if (height <= 1 || frequency >= maxFrequency) { + return 0; + } + if (frequency <= minFrequency) { + return Math.max(0, height - 1); + } + const position = + Math.log(frequency / maxFrequency) / + Math.log(minFrequency / maxFrequency); + return position * (height - 1); +} + +function createHannWindow(size: number): readonly number[] { + return Array.from( + { length: size }, + (_value, index) => + 0.5 * (1 - Math.cos((2 * Math.PI * index) / (size - 1))) + ); +} + +function interpolateMagnitude( + magnitudes: Float64Array, + binPosition: number +): number { + const lower = Math.max( + 0, + Math.min(magnitudes.length - 1, Math.floor(binPosition)) + ); + const upper = Math.min(magnitudes.length - 1, lower + 1); + const fraction = Math.max(0, Math.min(1, binPosition - lower)); + return ( + (magnitudes[lower] ?? 0) * (1 - fraction) + + (magnitudes[upper] ?? 0) * fraction + ); +} + +function validateOptions( + fftSize: number, + hopSize: number, + height: number, + sampleRate: number, + minFrequency: number, + minDecibels: number, + maxDecibels: number +): void { + if ( + !Number.isInteger(fftSize) || + fftSize <= 1 || + (fftSize & (fftSize - 1)) !== 0 + ) { + throw new RangeError('Spectrogram FFT size must be a power of two.'); + } + if (!Number.isInteger(hopSize) || hopSize <= 0 || hopSize > fftSize) { + throw new RangeError('Spectrogram hop size must be between 1 and FFT size.'); + } + if (!Number.isInteger(height) || height <= 0) { + throw new RangeError('Spectrogram height must be a positive integer.'); + } + if (!Number.isFinite(sampleRate) || sampleRate <= 0) { + throw new RangeError('Spectrogram sample rate must be positive.'); + } + if (!Number.isFinite(minFrequency) || minFrequency <= 0) { + throw new RangeError('Spectrogram minimum frequency must be positive.'); + } + if ( + !Number.isFinite(minDecibels) || + !Number.isFinite(maxDecibels) || + maxDecibels <= minDecibels + ) { + throw new RangeError( + 'Spectrogram maximum decibels must exceed minimum decibels.' + ); + } +} + +function clamp01(value: number): number { + return Math.max(0, Math.min(1, value)); +} diff --git a/packages/theia-extension/src/browser/sidscore-waveform-widget.tsx b/packages/theia-extension/src/browser/sidscore-waveform-widget.tsx index 05cc9d2..b41b445 100644 --- a/packages/theia-extension/src/browser/sidscore-waveform-widget.tsx +++ b/packages/theia-extension/src/browser/sidscore-waveform-widget.tsx @@ -17,11 +17,18 @@ import { type SidScoreScopeChannel, type SidScoreScopeMode } from './sidscore-scope-visualization'; +import { + createSidScoreSpectrogram, + SID_SCORE_SPECTROGRAM_MAX_DECIBELS, + SID_SCORE_SPECTROGRAM_MIN_DECIBELS, + spectrogramRowForFrequency, + type SidScoreSpectrogram +} from './sidscore-spectrogram'; export const SID_SCORE_WAVEFORM_WIDGET_ID = 'commodore-commander.sidscore-waveforms'; -const SCOPE_BUFFER_SIZE = 8192; +const SCOPE_BUFFER_SIZE = 16384; const SCOPE_DISPLAY_SIZE = 2048; const SCOPE_AUTO_GAIN_LIMIT = 3.0; const SCOPE_VERTICAL_HEADROOM = 0.92; @@ -32,14 +39,28 @@ const SCOPE_TRIGGER = 'rgba(214, 205, 182, 0.42)'; const SCOPE_WIDTH = 1000; const SCOPE_HEIGHT = 140; +type SidScoreVisualisationMode = 'waveform' | 'spectrogram'; + +interface CachedSidScoreSpectrogram { + readonly revision: number; + readonly sampleRate: number; + readonly spectrogram: SidScoreSpectrogram; +} + @injectable() export class SidScoreWaveformWidget extends ReactWidget { protected voiceState: SidScoreVoiceStateEvent | undefined; protected scopeBuckets: SidScoreScopeBucketsEvent | undefined; protected songMetadata: SidScoreSongMetadata | undefined; protected playbackLabel = 'Idle'; + protected visualisationMode: SidScoreVisualisationMode = 'waveform'; protected scopeMode: SidScoreScopeMode = 'free'; protected triggerVoice = 1; + protected scopeSampleRate: number | undefined; + protected readonly spectrogramCache = new Map< + number, + CachedSidScoreSpectrogram + >(); protected readonly scopeBuffers = new Map([ [1, new ScopeTraceBuffer()], [2, new ScopeTraceBuffer()], @@ -50,7 +71,7 @@ export class SidScoreWaveformWidget extends ReactWidget { super(); this.id = SID_SCORE_WAVEFORM_WIDGET_ID; this.title.label = 'SIDScore'; - this.title.caption = 'SIDScore Voice Waveforms'; + this.title.caption = 'SIDScore Voice Visualiser'; this.title.iconClass = codicon('pulse'); this.title.closable = true; this.addClass('cc-sidscore-waveforms'); @@ -73,6 +94,7 @@ export class SidScoreWaveformWidget extends ReactWidget { setScopeBuckets(event: SidScoreScopeBucketsEvent): void { this.scopeBuckets = event; + this.setScopeSampleRate(event.sampleRate); for (const voiceScope of event.voices) { this.scopeBuffers .get(voiceScope.voiceIndex) @@ -82,6 +104,7 @@ export class SidScoreWaveformWidget extends ReactWidget { } setScopeSamples(event: SidScoreScopeSamplesEvent): void { + this.setScopeSampleRate(event.sampleRate); for (const voiceScope of event.voices) { this.scopeBuffers .get(voiceScope.voiceIndex) @@ -95,6 +118,8 @@ export class SidScoreWaveformWidget extends ReactWidget { this.scopeBuckets = undefined; this.songMetadata = undefined; this.playbackLabel = playbackLabel; + this.scopeSampleRate = undefined; + this.spectrogramCache.clear(); for (const buffer of this.scopeBuffers.values()) { buffer.clear(); } @@ -103,20 +128,24 @@ export class SidScoreWaveformWidget extends ReactWidget { protected render(): React.ReactNode { const songDetails = this.renderSongDetails(); - const scopeDisplay = createSidScoreScopeDisplay( - this.scopeSnapshots(), - this.scopeMode, - this.triggerVoice, - SCOPE_DISPLAY_SIZE - ); + const snapshots = this.scopeSnapshots(); + const scopeDisplay = this.visualisationMode === 'waveform' + ? createSidScoreScopeDisplay( + snapshots, + this.scopeMode, + this.triggerVoice, + SCOPE_DISPLAY_SIZE + ) + : undefined; const voices = [1, 2, 3].map((voiceIndex) => ({ voiceIndex, telemetry: this.voiceState?.voices.find( (voice) => voice.voiceIndex === voiceIndex ), trace: - scopeDisplay.channels.find((channel) => channel.voiceIndex === voiceIndex) - ?.samples ?? [] + (scopeDisplay?.channels ?? snapshots).find( + (channel) => channel.voiceIndex === voiceIndex + )?.samples ?? [] })); return ( @@ -146,7 +175,9 @@ export class SidScoreWaveformWidget extends ReactWidget { voiceIndex, telemetry, trace, - scopeDisplay.triggered ? scopeDisplay.triggerPosition : undefined + scopeDisplay?.triggered + ? scopeDisplay.triggerPosition + : undefined ) )} @@ -196,57 +227,106 @@ export class SidScoreWaveformWidget extends ReactWidget {
- Mode + View
- {(['free', 'triggered'] as const).map((mode) => ( + {(['waveform', 'spectrogram'] as const).map((mode) => ( ))}
- Trigger -
- {[1, 2, 3].map((voiceIndex) => ( - - ))} -
+ {(['free', 'triggered'] as const).map((mode) => ( + + ))} +
+ Trigger +
+ {[1, 2, 3].map((voiceIndex) => ( + + ))} +
+ + ) : this.renderSpectrogramLegend()} ); } + protected setVisualisationMode(mode: SidScoreVisualisationMode): void { + this.visualisationMode = mode; + this.update(); + } + + protected setScopeSampleRate(sampleRate: number): void { + const nextSampleRate = validSampleRate(sampleRate); + if ( + this.scopeSampleRate !== undefined && + nextSampleRate !== undefined && + nextSampleRate !== this.scopeSampleRate + ) { + for (const buffer of this.scopeBuffers.values()) { + buffer.clear(); + } + this.spectrogramCache.clear(); + } + this.scopeSampleRate = nextSampleRate; + } + protected setScopeMode(mode: SidScoreScopeMode): void { this.scopeMode = mode; this.update(); @@ -317,7 +397,7 @@ export class SidScoreWaveformWidget extends ReactWidget { gap: '10px', gridTemplateColumns: '120px 1fr', minHeight: 0, - padding: '8px' + padding: '4px 8px' }} >
@@ -341,43 +422,251 @@ export class SidScoreWaveformWidget extends ReactWidget { env {telemetry ? Math.round(telemetry.envelopeLevel * 100) : 0}%
- - - {triggerPosition !== undefined && trace.length > 1 ? ( + {this.visualisationMode === 'waveform' ? ( + - ) : undefined} - {renderScopeTrace(trace)} - + {triggerPosition !== undefined && trace.length > 1 ? ( + + ) : undefined} + {renderScopeTrace(trace)} + + ) : this.renderSpectrogram(voiceIndex, trace)} ); } + + protected renderSpectrogram( + voiceIndex: number, + samples: readonly number[] + ): React.ReactNode { + const buffer = this.scopeBuffers.get(voiceIndex); + const sampleRate = this.scopeSampleRate; + const cached = this.spectrogramCache.get(voiceIndex); + let spectrogram = cached?.spectrogram; + if ( + buffer && + sampleRate && + (cached?.revision !== buffer.revision || + cached.sampleRate !== sampleRate) + ) { + spectrogram = createSidScoreSpectrogram(samples, sampleRate); + this.spectrogramCache.set(voiceIndex, { + revision: buffer.revision, + sampleRate, + spectrogram + }); + } else if (!buffer || !sampleRate) { + spectrogram = undefined; + } + return ( +
+ + {spectrogram && spectrogram.width > 0 + ? renderSpectrogramAxes(spectrogram) + : undefined} +
+ ); + } + + protected renderSpectrogramLegend(): React.ReactNode { + return ( +
+ {SID_SCORE_SPECTROGRAM_MIN_DECIBELS} dBFS + + {SID_SCORE_SPECTROGRAM_MAX_DECIBELS} dBFS +
+ ); + } +} + +interface SpectrogramCanvasProps { + readonly ariaLabel: string; + readonly spectrogram: SidScoreSpectrogram | undefined; +} + +class SpectrogramCanvas extends React.Component { + protected readonly canvasRef = React.createRef(); + + componentDidMount(): void { + this.draw(); + } + + componentDidUpdate(previous: SpectrogramCanvasProps): void { + if (previous.spectrogram !== this.props.spectrogram) { + this.draw(); + } + } + + render(): React.ReactNode { + return ( + + ); + } + + protected draw(): void { + const canvas = this.canvasRef.current; + const spectrogram = this.props.spectrogram; + if (!canvas) { + return; + } + const context = canvas.getContext('2d'); + if (!context) { + return; + } + context.fillStyle = '#1b1f24'; + context.fillRect(0, 0, canvas.width, canvas.height); + if (!spectrogram || spectrogram.width === 0) { + return; + } + + const image = context.createImageData( + spectrogram.width, + spectrogram.height + ); + for (let index = 0; index < spectrogram.intensities.length; index += 1) { + const [red, green, blue] = spectrogramColour( + spectrogram.intensities[index] ?? 0 + ); + const pixel = index * 4; + image.data[pixel] = red; + image.data[pixel + 1] = green; + image.data[pixel + 2] = blue; + image.data[pixel + 3] = 255; + } + context.putImageData(image, 0, 0); + } +} + +const SPECTROGRAM_COLOUR_STOPS: readonly (readonly [ + number, + number, + number +])[] = [ + [27, 31, 36], + [37, 74, 102], + [47, 127, 117], + [194, 168, 79], + [245, 239, 202] +]; + +function renderSpectrogramAxes( + spectrogram: SidScoreSpectrogram +): React.ReactNode { + const frequencyTicks = [10000, 1000, 100].filter( + (frequency) => + frequency > spectrogram.minFrequency && + frequency < spectrogram.maxFrequency + ); + return ( + <> + {frequencyTicks.map((frequency) => { + const row = spectrogramRowForFrequency( + frequency, + spectrogram.height, + spectrogram.minFrequency, + spectrogram.maxFrequency + ); + const top = spectrogram.height > 1 + ? (row / (spectrogram.height - 1)) * 100 + : 0; + return ( +
+ {formatFrequency(frequency)} +
+ ); + })} + + -{formatTimeSpan(spectrogram.timeSpanSeconds)} + + + now + + + ); +} + +function spectrogramColour( + intensity: number +): readonly [number, number, number] { + const clamped = Math.max(0, Math.min(1, intensity)); + const position = clamped * (SPECTROGRAM_COLOUR_STOPS.length - 1); + const lowerIndex = Math.floor(position); + const upperIndex = Math.min( + SPECTROGRAM_COLOUR_STOPS.length - 1, + lowerIndex + 1 + ); + const fraction = position - lowerIndex; + const lower = SPECTROGRAM_COLOUR_STOPS[lowerIndex] ?? [0, 0, 0]; + const upper = SPECTROGRAM_COLOUR_STOPS[upperIndex] ?? lower; + return [ + Math.round(lower[0] + (upper[0] - lower[0]) * fraction), + Math.round(lower[1] + (upper[1] - lower[1]) * fraction), + Math.round(lower[2] + (upper[2] - lower[2]) * fraction) + ]; +} + +function formatFrequency(frequency: number): string { + if (frequency >= 1000) { + return `${frequency / 1000} kHz`; + } + return `${frequency} Hz`; +} + +function formatTimeSpan(seconds: number): string { + if (seconds < 1) { + return `${Math.round(seconds * 1000)} ms`; + } + return `${seconds.toFixed(1)} s`; +} + +function validSampleRate(sampleRate: number): number | undefined { + return Number.isFinite(sampleRate) && sampleRate > 0 + ? sampleRate + : undefined; } class ScopeTraceBuffer { @@ -385,6 +674,11 @@ class ScopeTraceBuffer { protected writePos = 0; protected filled = false; protected lastValue = 0; + protected revisionNumber = 0; + + get revision(): number { + return this.revisionNumber; + } appendBuckets( scope: SidScoreVoiceScopeBuckets, @@ -403,12 +697,18 @@ class ScopeTraceBuffer { Math.abs(this.lastValue - min) <= Math.abs(this.lastValue - max); this.appendInterpolated(minFirst ? max : min, expandedBucketLength); } + if (scope.buckets.length > 0) { + this.revisionNumber += 1; + } } appendSamples(scope: SidScoreVoiceScopeSamples): void { for (const sample of scope.samples) { this.append(i16ToFloat(sample)); } + if (scope.samples.length > 0) { + this.revisionNumber += 1; + } } clear(): void { @@ -416,6 +716,7 @@ class ScopeTraceBuffer { this.writePos = 0; this.filled = false; this.lastValue = 0; + this.revisionNumber += 1; } snapshot(): readonly number[] { diff --git a/packages/theia-extension/src/electron-main/commodore-commander-electron-main-module.ts b/packages/theia-extension/src/electron-main/commodore-commander-electron-main-module.ts index e2c3f9a..f37e07b 100644 --- a/packages/theia-extension/src/electron-main/commodore-commander-electron-main-module.ts +++ b/packages/theia-extension/src/electron-main/commodore-commander-electron-main-module.ts @@ -128,6 +128,12 @@ type ElectronScreenCaptureStep = readonly text: string; readonly timeoutMs?: number; } + | { + readonly type: 'clickVisibleText'; + readonly selector: string; + readonly text: string; + readonly timeoutMs?: number; + } | { readonly type: 'waitForInputValue'; readonly selector: string; @@ -637,6 +643,14 @@ async function runScreenCaptureStep( step.timeoutMs ?? timeoutMs ); return; + case 'clickVisibleText': + await clickVisibleText( + window, + step.selector, + step.text, + step.timeoutMs ?? timeoutMs + ); + return; case 'waitForInputValue': await waitForInputValue( window, @@ -852,6 +866,41 @@ async function waitForVisibleText( ); } +async function clickVisibleText( + window: BrowserWindow, + selector: string, + text: string, + timeoutMs: number +): Promise { + await waitForVisibleText(window, selector, text, timeoutMs); + const clicked = await evaluateInWindow( + window, + `(() => { + const elements = [...document.querySelectorAll(${JSON.stringify(selector)})]; + const element = elements.find((candidate) => { + const rect = candidate.getBoundingClientRect(); + const style = window.getComputedStyle(candidate); + return rect.width > 0 && + rect.height > 0 && + style.display !== 'none' && + style.visibility !== 'hidden' && + (candidate.textContent ?? '').includes(${JSON.stringify(text)}); + }); + if (!(element instanceof HTMLElement)) { + return false; + } + element.click(); + return true; + })()`, + false, + timeoutMs + ); + if (!clicked) { + throw new Error(`Unable to click "${text}" in ${selector}.`); + } + await delay(150); +} + async function waitForInputValue( window: BrowserWindow, selector: string, diff --git a/packages/theia-extension/src/test/sidscore-spectrogram.test.ts b/packages/theia-extension/src/test/sidscore-spectrogram.test.ts new file mode 100644 index 0000000..8325f87 --- /dev/null +++ b/packages/theia-extension/src/test/sidscore-spectrogram.test.ts @@ -0,0 +1,141 @@ +import * as assert from 'node:assert/strict'; +import { describe, it } from 'node:test'; + +import { + createSidScoreSpectrogram, + frequencyForSpectrogramRow +} from '../browser/sidscore-spectrogram'; + +const SAMPLE_RATE = 8192; +const TEST_OPTIONS = { + fftSize: 256, + hopSize: 128, + height: 64, + minFrequency: 32 +} as const; + +describe('SIDScore spectrogram', () => { + it('creates one time column for each overlapping analysis frame', () => { + const result = createSidScoreSpectrogram( + sineWave(512, 1024, 1), + SAMPLE_RATE, + TEST_OPTIONS + ); + + assert.equal(result.width, 3); + assert.equal(result.height, 64); + assert.equal(result.intensities.length, 3 * 64); + assert.equal(result.timeSpanSeconds, 512 / SAMPLE_RATE); + }); + + it('places a sinusoid at its frequency on the logarithmic axis', () => { + const result = createSidScoreSpectrogram( + sineWave(512, 1024, 1), + SAMPLE_RATE, + TEST_OPTIONS + ); + const peakRow = strongestRow(result.intensities, result.width, 1); + const peakFrequency = frequencyForSpectrogramRow( + peakRow, + result.height, + result.minFrequency, + result.maxFrequency + ); + + assert.ok(peakFrequency > 900 && peakFrequency < 1150); + }); + + it('preserves frequency changes from left to right', () => { + const samples = [ + ...sineWave(384, 256, 1), + ...sineWave(384, 1024, 1) + ]; + const result = createSidScoreSpectrogram( + samples, + SAMPLE_RATE, + TEST_OPTIONS + ); + const firstFrequency = rowFrequency(result, 0); + const lastFrequency = rowFrequency(result, result.width - 1); + + assert.ok(firstFrequency > 200 && firstFrequency < 330); + assert.ok(lastFrequency > 900 && lastFrequency < 1150); + }); + + it('uses a fixed dBFS scale rather than normalising each frame', () => { + const fullScale = createSidScoreSpectrogram( + sineWave(512, 1024, 1), + SAMPLE_RATE, + TEST_OPTIONS + ); + const halfScale = createSidScoreSpectrogram( + sineWave(512, 1024, 0.5), + SAMPLE_RATE, + TEST_OPTIONS + ); + const fullPeak = strongestIntensity(fullScale.intensities); + const halfPeak = strongestIntensity(halfScale.intensities); + + assert.ok(fullPeak > halfPeak); + assert.ok(fullPeak - halfPeak > 0.04 && fullPeak - halfPeak < 0.09); + }); + + it('maps silence to the dBFS floor', () => { + const result = createSidScoreSpectrogram( + new Array(512).fill(0), + SAMPLE_RATE, + TEST_OPTIONS + ); + + assert.equal(strongestIntensity(result.intensities), 0); + }); +}); + +function sineWave( + length: number, + frequency: number, + amplitude: number +): readonly number[] { + return Array.from( + { length }, + (_value, index) => + amplitude * Math.sin((2 * Math.PI * frequency * index) / SAMPLE_RATE) + ); +} + +function strongestRow( + intensities: Float32Array, + width: number, + column: number +): number { + let strongest = Number.NEGATIVE_INFINITY; + let strongestRow = 0; + for (let row = 0; row < intensities.length / width; row += 1) { + const intensity = intensities[row * width + column] ?? 0; + if (intensity > strongest) { + strongest = intensity; + strongestRow = row; + } + } + return strongestRow; +} + +function rowFrequency( + result: ReturnType, + column: number +): number { + return frequencyForSpectrogramRow( + strongestRow(result.intensities, result.width, column), + result.height, + result.minFrequency, + result.maxFrequency + ); +} + +function strongestIntensity(intensities: Float32Array): number { + let strongest = 0; + for (const intensity of intensities) { + strongest = Math.max(strongest, intensity); + } + return strongest; +} diff --git a/tools/capture-theia-screenshots.mjs b/tools/capture-theia-screenshots.mjs index bb9e218..1437e4a 100644 --- a/tools/capture-theia-screenshots.mjs +++ b/tools/capture-theia-screenshots.mjs @@ -494,24 +494,51 @@ function createCaptureConfig(options) { ] }, { - outputPath: path.join(options.outputDir, 'theia-character-set-editor.png'), - sourcePath: options.characterSetPath, + outputPath: path.join(options.outputDir, 'theia-sidscore-player.png'), + sourcePath: options.sidScorePath, + source: sidScorePlayerSource, + marker: { + needle: 'VOICE 1 lead:', + offset: 0 + }, steps: [ { type: 'executeCommand', - commandId: 'core.collapse.all.tabs' + commandId: 'commodoreCommander.sidscore.play' }, { type: 'waitForVisibleText', - selector: '.cc-character-set-editor', - text: 'Import .64C', + selector: '.cc-sidscore-waveforms', + text: 'SIDScore playback', timeoutMs: options.timeoutMs }, - { type: 'wait', ms: 500 } + { + type: 'waitForVisibleText', + selector: '.cc-sid-instrument', + text: 'Instrument', + timeoutMs: options.timeoutMs + }, + { + type: 'clickVisibleText', + selector: '.cc-sidscore-waveforms button', + text: 'Triggered', + timeoutMs: options.timeoutMs + }, + { type: 'wait', ms: 1000 } + ], + afterSteps: [ + { + type: 'executeCommand', + commandId: 'commodoreCommander.sidscore.stop' + }, + { type: 'wait', ms: 250 } ] }, { - outputPath: path.join(options.outputDir, 'theia-sidscore-player.png'), + outputPath: path.join( + options.outputDir, + 'theia-sidscore-spectrogram.png' + ), sourcePath: options.sidScorePath, source: sidScorePlayerSource, marker: { @@ -529,10 +556,16 @@ function createCaptureConfig(options) { text: 'SIDScore playback', timeoutMs: options.timeoutMs }, + { + type: 'clickVisibleText', + selector: '.cc-sidscore-waveforms button', + text: 'Spectrogram', + timeoutMs: options.timeoutMs + }, { type: 'waitForVisibleText', - selector: '.cc-sid-instrument', - text: 'Instrument', + selector: '.cc-sidscore-spectrogram-legend', + text: '-96 dBFS', timeoutMs: options.timeoutMs }, { type: 'wait', ms: 1000 } @@ -849,6 +882,23 @@ function createCaptureConfig(options) { }, { type: 'wait', ms: 250 } ] + }, + { + outputPath: path.join(options.outputDir, 'theia-character-set-editor.png'), + sourcePath: options.characterSetPath, + steps: [ + { + type: 'executeCommand', + commandId: 'core.collapse.all.tabs' + }, + { + type: 'waitForVisibleText', + selector: '.cc-character-set-editor', + text: 'Import .64C', + timeoutMs: options.timeoutMs + }, + { type: 'wait', ms: 500 } + ] } ] };