Skip to content

feat(vlm): add Gemma3n audio reference path (#875)#883

Merged
inureyes merged 1 commit into
mainfrom
feature/issue-875-gemma3n-audio
Jul 23, 2026
Merged

feat(vlm): add Gemma3n audio reference path (#875)#883
inureyes merged 1 commit into
mainfrom
feature/issue-875-gemma3n-audio

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

  • Implement the pinned Gemma3n waveform processor, audio Conformer tower, projection/pooling, hard/soft audio embedding merge, and per-layer-input integration.
  • Integrate audio placeholders and preparation into CLI and OpenAI-compatible serving while keeping request-scoped multimodal state isolated across concurrent sequences.
  • Match the maintained reference's left-padding contract, BF16 convolution boundary, and finite dtype-specific attention mask minimum required for token-exact greedy ASR.

Validation

  • Pinned Transformers source: 181beb3ba4c47098ed8cbc97ee250d1d45ae0107; pinned checkpoint revision: af70430f84ea4d7ac191aaa2bd8e14d2a5e8f6ee.
  • GB10 CUDA CLI generated the official 16-token sequence exactly: Chapter 1. Mrs. Rachel Lind is surprised.\n\nMrs. Rachel Lind.
  • OpenAI-compatible streaming generated the same 16-token sequence across SSE deltas and completed with [DONE].
  • cargo test --release --features cuda gemma3n --lib -- --test-threads=1: 34 passed, 0 failed, 14 ignored.
  • cargo test --release --features cuda gemma3n --lib -- --ignored --test-threads=1: 14 passed, 0 failed, including request-scoped per-layer-input isolation tests.
  • cargo fmt --all -- --check, cargo check --release --all-targets --features cuda, and cargo clippy --release --lib --bins --tests --features cuda -- -D warnings passed with Rust 1.93.1.
  • The stricter all-target Clippy invocation additionally reaches a pre-existing print_literal warning in examples/qmm_gemv_microbench.rs that is unchanged from origin/main; all issue-attributable library, binary, and test targets are clean.

Closes #875

@inureyes inureyes added status:review Under review type:enhancement New features, capabilities, or significant additions priority:medium Medium priority area:models Model architectures, weights, loading, metadata area:inference Generation, sampling, decoding (incl. speculative, DRY) labels Jul 23, 2026
Implement the pinned waveform frontend, isolated Conformer encoder, hard/soft audio projection, fixed-token prompt merge, and CLI/server integration. Match the maintained processor left-padding contract, checkpoint convolution dtype boundary, and finite attention-mask minimum so deterministic ASR remains token-exact.

Validate official weight layouts, preserve request-scoped per-layer inputs, and cover processor, encoder, merge, CLI, and OpenAI streaming behavior with CUDA regressions.

Refs #875
@inureyes
inureyes force-pushed the feature/issue-875-gemma3n-audio branch from 1ca0e13 to 920b6c1 Compare July 23, 2026 04:02
@inureyes
inureyes merged commit f0bc6a1 into main Jul 23, 2026
5 checks passed
@inureyes inureyes added status:done Completed and removed status:review Under review labels Jul 23, 2026
@inureyes
inureyes deleted the feature/issue-875-gemma3n-audio branch July 23, 2026 04:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:inference Generation, sampling, decoding (incl. speculative, DRY) area:models Model architectures, weights, loading, metadata priority:medium Medium priority status:done Completed type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(vlm): add a token-exact Gemma3n audio reference path

1 participant