Speech understanding

Speech input for LLMs

Three ways to feed speech to an LLM: continuous encoder embeddings, discrete semantic tokens, or codec tokens. Most models use the first.

Per-model paths

Why 12.5 Hz converges

Whisper emits 50 Hz frames. Voxtral notes that 30 minutes of audio at 50 Hz is a sequence of 90k tokens, which is why it adds a 4x downsampling adapter (arXiv 2507.13264). Moshi's Mimi codec also runs at 12.5 Hz (arXiv 2410.00037).

Trade-offs

• Continuous: richest acoustic and paralinguistic signal, easy to train as ASR, but the output side needs a separate path.
• Discrete: one vocabulary for input and output, enables self-supervised next-token training, loses detail.
• Codec: the model hears and speaks the same tokens, but multi-codebook streams cost more per frame. Inference This suits duplex.
• Inference Downsampling to 10-12.5 Hz is the main sequence-length lever. It costs fine acoustic detail, which motivates hybrids like Kimi-Audio.

ASR with speech LLMs: offline and streaming

ASR is one more speech-to-text task on the encoder, projector and LLM stack. The design questions are how much to train, and how to stream.

Training recipes

• Freezing the LLM is the direct guard against forgetting. Granite-speech states that in text mode it calls the underlying LLM without LoRA, which preserves its text capabilities.
• Canary-Qwen's LLM mode takes a text transcript only, not raw audio (NVIDIA model card).

Streaming approaches

Inference Chunk size and lookahead trade latency against WER.

Multilingual scale

Omnilingual ASR (Meta) covers over 1,600 languages, in models from 300M to 7B parameters (arXiv 2511.09690). Qwen3-ASR covers 52 languages and dialects (arXiv 2601.21337).

Leaderboard state

The Open ASR Leaderboard compares 86 systems on 12 datasets, with English short- and long-form and multilingual tracks (arXiv 2510.06961). Its authors observe that Conformer encoders paired with Transformer decoders reach the best average WER, while CTC and TDT decoders give the best RTFx.

Open question and scope

• Not covered here: transducer plus LLM hybrids and CTC-prefix joint decoding.

Whisper and the Whisper encoder

Whisper is an encoder-decoder Transformer trained on 680,000 hours of weakly supervised audio. Its encoder became the default front end for speech LLMs. Source: arXiv 2212.04356 unless stated.

Architecture

• Confirmed Input is 16 kHz audio turned into an 80-channel log-magnitude mel spectrogram (25 ms window, 10 ms stride), in 30 s segments.
• Confirmed The stem is two conv layers (width 3, GELU); the second has stride 2, so one encoder frame covers 20 ms.
• Derived: the 10 ms hop gives 100 mel frames per second (100 Hz). The stride-2 conv halves that to one frame per 20 ms (50 Hz). A 30 s window is therefore 30 s / 10 ms = 3,000 mel frames and 3,000 / 2 = 1,500 encoder frames. Short audio is padded to 30 s, so compute is paid for the full window.

Rule (derived arithmetic):

• large-v3: same architecture, 128 mel bins instead of 80, and a Cantonese token. Reported OpenAI reports a 10% to 20% error reduction over large-v2 (OpenAI model card).
• large-v3-turbo: a fine-tuned, pruned large-v3 with the decoder cut from 32 to 4 layers (OpenAI model card). The card lists it at 809M parameters.

Multitask token format

The decoder prompt carries the task as tokens: start-of-transcript, language, task (transcribe or translate), then timestamps or no-timestamps. One model therefore does ASR, X-to-English translation and language ID. Timestamps are quantised to 20 ms.

Training data and recipe

• Filtering: drop machine-generated transcripts (all caps or all lower, no punctuation). Compare audio language ID (VoxLingua107-trained) with CLD2 on the transcript and drop mismatches; English transcripts of other-language audio are kept as translation data.
• Optimiser: AdamW, gradient-norm clipping, linear decay after 2,048 warmup steps, batch 256, FP16 with activation checkpointing, 2 to 3 passes over the data.
• Augmentation: none for the first models. Large-v2 trained 2.5x more epochs with SpecAugment, stochastic depth and BPE dropout.

Long-form decoding

Whisper slides a 30 s window over long audio and decodes with heuristics (paper section 4.5).

• Beam search with beam 5.
• Temperature fallback from 0, in steps of 0.2 up to 1.0, when average log-probability is below -1 or the gzip compression ratio exceeds 2.4.
• Condition on the previous window's text only when temperature is below 0.5.
• No-speech threshold 0.6 to skip silent windows.

Limitations

• The model card warns that predictions may include text that was not spoken (hallucination) and that the sequence-to-sequence architecture is prone to repetitive text. It also states that the models cannot be used for real-time transcription out of the box.
• The fixed 30 s window, non-causal encoder and autoregressive beam decoder make streaming hard.

The encoder as a speech-LLM front end

Inference Three issues recur. Padding to 30 s wastes compute on short turns. 50 Hz is too long a sequence for an LLM, so every system adds an adapter (50 Hz / 4 = 12.5 Hz). Bidirectional attention blocks streaming, so teams either make the encoder causal (GLM-4-Voice) or replace it (AuT).

Making the Whisper encoder stream

Whisper does not stream for three reasons.

• The encoder takes a fixed 30 s window (arXiv 2212.04356), and Voxtral describes it as a fixed receptive field of 30 s (arXiv 2507.13264).
• Streaming wrappers pad every buffer to the full context length, so each partial result pays for 30 s (arXiv 2506.12154).
• Encoder attention is bidirectional, so every frame sees later audio (GLM-4-Voice replaces it with block-causal attention, arXiv 2412.02612). The model card states the models cannot do real-time transcription out of the box.

Four families fix this, with different cost.

• Causal fine-tuning. CarelessWhisper fine-tunes Whisper with LoRA so each encoder frame attends only to its own chunk and earlier chunks, and it caches encoder keys and values instead of re-encoding padded buffers (arXiv 2508.12301). It trains on 960 h of LibriSpeech with chunk sizes of 40 to 300 ms. The U2-style variant puts a CTC head on the Whisper encoder, trains with dynamic chunk masks (0.1 to 1.0 s) on inputs shorter than 30 s, and rescores with the original decoder (arXiv 2506.12154). GLM-4-Voice makes the encoder causal inside its tokenizer: causal convolution before the Transformer and block-causal attention, then pooling and vector quantisation give 12.5 Hz tokens (arXiv 2412.02612).
• Replace the encoder. Qwen3-Omni trains AuT (Audio Transformer) from scratch on 20 million hours of supervised audio, with 8x Conv2D downsampling to 12.5 Hz and block-wise window attention for real-time prefill caching (arXiv 2509.17765). Qwen3-ASR reuses AuT with a dynamic flash-attention window of 1 s to 8 s, so one model serves streaming (short chunks) and offline (long queries) (arXiv 2601.21337).
• Wrap the offline model. whisper_streaming re-runs Whisper on a growing buffer and commits words once consecutive runs agree (LocalAgreement), at 3.3 s latency (arXiv 2307.14743). Simul-Whisper uses the cross-attention alignment to guide decoding on chunks, with a truncation detector for cut words (arXiv 2406.10052). The U2 authors note that wrappers reprocess the same audio and leave a training-inference mismatch (arXiv 2506.12154).

Streaming audio encoders beyond Whisper

Most streaming encoders do one of two things: restrict attention to a causal or windowed context, or cache past activations.

• Qwen3-ASR numbers: the paper reports 2.69 average WER offline vs 3.33 streaming for the 1.7B model, and 3.48 vs 4.40 for 0.6B, with streaming run at a 2 s chunk size (arXiv 2601.21337). It also reports an average time to first token as low as 92 ms for the 0.6B model.
• DSM-ASR: feeds Mimi tokens (12.5 Hz) and delays the text stream by a default 2.5 s, with timestamp precision of 80 ms (arXiv 2509.08753).
• Cache-aware Conformer: with a regular look-ahead, latency compounds with depth (1,360 ms for 17 layers and one token of look-ahead); chunk-aware look-ahead does not. On LibriSpeech test-other with an RNNT decoder, the cache-aware model reaches 6.3% WER at 1,360 ms, against 11.3% for buffered inference at 2,000 ms (arXiv 2312.17279).
• The model card lists a 120M-parameter cache-aware FastConformer-RNNT with 17 layers and latency of 80 to 160 ms.

Ecosystem

Interview insights