Speech generation

Codecs and speech tokenizers

The tokenizer fixes the sequence length the LM sees and the ceiling on audio quality. Four families matter.

Qwen3-TTS ships two tokenizers: 25 Hz single-codebook and 12 Hz multi-codebook (arXiv 2601.15621).

Trade-offs

Evidence on low frame rates

A controlled frame-rate ablation reproduces a quality cliff at 6.25 Hz. Its authors trace the cliff to the training configuration (fixed clip duration gives too few tokens at low rates), not to a fundamental barrier. After the fix, WER degrades smoothly with phonemic load down to 3.1 Hz and 1.6 Hz (arXiv 2606.16969). Another paper pools X-Codec-2.0 from 50 Hz to a 25 Hz latent rate and reports a 0.29 UTMOSv2-based MOS gain (arXiv 2601.20185).

Generation paradigms

There are four ways to generate speech from text or from an LM state. The dominant high-quality design is the hybrid.

AR codec LMs

• Llasa: a single Llama-style Transformer over X-Codec2 tokens, 1B/3B/8B, with inference-time search using speech-understanding verifiers (arXiv 2502.04128).
• Spark-TTS: Qwen2.5 LLM plus BiCodec (arXiv 2503.01710).
• CosyVoice 2: an LLM predicts speech tokens, then a chunk-aware causal flow-matching model decodes them (arXiv 2412.10117).
• Newer entries: IndexTTS2 (arXiv 2506.21619), Step-Audio-EditX (LLM-based audio editing with reinforcement learning; arXiv 2511.03601), Higgs Audio v2 (pretrained on over 10 million hours of audio, per the Boson AI repo).

Multi-codebook handling

An RVQ frame has several codes, so the model needs a trick.

• Sesame CSM: a Llama backbone models the zeroth Mimi codebook from interleaved text and audio, and a smaller audio decoder with a linear head per codebook models the rest; Mimi runs at 12.5 Hz (Sesame blog).
• Moshi: $K = 2Q + 1 = 17$ streams per step with $Q = 8$ codebooks (8 user audio, 8 own audio, 1 text). Its RQ-Transformer splits temporal and per-codebook (depth) modelling (arXiv 2410.00037).
• Qwen3-Omni's Talker uses an MTP module to emit codec frames (arXiv 2509.17765).

NAR notes

SoundStorm uses bidirectional attention and confidence-based parallel decoding to generate codec tokens, and the authors report 30 s of audio in 0.5 s on a TPU-v4 (arXiv 2305.09636). MaskGCT is two-stage (text to semantic tokens, then semantic to acoustic tokens), mask-and-predict in both, with no explicit alignment or phone-level duration model, trained on 100K h (arXiv 2409.00750).

The "LM for content, flow for acoustics" split

• Inference Why the LM: text streaming, interleaving with text, reuse of LLM serving stacks, in-context learning.
• Inference Why flow: it can restore acoustic detail lost to quantization, and CosyVoice 2 shows it can decode chunk by chunk (arXiv 2412.10117).

Diffusion and flow matching for speech

Flow matching is the objective behind many recent TTS systems. It plays two roles: fully NAR TTS, and the acoustic decoder after an AR LM.

Diffusion objective (DDPM)

The forward process mixes clean data $x_0$ with Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$. The network $\epsilon_\theta$ learns to predict that noise from the noisy input and the timestep $t$.

Here $\bar\alpha_t$ is the cumulative noise schedule, so it fixes how much signal survives at step $t$. Sampling runs the learned denoiser backwards from pure noise.

This is the noise-prediction objective of the DDPM paper (arXiv 2006.11239).

Conditional flow matching objective

Flow matching regresses a network onto a velocity field. With a linear (optimal-transport) path, $x_0$ is noise and $x_1$ is data.

The target velocity $x_1 - x_0$ is constant along the path, so training is plain regression. There is no noise schedule and no SNR weighting.

Sampling integrates the learned ODE from noise to data:

The linear path and velocity regression follow Flow Matching with optimal-transport paths (arXiv 2210.02747).

Classifier-free guidance (CFG)

CFG pushes the output toward the condition $c$ by extrapolating away from an unconditional prediction $\varnothing$. The guidance scale $w$ sets the strength.

The cost is two forward passes per sampling step, so CFG doubles inference compute. The same form applies to a diffusion network by swapping $v_\theta$ for $\epsilon_\theta$.

Classifier-free guidance was introduced for diffusion models (arXiv 2207.12598); the form above applies it to a velocity network.

In speech, ZipVoice targets the CFG cost (arXiv 2506.13053). Lipman et al. show OT paths train and sample faster than diffusion paths (arXiv 2210.02747).

Where it sits

Counterexamples and traps

ASR has little flow work. Drax is the clearest, using discrete flow matching for NAR ASR (arXiv 2510.04162).

Continuous autoregression and next-token diffusion

A smaller, newer branch drops discrete tokens. An AR model predicts continuous latents, and a small diffusion or flow head samples each next frame or patch.

Inference The two lists below are design analysis, not results from the papers.

Streaming and low-latency generation

Streaming TTS needs a causal, low-frame-rate codec on the AR side, and a chunkable, few-step decoder on the acoustic side.

First-packet numbers

Latency levers

MeanFlow's headline FID 3.43 is an ImageNet one-step result (arXiv 2505.13447). IntMeanFlow reports 1-NFE token-to-spectrogram and 3-NFE text-to-spectrogram inference for speech (arXiv 2510.07979).