Materials Speech-to-speech: literature review

Speech-to-speech: literature review

Foundations

How to read this page

Scope: how to model speech-to-speech (S2S) systems, treating the reasoning LLM as a downstream component. It covers newer-generation models and strategies in ASR and speech generation, plus a system design for Synthesia.

Each section is one topic, at most two printed pages, and ends with a "Say this in the interview" line. The cheat sheet in the last chapter is the one-page recall aid.

Grounding legend

Every model detail and number on this page is traced to a research paper (cited inline as arXiv IDs) or to the vendor's own page (linked inline). Numbers that the authors or vendor measured on their own systems are phrased as "the authors report".

  • Reported A number from the authors' or vendor's own first-party source.
  • Inference Design opinion, not fact.

Three traps

Three traps to avoid out loud
  • MiniMax-Speech introduces a "Flow-VAE" (arXiv 2505.07916). Do not read the name as flow matching.
  • Qwen3-Omni's waveform decoder is a lightweight causal ConvNet over a multi-codebook codec (arXiv 2509.17765). It is not a DiT or flow model.
  • MaskGCT is mask-and-predict in both stages (arXiv 2409.00750). Do not call it flow matching.

The S2S design space in one picture

Every speech-to-speech system answers the same five questions. The model zoo looks large only because the answers combine in many ways.

4 Streaming / duplex scheme wraps the whole path Input rep decision 1 Backbone LLM swappable Output rep decision 2 Renderer acoustic, decision 5 speech in speech out Text stream scaffold / reasoning decision 3: text / speech coupling
The five decisions around one backbone LLM. Decision 4 (streaming/duplex scheme) wraps the whole path.
Decision Options Common pick Inference
1. Input representation Continuous encoder embeddings; discrete semantic tokens; codec tokens Continuous embeddings from an ASR-pretrained encoder at about 12.5 Hz
2. Output representation Multi-codebook codec tokens; single-codebook semantic tokens; continuous latents Semantic or codec tokens, rarely continuous
3. Text/speech coupling Cascade; text-first (half-cascade); Thinker-Talker; interleaved; parallel streams Text-first or interleaved
4. Streaming/duplex scheme Turn-based; chunked; time-multiplexed; dual-stream; state head Chunked, with a duplex layer on top
5. Acoustic renderer Codec decoder; flow-matching decoder plus vocoder; causal ConvNet AR LM for content, flow or ConvNet for acoustics

The core tension

Speech encoders emit frames at 50 Hz (Whisper, as used in Voxtral; arXiv 2507.13264), while text is far sparser. Every decision above is a way to give the LLM a sequence it can model at text-like cost. The goal is to do that without losing prosody, speaker identity or timing.

Why the LLM can be "downstream"

If the backbone sees speech as embeddings and emits tokens a separate renderer turns into audio, the reasoning model can be swapped, frozen or fine-tuned lightly. Freeze-Omni (arXiv 2411.00774) takes this to the limit by freezing the LLM.

The open problem

Keeping text-level intelligence once the model must speak. URO-Bench reports that open-source spoken dialogue models lag behind their backbone LLMs on instruction following and suffer from catastrophic forgetting (arXiv 2502.17810).

In the interview, say:

"I split any S2S system into five choices: input rep, output rep, text coupling, duplex scheme and renderer. Most new models differ in one or two of them, not all five."