Speech-to-speech: literature review
Model-by-model literature review of speech-to-speech systems: speech input and ASR, codecs and generation, streaming and full-duplex models, control, multimodal generation, common problems and system design, with inline citations.
-
1. Foundations
The speech-to-speech design space and the five decisions every system makes.
-
2. Speech understanding
How speech enters an LLM, and how ASR is done with speech LLMs, offline and streaming.
-
3. Speech generation
Codecs, autoregressive and masked generation, diffusion and flow matching, continuous autoregression, and streaming.
-
4. Speech-to-speech systems
Architectures that put an LLM between speech in and speech out, full-duplex conversation, and multilingual coverage.
-
5. Streaming conversational modelling
Rates and clocks, producing text and speech together, Moshi and other full-duplex models, turn-taking, streaming inference, training recipes, RL rewards, and conversational evaluation.
-
6. Control and quality
Steering voice, tone and emotion; post-training and evaluation for speech generation.
-
7. Multimodal generation
Audio-visual diffusion, autoregressive and unified models across modalities, tokenizers beyond speech, and multimodal LLM architecture.
-
8. Common problems
Failure modes in speech generation, speech-to-speech and audio-visual generation, grounded in Synthesia products, with metrics and mitigations.
-
9. System design
The target: a unified speech LLM (streaming ASR + expressive TTS, barge-in and backchannels) with a downstream reasoning LLM as a tool; the cascade baseline; real-time avatars; and a Synthesia-style audio-visual platform.
-
10. Reference
Cheat sheet of key models.