Quick read

Foundations: the S2S design space

• Every S2S system answers five questions: input rep, output rep, text/speech coupling, duplex scheme, acoustic renderer.
• Core tension: speech is dense (50 Hz or more), text is sparse; each choice gives the LLM a text-like-cost sequence without losing prosody, identity or timing.
• If the backbone sees embeddings and emits tokens for a separate renderer, the LLM can be swapped, frozen or lightly tuned (Freeze-Omni, arXiv 2411.00774, freezes it).
• Open problem: keeping text-level intelligence once the model must speak; open S2S models trail their text backbones on instruction following (URO-Bench, arXiv 2502.17810).

Full chapter

Speech understanding: input and ASR

• Most models feed the LLM continuous embeddings from an ASR-pretrained encoder plus a small projector (Voxtral, Qwen3-Omni, Granite-speech, Step-Audio 2, Canary-Qwen).
• Discrete tokens appear when the same vocabulary must be generated (GLM-4-Voice, 12.5 Hz, 175 bps); codec tokens suit duplex (Moshi's Mimi, 12.5 Hz, 8 codebooks) Inference.
• 12.5 Hz converges because Whisper emits 50 Hz frames and a 4x adaptor is a necessity; Kyutai also chose it to sit near the text token rate.
• Downsampling to 10-12.5 Hz is the main sequence-length lever and costs fine acoustic detail, which motivates hybrids like Kimi-Audio (arXiv 2504.18425) Inference.
• Streaming ASR: chunk size and lookahead trade latency against WER; no verified table exists Inference.
• Canary-Qwen freezes the LLM and trains encoder, projection and LoRA (5.63 mean WER on Open ASR Leaderboard, RTFx 418, HF card).

Full chapter

Whisper in one page-half

• Encoder-decoder Transformer, 680,000 h weakly supervised; one model does ASR, X-to-English translation and language ID via task tokens.
• Hz ladder: 16 kHz audio, 100 Hz log-mel (80 bins, 128 in large-v3), 50 Hz encoder frames (1,500 per 30 s window), 12.5 Hz after a 4x adapter.
• Training: filtered web data, language-ID cross-check, large-v2 added SpecAugment and 2.5x epochs; large-v3 used 1M h weak plus 4M h pseudo-labelled by v2.
• Long-form: sliding 30 s windows, beam 5, temperature fallback when logprob is below -1 or compression ratio above 2.4, no-speech threshold 0.6.
• Weak spots: hallucination on silence, repetition loops, no streaming; large-v3 hallucination reports conflict; turbo (4-layer decoder) is not trained for translation.
• Speech LLMs reuse the encoder (Qwen2-Audio, Voxtral, GLM-4-Voice causal variant); Qwen3-Omni replaced it with AuT Inference.

Full section

Speech generation: codecs, AR, flow, streaming

• The tokenizer fixes the LM's sequence length and the quality ceiling; the trade is rate against fidelity.
• Mimi is causal, 12.5 Hz, about 1.1 kbps, with the first codebook distilled toward WavLM; WavTokenizer runs at 40 or 75 tokens/s; VibeVoice uses continuous 7.5 Hz latents.
• Low rate bites: a 2026 probing study reports WavTokenizer-75 WER 11.37% and WavTokenizer-40 WER 24.36% (arXiv 2606.16969).
• The dominant high-quality design is hybrid: an AR LM for content and prosody, flow matching for acoustics (CosyVoice 2, Seed-TTS, Qwen3-TTS, FlashTTS).
• Flow matching largely replaced diffusion: a plain regression target, straighter paths, fewer steps, natural infilling Inference.
• Continuous AR (MELLE, DiTAR, VibeVoice) removes quantization loss but costs per-step sampling, harder RL and weaker sampling control.
• Latency levers: for flow decoders, step distillation (FlashTTS 325 ms at 2 NFE vs 843 ms for CosyVoice 2 at 10 NFE); for AR, lower token rate plus a causal codec Inference.

Full chapter

Speech-to-speech systems: architectures, duplex, multilingual

• Six coupling families, from loose to tight: cascaded, half-cascade, Thinker-Talker, interleaved, parallel streams, codec-free; production leans on the first three.
• Speaking costs reasoning quality; mitigations are text-first generation, a frozen LLM, thinking variants and hidden-state alignment.
• Half-cascade or Thinker-Talker with a frozen or lightly tuned LLM is the pragmatic consensus; native interleaved is the frontier Inference.
• Duplex schemes: dual-stream (Moshi, dGSLM), time-multiplexed (OmniFlatten), state head (Freeze-Omni, MinMo), shared timeline (MiniCPM-o 4.5), cascade-duplex (DuplexCascade).
• Data is the bottleneck: Fisher is about 1,960 h dual-channel, synthetic TTS dialogue has artefacts and no real overlap, unlabelled audio scales (Kimi-Audio more than 13M h).
• Benchmarks: Full-Duplex-Bench v1 and v1.5, VoiceBench, URO-Bench; evaluation is maturing but not comparable across papers.
• Multilingual is lopsided: Qwen3-Omni has text in 119 languages, speech understanding in 19, speech generation in 10; text streams carry supervision, scaffolding and capability transfer.

Reported DuplexSLA reports 98.33% backchannel handling vs 40% or less for baselines; PersonaPlex-7B-v1 reports 90.8% turn-taking and 240 ms interruption latency. Both are self-reported or secondary.

Full chapter

Streaming conversation: rates, Moshi and full-duplex models

• Moshi runs text, agent audio and user audio on one 12.5 Hz clock: 17 streams per 80 ms frame.
• A 4096-dim temporal transformer runs once per frame; a 1024-dim depth transformer fills the 17 sub-tokens.
• Text is padded (about 65% PAD), so it fits the audio clock; GLM-4-Voice instead interleaves 13 text : 26 speech tokens.
• Moshi's latency is 160 ms in theory and about 200 ms in practice; DSM makes the text-audio delay a per-stream setting.
• Duplex families: dual-stream (Moshi), chunked (SyncLLM), flattened (OmniFlatten), control head (MinMo, Freeze-Omni), action channel (DuplexSLA).
• Latency figures use different definitions and are not comparable Reported.

Full chapter

Streaming conversation: training, RL rewards and evaluation

• Recipe: strong text LLM, speech tokens, ASR/TTS and interleaved pretraining, spoken SFT, then DPO or RL.
• Choice: frozen LLM (Freeze-Omni, MinMo) or mixed text (Moshi, GLM-4-Voice, Kimi-Audio, Step-Audio 2, Qwen-Omni).
• DPO goes on text-token probabilities (Moshi-Aligned, arXiv 2506.21463) or the Talker (Qwen-Omni); RL is PPO/GRPO (Step-Audio 2) or GSPO (Qwen3-Omni).
• Online RL: group rollouts, decode, composite reward, per-group advantages, KL, held-out judges. Kyutai is the only lab with public voice-RL checkpoints.
• Duplex rewards are timing events from VAD (DuplexPO, arXiv 2607.07148; Kyutai, arXiv 2606.11167) plus a text-quality term against drift.
• Evaluate on Full-Duplex-Bench v1, v1.5 and v2 and TurnBench; humans start a median 151 ms before the turn ends.
• Report latency at a fixed false-yield rate, from gold end-of-turn Inference.

Full chapter

Control and quality: steering, post-training, evaluation

• Five control channels, usually combined: reference clip, natural-language instruction, inline tags, duration targets, style guidance at sampling time.
• IndexTTS2 (arXiv 2506.21619) separates emotion from timbre, so the emotion prompt can come from another speaker, and offers exact-duration control.
• Exact durations let audio match a video block length, the hook for avatar sync Inference.
• Post-training uses ASR and speaker-verification rewards (WER, SIM): DiffRO scores tokens with no waveform in the loop, F5R-TTS applies GRPO to flow outputs, GROW does on-policy GRPO for AR-diffusion TTS.
• The recurring failure is reward hacking; a prosody study found GRPO on CER and NLL rewards made prosody monotone, fixed by iterative DPO with human preference pairs (arXiv 2509.18531).
• Always pair automatic rewards with a naturalness term and listening tests Inference.
• Synthesia Express-2 (Sept 2025) pairs Express-Voice with Express-Video; the blind-test result is a company claim with no published scores Reported.

Full chapter

Multimodal generation: audio-visual diffusion and AR models

• Video backbones share one recipe: 3D causal VAE, DiT, flow matching (Wan, HunyuanVideo 1.5, LTX-2, Seedance 1.5 pro) Inference.
• Audio-driven avatars reuse those backbones with audio cross-attention (OmniHuman-1.5, HunyuanVideo-Avatar, MultiTalk, InfiniteTalk, Wan2.2-S2V).
• Ovi generates audio and video jointly: twin DiT towers, bidirectional cross-attention in every block, audio RoPE scaled by $31/157$, loss weights $0.85/0.15$.
• Streaming distils a bidirectional teacher into a causal few-step student with a KV cache (CausVid, Self Forcing, LiveTalk); error accumulation is the failure mode.
• AR image models split into discrete (VAR, Emu3), continuous with a diffusion head (MAR, Fluid), and hybrids (Transfusion, Show-o2, BAGEL).
• MELLE, DiTAR and VibeVoice are the speech analogue of MAR and Fluid.

Full chapter

Multimodal generation: tokenizers and multimodal LLMs

• The tokenizer matters more than the generator: causal, compressed latents aligned to frozen SigLIP or DINO features Inference.
• Discrete codecs (MAGVIT-v2 LFQ, FSQ, DAC, EnCodec) suit LMs; continuous VAEs (Wan, Cosmos CV, VA-VAE, Stable Audio Open) suit diffusion and flow.
• AVTok (arXiv 2606.30811) is an early joint audio-video tokenizer; VideoPoet concatenated MAGVIT-v2 and SoundStream vocabularies.
• Multimodal LLMs share one template: vision encoder, token-reducing connector, decoder LLM.
• Qwen3-Omni's TM-RoPE uses absolute-time IDs (80 ms per audio ID), so streams can be any length; Qwen2.5-Omni used fixed 2 s chunks.
• StreamingVLM keeps a fixed KV cache and runs at up to 8 FPS on one H100 (arXiv 2510.09608).

Full chapter

Common problems: speech and audio-visual

Ranking, fixes and metrics are my judgement Inference. Synthesia publishes no failure metrics for Express-2 Confirmed.

Full chapter

System design: the unified speech LLM

• One speech model owns everything audio: streaming ASR with emotion tags, turn-taking, backchannels, barge-in and expressive TTS on one 12.5 Hz clock Inference.
• Reasoning stays in a swappable text LLM behind a tool call: transcript, tags and turn state go in, streamed text comes back and is spoken.
• Fillers and backchannels come from the speech LLM, so the extra LLM hop is covered; the hop and two models to serve are the honest costs.
• The speech LLM, not the text LLM, yields on a real barge-in, ignores a user "mm-hm", and sends the played and heard text back Inference.
• Precedents: Kyutai DSM (ASR and TTS as delays of one architecture), Step-Audio 2 tool calls, Qwen Thinker-Talker, Freeze-Omni and MinMo state heads Confirmed.
• Train with unlabelled audio, interleaved synthetic data, dual-channel conversation, then DPO on text tokens and timing RL (DuplexPO, Kyutai).
• Native full-duplex is the research track; the independent ASR, LLM and TTS cascade is the baseline and fallback.

Full section

System design: voice agent and talking avatar

• Target: a unified speech LLM with a downstream text LLM (previous page). The cascade below is the baseline, launch fallback and comparison point Inference.
• A cascaded voice agent is overlapping streams with a turn manager on top; latency comes from overlap, not any single model.
• Use WebRTC to the client (LiveKit, Pipecat) with a control data channel; persistent bidirectional streams between servers, with data and control planes separate.
• Replace a bare VAD timeout with a semantic end-of-turn model; it shifts the dead-air versus cut-off curve but does not remove it Inference.
• Never feed raw partials to the LLM; start it at the end-of-turn candidate and cancel on change Inference.
• Chunk LLM text for TTS with a small first chunk at a clause boundary, then full sentences Inference.
• Barge-in: cancel LLM and TTS, flush buffers, truncate context to what was actually played, return to listening; echo cancellation is the precondition.
• Avatar: the speech LLM drives a causal few-step renderer; the cascade is the fallback and native duplex is the research track Inference. Audio is the clock.

Cascade fallback budget:

Full chapter

System design: audio-visual avatar platform

• Two products, two SLOs: batch video (3-10 min render, 1080p, 30 fps) and interactive agents (under 1 s) Confirmed.
• Shared consent-gated identity and voice stores; consent is a hard pipeline stage Inference.
• Batch: script, TTS, audio-to-motion, motion scoring, few-step DiT render (2 steps), QA, webhook. Dubbing adds ASR, MT and duration-controlled TTS.
• Audio-then-motion cascade in production, joint audio-video (Ovi style) as a research track; causal few-step student for interactive Inference.
• Interactive voice: a unified speech LLM with a text LLM behind a tool call; the ASR, LLM, TTS cascade is the fallback and baseline Inference.
• One clock: absolute-time positions across voice and video (Qwen3-Omni style; Ovi rescales audio RoPE) Inference.
• Cross-paper Sync-C is not comparable, so build an internal benchmark first; guard preference rewards against hacking.

Full section

Model cheat sheet

"n/s" means not stated in the research notes; "Cont." means continuous embeddings.

Five sentences to remember

1. Input converges on an ASR encoder at about 12.5 Hz.
2. Output is an LM for content plus a flow or ConvNet renderer.
3. Text is the scaffold and the supervision.
4. Duplex is limited by data and evaluation, not architecture.
5. Latency comes from token rate, NFE and chunking.

Full chapter